Coming sort order changes for "editorial preference"

Hi everyone,

Last night, while starting to flesh out the listings with review excerpts, I noticed a number of odd behaviours with the existing editorial preference (now editor acclaim) sort order algorithm. For instance, a number of items marked "exceptional" aren't ending up at the top of the list, where they probably should be. There are three reasons, in all, only one of which is intentional.

Initially, when building the site, I thought we'd be catering to people looking for serials -- stories that were updating on a regular basis. As a result, the algorithm gives a high premium to update frequency, which means that stories that are already complete fall a ways down the list. This has the benefit of keeping "new" material at the top of the list, but has the cost of pushing potentially great material well down the list. I'm not sure yet how/if I'm going to address this, but I will be looking into over the next few days.

The other two issues are unintentional side-effects of some of the internal workings of the algorithm. The current algorithm pays an enormous premium for high ratings, and a fair premium for each additional editorial rating. I made those choices, originally, to solve some problems I saw in the early days, but I think those choices are now having consequences of their own -- things I'm only now able to quantify with the richer set of information displayed on the new listing indices. So, I've started redesigning the algorithm to address these issues.

What does all of this mean? Well, primarily it means that the "editorial preference" order is likely to change in the next few days. My goal is to make the sort order seem more natural, with respect to the displayed recommendations and editorial rating. As I said, I'm not yet sure what I'll be doing about update frequency, but I will definitely be working to resolve the other two issues.

So, if you see your listing jump around in the sort order in the coming days, this is why.


Hi all,

So, I've just implemented these changes on the site. The new algorithm should more closely match the descriptive text on the new indices.

In terms of the changes, I have greatly reduced the premiums paid for extra editorial ratings, extra editorial recommendations, and high update frequency. They still come in to play, but are much smaller than they used to be. I've also added a fifth sort order ("all-time best"), that effectively eliminates the update frequency premium -- a sort order where the "classics" float to the top.

Finally, I've notified the editorial staff of what sorts of things we should be doing to ensure the new sort algorithms work properly. Over the next few weeks, we'll be trying to supply the additional information needed to bring everything into line.


I've been looking at the way the editorial listings work right now, and it appears a bit out of whack. It seems to count editor star scores only and ignores everything else, such as member scores and overall ratings (and even general editorial ratings), so that everything with one or more editor reviews of 4 stars or above is randomly assembled onto the first few pages. There no longer seems to be any rhyme or reason to the ordering of the listings.

Am I the only one thinking along these lines?



Hi Ryan,

The problem stems from the fact that the formula for the average editorial rating (the primary currency of the default listing) has changed.

In the past, the average was calculated from the ratings from any and all editorial reviews on the listing. This meant that if there was only one editorial review, its rating would become the average editorial rating -- any ratings that did not include a review were ignored. The new definition is somewhat more complicated. First, it no longer requires a review -- the ratings are used directly, whether there is an attached review or not. However -- and here's the problem you are seeing -- I no longer consider one editorial rating for the average. If there are two editorial ratings within 1 star of each other, I'll take the average. Otherwise, it takes three or more, from which I discard the highest and lowest rating (one of each) and average what is left.

For the ranking algorithm, listings with an official average editorial rating are treated as if all editorial ratings are at the average, while listings that don't yet have an official average editorial rating use the individual editorial ratings at a 12.5% discount.

The actual math in the ranking algorithm then gets even more complicated, taking into account the number of recommendations and the weight of opinion. And the default ranking (not the all-time best ranking) also includes a bonus for being "ongoing", so stuff that is currently updating tends to float to the top.

Every one of these decisions was made for reasons of long-term fairness -- a means of measuring, roughly, "what to we collectively prefer today" -- but a lot of the existing records have insufficient information to fully participate. We're working on filling in the additional data -- specifically, getting additional editorial ratings on every listing. But it's going to take some time.

There's also some weirdness in the way I'm displaying the average editorial rating that will need to be addressed at some point soon.


Chris; Thank you for the effort in coming up with a workable formula. Most sites don't care.

After writing that, it occurred to me that better labeling was in order, so I've changed the ranking names to be a bit more descriptive.

Also, as a side note, this algorithm is new, and I consider this a "real world" testing phase. It might need to be adjusted, yet -- particularly if we don't actually get enough additional ratings in place to make the thing fair.

Finally, it's too early to say for sure -- as holiday traffic patterns may not continue -- but it looks to me like the new listings are encouraging browsing. I'm seeing a bit more traffic to listings that don't appear on the home page, which is what I was hoping to accomplish. There's still a lot of work to do -- adding review excerpts, specifically -- but, I'm cautiously optimistic that the new design is working out.

Hi all,

FYI, I'm moving the conversation on the ratings and rankings back here from the other thread.

I've updated the algorithm with the changes I mentioned in the other thread. The new algorithm all-but-eliminates the bonus paid for multiple editorial ratings. The "official" average editorial rating is still calculated as described above, and is considered the most reliable rating, but "unofficial" averages are used at only a very small discount. The benefit is that we can now take our time getting additional ratings on the collection, as rankings won't change much if the additional ratings agree with the first. The cost is that listings may jump large distances if a second or third editorial rating significantly changes the average (the old algorithm was better at keeping the rankings stable -- ensuring most listings moved only up in rank with additional ratings).

Bonuses for recommendations and (in the "current favourites" listing) update frequency are still paid as before.

I've also redone the rating descriptions on the indices, to make things more transparent.