Last night, while starting to flesh out the listings with review excerpts, I noticed a number of odd behaviours with the existing editorial preference (now editor acclaim) sort order algorithm. For instance, a number of items marked "exceptional" aren't ending up at the top of the list, where they probably should be. There are three reasons, in all, only one of which is intentional.
Initially, when building the site, I thought we'd be catering to people looking for serials -- stories that were updating on a regular basis. As a result, the algorithm gives a high premium to update frequency, which means that stories that are already complete fall a ways down the list. This has the benefit of keeping "new" material at the top of the list, but has the cost of pushing potentially great material well down the list. I'm not sure yet how/if I'm going to address this, but I will be looking into over the next few days.
The other two issues are unintentional side-effects of some of the internal workings of the algorithm. The current algorithm pays an enormous premium for high ratings, and a fair premium for each additional editorial rating. I made those choices, originally, to solve some problems I saw in the early days, but I think those choices are now having consequences of their own -- things I'm only now able to quantify with the richer set of information displayed on the new listing indices. So, I've started redesigning the algorithm to address these issues.
What does all of this mean? Well, primarily it means that the "editorial preference" order is likely to change in the next few days. My goal is to make the sort order seem more natural, with respect to the displayed recommendations and editorial rating. As I said, I'm not yet sure what I'll be doing about update frequency, but I will definitely be working to resolve the other two issues.
So, if you see your listing jump around in the sort order in the coming days, this is why.