Reviewing The Reviews

Thought I'd get a feedback loop set up for all of WFG's reviews (editor and member alike). Here's the deal - you post up (and link!) any review that you like, or a review that you don't, and you tell us why. It can be by a member, editor, whatever (speaking for myself, you can kill me if I've written a bad one) and trust me, I won't bite =).


A couple of guidelines:

1, Be honest. The whole WFG community will probably be learning from what you say here, so please tell us what you think.


2, Be polite. You know that guy. Don't be that guy.


3, Carrot over stick. Point out reviews you've enjoyed, not just the ones that've irked you off, and plug reviewers you think are worthy of recognition. I'm sure we can all do with better reviews, and praising people for their efforts is a good way of getting there.


4, Learn! As I said, we can all be better reviewers.


5, Teach! If you have issues with a review, please post up suggestions and alternatives. It's much easier to see your argument if you demonstrate how it should've been done.


Anyone up to this?


Allow me to get the ball rolling:


I think the member reviewer that I've enjoyed most recently is Miladysa. She's very honest with what she says, and everytime I read one of her reviews (of which there are many), I get the feeling that this is somebody who absolutely loves reading web fiction. Not particularly sure about the rating bias she has (most of her reviews are 4-5 stars, which is scary, but I'd like to think that's because she reviews what she likes) or the fact that some of the reviews she does are vague. But there's this very ... bubbly tone to what she writes, and I like that. I really, really like that.


Not all reviews are quite so charming. I was doing one on The Tom Drake Experience not too long ago, and there was this one review that irked me off to no end:




And it was a five star review. I mean, what does that tell me, as a reader? If I was new to WFG, and I stumbled onto this page by accident, would I be impressed by a three sentence five star review? Hrmmph. =(


I think this a brilliant idea ejames :D


I will BBL to add my bit about reviews but for the time being here is my response to the points you have raised regarding my own:


1. Thanks :D


2. I am honest in all my reviews


3. I LOVE web fiction!!!!


4. Web Fiction writing and reading is the way I relax and I prefer to make it as pleasurable an experience as I possibly can.


So far I have probably read more than four times as many sites as I have taken the time to review or recommend. I have only committed my (much treasured free) time to reviewing the ones I thoroughly enjoy/ed and can be really be positive about.


There are other sites that I am enjoying and intend to review when there is a bit more of the story posted. This is because I have noticed many have started well, received great reviews in some cases and have then been abandoned (I'm not talking about the writer taking a break - heaven knows we are entitled too!) and as a reader you are left mid air.


I realise that some people may say that I should take the time to review the sites I did not enjoy and give the writer some feedback and an opportunity to address the issues I think are there. My answer to that is that in most cases these sites have already received a review pointing out the same things and a. I do not wish to rub salt into an open wound and b. I can add no more than what has already been said.


Finally, I have noticed that many writers (including myself)have taken on board the feedback given in their reviews and edited their stories and/or adressed the issues within their sites. In such cases I have updated my reviews accordingly and have raised my star ratings. I would love to see some of the Editors do the same thing where warranted.


Once again, many thanks for starting this thread!


I don't know if I need to link to them, exactly, because I want to broadly say that I like Eli's reviews. I've been an admirer of his clear writing style since Novelr, and it's been a privilege to write some articles there and work with him here. I find his reviews are deeply thoughtful, and insightful. I measure my best reviews by his standard, as I think Eli tries to both educate about literature in general, and also, I think he tries to help explore the possibilities for fiction specifically offered by the Web.


The only other editor that I think goes into a similar depth in a review might be Chris, even though everyone tries to be as honest and informative as possible. But the three of us seem to attempt essays every once in a while. :)


I like Miladysa's and Donna Siranni's reviews. I can't think of any reviews in particular, but I have the reviews in an RSS feed so I enjoy reading them and checking out the new listings.


Honestly, I'm a little bit afraid of the editors. A few of them seemed to be a little biased (I'm not going to say any names) as a lot of reviews start off stating "I'm not really into this genre, but...".


I'm updating a few of my reviews to have a summary of the story, my likes, dislikes, and my overall. I find that this works better (I don't know why I really didn't think of this before.)


At any rate, I was curious. How do the editors go about choosing which stories they review?


Personally, I mainly go with what sounds interesting. That being said, I also look at whether something is really old and hasn't gotten reviewed yet. In that case I'll do it if no one else seems interested (whether it sounds interesting or not).


I suspect that other editors do the same.


I've only been here a short time, but here's one review that I'm having difficulty with which I found when I started reading "Street"... http://webfictionguide.com/members/snitschke/street/


The reviewer's opinions are generally favorable with what she makes clear are some misgivings, but the score and her conclusion do not match up at all with the body of the review. Apparently she revised the score, because she had allowed her personal preferences to get in the way, which strikes me as practically heresy. If you're going to review something, then you have to review it from where you stand, nowhere else. Now, I get the feeling that there is definitely something I missed, and if I'm opening up an old can of worms then apologize, but her words don't seem to argue for a nearly perfect grade. As if she had an opinion about the work, but changed it to fit...expectations.


And I'm not trying to make a comment on the work she reviewed here, or slight Sonja Nitschke in any way, but this struck me as a pretty large discrepancy right out of the gate. I haven't yet had the chance to read many other reviews and only written two myself, so I can't comment any further on anything else yet, so unfortunately I have no positives to balance out my observation (if it bears out as a negative, that is).


@Miladysa: You're welcomed. =)


On the positive skew in member reviews: it's actually quite worrying for us editors, to tell you the truth - if everyone were to post up only 4 stars and above, that wouldn't make WFG very useful for readers, no? I do understand your decision to only review the good ones, and I appreciate your willingness to revise your scores upwards should the writer improve. But we're also starting to wonder just how problematic this might be in the future - there's often a huge dissonance between editor and member reviews on less popular works. It's as if the member reviews that come after an editor are purposely polarized as to try to counter-act the editor in question. =( Also, reviews from the review exchange thread tend to be positive, and even if they were negative the scores are very rarely below 3. Is this because of the nature of the exchange? Or is it because the members are chums? I'm not really sure. Anyone has any other opinions on this trend?


And yes, on the upward revision of scores, we tend to do that if we have a bit of breathing room with the unreviewed listings. I'm on assignment from Chris, for instance, to review the listings that have too little editorial reviews.


@Gavin: Thanks Gavin! This is rather ironic, considering that I give your reviews more weightage when considering what works to read next. And, yes! We do have a tendency to write essays to justify our scores. I think we have a tendency to write too much.


@Sora:


A few of them seemed to be a little biased (I'm not going to say any names) as a lot of reviews start off stating "I'm not really into this genre, but...".


Any suggestions or alternatives to this? Do explain, please.


I've generally thought that it was only right to disclose any bias you might have against (or for) a genre at the outset, and then proceed to attempt to rate the review based on normal expectations of said genre. But I may be mistaken, of course. What you've said about the editors being scary does seem to apply across the board, however - we've had reports of people being afraid to list with us. Is there any reason you may think for this? The tone of reviews, perhaps?


@Samazing: I'll let Sonja respond to that herself, but for the record, my next assignment is Street, and I'll probably be writing another essay.


Eli: I reckon it's not a bad thing that the editorial part of the site puts a bit of pressure on writers -- it's what makes the WFG different from every other web fiction listing site. Basically, if you submit here then you'd better bring the goods.


The main thing people seem to be scared about, at least to my mind, is that a bad review here could put off readers that might've given it a chance otherwise. They're wrong, of course, and that's a very counterproductive attitude for a writer, but it's a very easy trap to fall into.


Sora has a point about the reviewer bias, though. I think it'd go a lot smoother if you matched up editors and stories with their own preferences rather than just selecting at random. One possible system is to give stories two editor reviews to start with, one by someone who likes the genre and another by someone who doesn't. That would broadly investigate the story's appeal to any passing reader whether it's something they'd normally pick up or not.


As for Sonja's Street review, I'm not going to comment on it myself. I expect Sonja can speak for herself.


I hope it's a good essay, by the way. ;)


Regards,

Ryan


@ejames - As a reader I am only going to read what appeals to me - I'm not reviewing because it is expected of me as an editor.


If I am not enjoying a story I abandon it after a fair whack - I'm not then going to waste my precious free time reviewing it when I could be working on my own story. Neither am I going to bother reviewing many of the editor sites which have already received numerous reviews from other editors unless I really rate the story. When I really rate a story I then take the time to write a review and rate it highly.

Where I have reviewed a site it is guaranteed that my review is based on the ENTIRE story at the date of that review - not just five chapters. Where I have taken the time to review a story with only a few chapters posted you can bet that I am really looking forward to reading more from that writer.


As for possible bias in the review exchange I personally find that idea insulting. Perhaps our reviews are more positive because we have volunteered to review and do not feel obligated to do so?


The WFG claims to be reader rather than writer orientated - if editor reviews are so far removed from reader reviews then I think the editors need to look at their review rating. When I write a review the gist of it is based on the enjoyment I gained from the story and how enjoyable I think it will be to another reader.


In my opinion some of the editor reviews can border on nasty - I'm not going to single anyone out because I do not think there is any one person in particular.


"It's as if the member reviews that come after an editor are purposely polarized as to try to counter-act the editor in question."


After reading some reviews I wonder if the editors have actually read the same story as myself and from comments on other sites I know others have the same thoughts too. Could this be because some of you are not taking the time to read through the whole story? Possibly.


I know your time is precious like the rest of us but you have chosen to be in that position and if you are going to do reviews I think you should read through what is available at the time of that review. Personally I do not think it would be a bad idea if a story with only a few chapters was left to develop before they receive an editor review.


Finally - my favourite editor is Chris Porrier - you can tell from his reviews that he has taken the time to read through the story, put some thought into constructive rather than destructive criticism and he is always fair. I do not think Chris has frightened anyone away from here but I am sure some of the editors have.


I just wanted to add that typos, spelling, grammatical and other such errors do not in any way distract from my enjoyment if the story grips me. I am sure that some WFG members will vouch for the fact that where I have noticed such oversights I have taken the time to mention them in the comments section or sent an email to the author. I have also entered into lengthier emails with writers giving additional feedback where requested.




@Ryan: Good point about Sora's idea. And as for the essay, I'm not saying either way - my only job is to be honest. =)


@Miladysa: Thank you for your feedback - what you've said will allow us to improve the site, and the user-experience. However, it appears to me that you've taken what I said personally, and I apologize if I've given the impression that I was singling you out (and how could I, considering I enjoy your reviews?).


I struggled with the idea of not finishing a whole work before doing a review, but you have to understand the pressure and the reality of being an editor at WFG. We have to cover and evaluate all new listings to ensure that our users are able to find things they'll like to read; so we work under a deadline and a steady backlog of unreviewed, unrated new works. I've written about the process in Novelr here, and allow me to quote the related portion:





An alternative would be greatly appreciated, but at the moment this is the best we've got. Trust me, we're trying hard.


As for possible bias in the review exchange I personally find that idea insulting. Perhaps our reviews are more positive because we have volunteered to review and do not feel obligated to do so?



Do you suggest, then, that we close an eye to the possibility of bias? I believe, Miladysa, that it's fully within our scope to be worried about the quality of member reviews, and about how we're going to effectively leverage the community to help the a new reader find good things to read. We're constantly trying to overcome and compensate for bias, or for gaming the system. In fact, WFG started in part from an analysis I did on the fallacy of small crowds in Pages Unbound. For instance, when I say that a review is purposely polarized as to counteract an editorial rating, I mean this and this. Look at those and tell me that our concerns are unfounded. Chris built WFG from the ground up trying to prevent fanboys from gaming the system and shooting a sub-par work to the top of the listings. This is a challenge that we've faced on a day-to-day basis, working here. But I believe it is a necessary challenge, if we want online fiction to be widely read.


I agree with you that WFG is reader oriented, not writer oriented (though that issue is currently heavily debated by the editors). But let's set aside that debate for a moment and accept that I agree with you: WFG is reader oriented in the sense that we try our best to help readers find good things, new things to read. This is our editors' focus. Your point about how editors need to look at their ratings if they're dissonant from the member's ratings misses this point - our job is to tell you if it's worth your time. Members, however do not have this obligation, and they review for different reasons. Some review because they know and like the author. Others just want to share what they've found. Still others do it because of a review exchange, or maybe they do it because they're not happy with the scores an editor has given. And is it not logical that when you do a review exchange, you don't want to be too harsh, that there is a psychological factor that prevents you from being too negative with the writer you're exchanging a review with? This is the chum factor that I speak of. Mind, we're supportive of the review exchange, because we think it's healthy. But we do have questions, and we're constantly thinking about the possible downsides, because it's our jobs to do so.


I know your time is precious like the rest of us but you have chosen to be in that position and if you are going to do reviews I think you should read through what is available at the time of that review. Personally I do not think it would be a bad idea if a story with only a few chapters was left to develop before they receive an editor review.



Please, consider your words. I feel uncomfortably taken for granted when you put it like that. I too, have family, and all of us have our daily jobs and our real-world responsibilities. We do this only because we're passionate about the cause of online fiction. As for the part about editors witholding from reviewing new works, we've recently had a discussion about that in the private editor forums, and the very suggestion you've proposed have been in effect for two weeks now.


In my opinion some of the editor reviews can border on nasty - I'm not going to single anyone out because I do not think there is any one person in particular.



Do explain, and give me your thoughts, and concrete examples. The whole point of this thread is to overcome some of the perceptions people have about editorial reviews. We're trying to get to the bottom of this and solve it. Is it our tone? Is it the way we handle new writers? How can we improve? Help us.


@Samazing


I wrote the review when WFG was still growing up. Editors roles were still being hammered out, WFG's purposes were still trying to be defined, even the meaning of the Star Ratings was being questioned and changed. The ratings didn't mean then what they mean now.


Shortly after I wrote the review, there was talk that some of us were coming across reviewing from not quite the right direction. I considered everything that was said in that thread and realized that, as an editor, I did not have the luxury of reviewing from my own tastes (which is what I had done before). You said,


"Apparently she revised the score, because she had allowed her personal preferences to get in the way, which strikes me as practically heresy. If you're going to review something, then you have to review it from where you stand, nowhere else."


However, I stand for a lot of things: I stand for good writing, three dimensional characters, and plot pacing. All of the above is what I base my star rating. I let people know my personal opinion, but a star rating is supposed to be a flag that helps people decide if they want to read it or not -- not as an echo of my personal opinion.


Am I wrong to think that, as an editor, my opinion shouldn't be reflected in the star rating? I don't know. We editors get a lot of mixed messages. People have implied that they are afraid to list with us. People have said that we are biased in some form or another. Is it really a surprise that I attempt to keep my unmeasurable, and oftimes illogical personal preferences away from the very measurable, logical algorithms which incorporate the star ratings into the functions of WFG?


I've changed my score of Street three times -- each time after the site was changed or the ratings redefined or something like that. It looks as if I need to change it again with a clarification of the review because 5 stars didn't mean then what it means now.


I need to go back through this thread and read everything (I'm skimming) but I just want to say:


In my own reviews, I acknowledge my bias. So far, I've noted that I don't like vampire stories (overdone in the past 100 years)and I don't like diaries, I prefer novels. That's so that readers can know how to look at my reviews -- if they like vampire stories, they can automatically assume a story I rate 3 or 4 stars will in fact probably be a 4 or 5 star story for them, as the vampire thing won't be a problem.


Part of the idea for WFG (my understanding, anyway) was that by having editors with their own styles, readers can get to know those personalities and find an editor whose style fits their own -- If someone likes fantasy stories, for instance, they'll find that I do too, based on how I review. (that's just an example)


We need to acknowledge our biases as a way of explaining why we rate the way we do. A 2 or 3 or 5 star story isn't rated arbitrarily, it's rated based on our individual preferences. No one is objective -- the site itself strives for objectivity by letting members and multiple editors review stories, to come up with an average rating.


@ejames You are correct - I saw a red mist earlier and took it personally - I apologise.


I do not have the time right now to comment on some of the good points you have made here and for the most part I understand what you are saying and agree with you.


For the moment I just want to say that from my point of view the thing that irks me the most - and I am rushing and just don't have the time to quote it properly - is something about part of your job being to help the reader to find something worth their time or something like that.


That's probably the problem with the editor reviews for my part - I want to and am quite capable of judging for myself. I allow the writer to be my guide here. I think - maybe wrongly - that as editors your tastes are probably geared towards more 'literary' fiction than mine or like to give that impression.


Stuff like Lord of the Flies nearly destroyed my passion for reading and is probably one of the reasons I did not try my hand at writing until now. A few of the editor reviews seem to scoff at the writer for daring to attempt their hand at writing or maybe that is just my perception and I am wrong.


Miladysa: take your time. I'm going to sleep now (over at my end), but I'm looking forward to your suggestions and ideas.


PS: Actually me (and possibly Gavin) are the odd ones out. The rest of the editors prefer genre fiction, and they write genre fiction too. And that's good, because there are very few literary works on the net.


@snitschke


I guess, then, that we'll have to disagree on that point. From where I stand, the entire purpose of reviewing something is to deliver your own opinion, taking into account your own biases. Acknowledging your biases is important so that readers of your review can come to know you and whether or not you reflect their own tastes. I understand that you score on good writing, three dimensional characters, and plot pacing, but in the end your judgment of those is simply another opinion. Perhaps a more valid one, or perhaps less, but it is still no less subjective and there is no use attempting to cloak it in a veneer of objectivity.


But what I feel the real issue here is that if the opinion you deliver in the body of your review does not match up with the star score assigned, then it makes the reviewing process seem dishonest. I mean, if you can feel that something is pretty blah, but still give it a good score, then the opposite is entirely possible as well. Readers seeking out reviews might not be sure exactly what to believe (or, at least, what you believe) which would introduce an element of doubt in your site. Worst case scenario, granted, but I'm truly of the opinion it is best to tell them what you actually believe in both your text and star score, or else one will undercut the other.


Hi Samazing,


For me, setting the star rating to a review is a rather complicated process. Generally, I try to pick the rating to match the tone of the review, and the review is a balance between personal reaction and an analysis of the overall quality of the piece. I can recognize quality without enjoying a piece, and I can enjoy a piece without it being great writing. As an editor, I've got to weigh more concerns than the average member does. Generally speaking, something with really good writing is not going to get less than 4 stars from me, even if it isn't my kind of story.


For me, there's a bit of play -- a half point, generally -- that I can use to get the rating to match what I perceive to be a fair rating. If the rating is further off than that from the tone of the review, I take it as an indication that the review needs work. It's all subjective, of course, but that doesn't make it easy.


In the end, if you want to know what I really think, read the review. That's the thing I spent the time crafting. The rating is a much lower-resolution version thereof.


Chris.


@ Samazing


Assigning a star score is simplistic because there are a thousand different facets I must consider and then force all of that into stars with vague, subjective definitions. It is impossible to represent everything with a simple picture. For Street, I chose to rate the story based more on the mechanics of the writing than my personal enjoyment because I could not detail what, exactly, I didn't like about it. In my mind, giving Street a rating based on, "Something was missing" seemed unfair based on the quality of his writing. Since the criteria I mentioned above outnumbered my nebulous feeling of "Something's missing", basing the star rating on the substantiated criteria seemed the most logical thing to do.


I don't see how this is dishonest. My thoughts about the story -- both personal enjoyment and writing mechanics -- are in the review to be read by everyone. The star rating still reflects my opinion of his work, just not all of them (for that, indeed, would be impossible).


@ejames


I commented on the article you linked.


I took a look at the two reviews you gave as examples. I am not able to speak for the writers of those reviews but I can tell you that it took me a long time and support before I attempted to write a review. For me personally it was quite a traumatic thing to do. Believe it or not - it was also hard for me to leave my first comment here on the forum and to submit my site in the first place.


What I see in those two reviews is that two members took the time to say something about a story they were/are enjoying and gave a high rating. I do not know the writers of those reviews personally but I do know it took guts to have a go. Their reviews may not be worthy to some minds but they are worthy enough to mine.


I have a very similar short yet sweet 5 star review of my own story - the only 5 star review I have LOL Now you could have used that review as an example. I am not 100% certain of my facts but I suspect that my 5 star review was written by my 10 year old grandaughter who should not even be reading my story in the first place but I am aware she is doing so from my sitemeter statistics. I could give her a lecture for the spelling mistakes but I think they are there to deliberately put me off the trail.


I really like what you are doing here at the WFG but I am begining to think that as Editors you may be more comfortable with having just the guide listings and your own reviews and articles. If you want the member participation then I think you need to let everyone who wants to participate at whatever level they wish to or are capable of.


I'm going to leave it there now or else I think this could turn into the ejames and miladysa saga. It has been fun exchanging viewpoints with you :D