WFG Development Log

It’s been radio silence for a couple of days, so I figured I would start a thread where I can post updates as they occur.

There has been a lot of feedback on the Discord, and I am going to start collating some of it here as well, so others may offer opinions.

1 Like

The initial timeline of course got immediately shattered when I discovered that my hope of being able to accept new submissions on a separate website and insert them into the Old!WFG database as a workaround until the new site was ready wouldn’t work.

The second discovery was that TWF is so tightly integrated with WFG, that they pretty much have to be developed at the same time and launched at the same time.

The current plan is to get a basic version of WFG done, and deploy it to a subdomain so we can start gathering feedback. Then use that version of WFG to provide listings for a subdomain version of TWF.

One of the major hurdles I am working on right now is the import of old data. The WordPress site was launched at a time were WordPress’s default database schema was… odd. The current roadblock is that every review, listing, and forum post was stored under a single table called “Posts”. Certain things like listings then had metadata tables with even more information. In order for an import to be successful, we pretty much have to write a script that goes through 26,000 some odd entries, finds the ones we want, links them with the associated user, finds their metadata, and gathers this all up to convert into a completely different database schema.

This will take time, but I am hoping to get a “dev preview” up for people to play with sometimes in the next week.

Suggestion: Would it be possible to make it so that we can be referred to by the work(s) of our choice? That is, I’m currently referred to as “theredsheep, author of Pyrebound.” Now, suppose I post a link to a gag project I made for a laugh, titled “Steve Irwin and the Sweet Kangaroo Lover.” The way the system currently works, or used to, I would be known as “theredsheep, author of Steve Irwin and the Sweet Kangaroo Lover” until I posted some other work of fiction. This sounds stupid, but something similar happened to at least one other author on here.

1 Like

When I get to the review system I will definitely keep this in mind.

Would it be possible to make a two/three tiered approach to the data?

What I mean is, when I migrated Eviscerati.Org from Drupal over to WordPress, I set up the site so that I could start moving over new content immediately, and I’ve been bringing in the old stuff in stages. I was dealing with many different kinds of categories (articles, webfiction, webcomics, podcasts, the occasional YouTube video) so the main issue was keeping the content flexible enough to accommodate everything, and then I started pulling all the old stuff in under the new format (by hand, since the conversion tools available didn’t quite cover all bases.)

For WFG, I think it might be a little easier: if you made old WFG static and provided a link to it, it would serve reasonably well as archives. You could have those of us who are still lurking in the forum re-submit our fiction as a way to test the submission system, and then you can open it up for new submissions.

At that point you’d have a dynamic new system and static archives. Then you can figure out how to either automatically pull in the static stuff, or find volunteers to do it “by hand.” Or you could just leave the static stuff and have people resubmit their fiction in the new engine.

Going further, you could add to the submission form “Is your story currently archived in the static area?” and ask them to link to where it is, and have that serve as a way to fast-track existing archived content.

(This is all theoretically possible, but I recognize it might not necessarily be practical. I’m just throwing out ideas.)

That’s certainly a fallback solution.

I want to try to import the old data first, because one of the major requirements of the new website is to not burn out staff, and any form of hand converting stuff would lead to pretty severe burnout and administrative overhead. (There are ~26,000 posts in the database, spread over listings, reviews, etc)

I also want to keep our current listings so we can port over reviews easily, and not lose data.

I really don’t want to force authors to resubmit their old stuff, beyond re-tagging existing listings (if necessary).

I know we could get a semi-working site up quickly, but I would rather do it the opposite way: new site on subdomain until it is ready to swap in, with the old site serving requests until then. That way we can get feedback, do a swap of the old TWF (as it will be using the API of the new site and won’t particularly care if it is on a subdomain or not) and when we are ready, swap over WFG.

Does that sound good?

1 Like

It sounds great! It just seemed like you were fretting over not being able to move forward as quickly as you were expecting, so I was brainstorming based on past experience.

Today’s progress: Added rate-limiting to submissions, prepped the codebase for eventual open sourcing, added initial Listings sort/filter page, added sort by tag, added find Listings with similar tags, added clickable tags on Listing detail page.

Tomorrow I am hoping to add a simple audit/change log for Listings, and then I am pretty confident in working full time on getting the old data ported over.

Hi! When I clicked on the “forgot password” link and entered my e-mail address, it told me that it found an account under the e-mail address I entered. I’m pretty sure that’s typically considered a security hole, because it allows anyone to find out if an account exists with a particular e-mail address (if you already know the username, it makes guessing username/password combos much easier, as well as knowing that you can spam the person with reset requests). I’ve created and run large-scale, web-based data management services before, and we always made sure that the “forgot password” response would say something like “If an account exists using that e-mail address, you will receive an e-mail with a password reset link.” It’s not as user-friendly, but it’s more secure. Just a suggestion.

This has now been fixed. It was implemented when I set up the forum, but the migration flipped it back. It should no longer reveal email address existence anymore.

Thank you for your feedback!

After a (very) trying day, I have finally successfully (as far as I know) migrated all the data from WFG* into the new system.

Uh, yeah, that asterix, well, there was some fun things that were discovered during this process.

Two ghost reviews which are broken on the main site, reviews without ratings, wildbow’s Twig entry got overwritten back sometime in October and all the reviews attached to it got orphaned, and so many other things that I have been trying to purge from my memory.

Speaking of reviews, I thought I would detail the new review system and how reviews were transferred over from the old system.

Old reviews were:
A checkbox if you wanted to recommend a listing. (optional) (People apparently ended up using this as a superlike, since it attached their username to the listing and put the listing on their page. There are a LOT of 5/5 rating reviews with no recommendation checked.)

A rating from 0 to 5 stars.

A review title and post content. (optional)

After a huge amount of research, I ended up with the following for the new review system:

A review title (optional)

A review body (required)

If you are posting as an author page you own (more on this later)

A “Would you recommend this fiction to others? Yes/No” button (required).

The new system is much simpler, but the means it was arrived at was anything but. 5 star ratings are highly subjective, personal to the person rating, and tend to form a J curve. (The average trends toward 4.5-4.7). With accounts no longer being restricted, I wanted reviews to feature more heavily in ranking calculations, but at the same time limit the ability to “review bomb”. To ensure full transparency, all reviews are public, and there can no longer be ratings without a review.

So, how does the transfer work from old reviews to new?

If your old review recommended the listing, your new one will too (duh). If you rated a listing 3/5 stars or more, your new review will be marked as recommended.

I know this is a major change, and I am hoping to gather feedback. Please note that all the old data is still there, and can be used as needed.

As for other changes, the following has been done:

All user accounts were already imported into the forum, but just in case they have also been imported into the new system. This shouldn’t affect anything, as you still need to log in via the forums, but it helps keep the audit trail unbroken.

If you had ever posted a listing, an Author Page has automatically been generated for you.

A quick note: Author pages are different than user pages. A single user can have multiple author pages. So if you publish under multiple names, you will no longer need multiple accounts to manage your listings.

All listings have been attached to their respective author pages.

If someone other than you submitted your listing, you will need to go through a flagging system to request ownership transfer. This is a manual process (as verification cannot be easily or reliably programmatically achieved) and will be slow upon launch of WFG 2.0. (Thank you in advance for your patience.)

Multiple users can “own” an Author page, so if you co-write with someone else, your combined author page will be editable by both of you, as well as any listings attached to it.

Users can now freely update their listings as needed.

Users can post reviews on listings under an author alias. You can still only have one review per listing, no matter if it is under your username or author page name.

With the data migration complete, I will be focusing on building out the site into a MVP (Minimum Viable Product) that can be fully navigated, then deploying the site to a subdomain for feedback and testing.

I look forward to finally getting something in front of you guys, and hope to have it up tomorrow. But for now… sleep.

1 Like

Well, that took much longer than expected. Had to solve a pretty thorny Auth/SSO issue, but that now (seems) to be working.

I am making the following changes to the migration script:

Change minimum star rating to “recommend” from 3.0 to 3.5, to more accurately reflect the hover message on the old WFG site (3.0 being “Meh”).

Removing workarounds for orphaned reviews. “Twig” has now been reinstated onto the old WFG site (Thanks @cpoirier!) and orphaned reviews should now have found a nice, loving family to belong to.

I am now going to work on re-importing the old dataset into the new framework, so I can have people test the account integration and start working on the visual styling and display of data for feedback.

The long march continues.

2 Likes

First of all, much appreciation to what you are doing, rocode!
Not sure if I should open this can of worms because I don’t want to derail this thread and distract from your good work, but as someone who contributed a lot of those historic reviews and recommendations I do have a thought!..If you are going to equate recommendations with any review over 3.5 stars, doesn’t that make them redundant? To me, the review ratings tried to be an objective measure of the story’s quality while the recommendations were more about personal preference. eg. If someone liked a story I recommended, that could lead them to others I recommend that might also be to their taste. There’s a nuance there that will be lost. I’m not necessarily opposed to a different format going forward, but why not at least leave historic recommendations as they were?

Unfortunately, to leave the old reviews “as they were” requires re-implementing the old system in its entirety, or archiving the old site, and not having any of its user content visible on the new site.

We are using none of the code from the old website. Everything is being rewritten from scratch, often without looking at the old code (because we are no longer based on wordpress, or constrained by wordpress design philosophy).

My choices were to try to port over the old data in some form into a new system, to re-implement the old system in its entirety, or to archive the old site and have the old data not visible at all on the new site.

The first option required, by necessity, some transformation of the old data. The second option did not require any transformation of the old data, but was simply infeasible in terms of implementation and time (it would be a year long project). And the third option, while attractive, violated one of my primary goals in this project: To not invalidate the reviews, listings, and forum posts of the community from 2008 until today.

One possible solution may be to create user submitted “collections” of recommended stories, so you can review as you please, but also publish a list of stories with a specific theme, or a recommendation set.

1 Like

Thanks for the explanation. So, if I understand correctly, what you’re saying is that porting over the old recommendations, as they were, does not fit the new database schema?

Correct. The only way to get them to fit would be to adjust the new schema to reflect the old schema.

But that is only partially the issue. The real issue is: The new system gets rid of star ratings, for a multitude of reasons, and we are converting old star ratings to the new rating system.

We either map the old ratings to new, or archive the old ratings (in which case they would have no interaction with the new system, would not appear on listings as reviews, and would not be counted in ratings/totals).

OK, thanks. I didn’t realize that.

Is this a good thread for bug reports/site issues, or is there somewhere else you’d like to see them?

It is indeed! Please note that if it relates to the current main site ( http://webfictionguide.com ) there is a very limited amount of things that can be fixed.

Will do! It looks like the thing I wanted to report has been resolved. I was getting redirect errors a lot yesterday when I tried to use the forums.webfictionguide.com subdomain. I’m not seeing the same issues today.