PG, we've talked in the brennus chatroom some. It seems to me that you're going through a rough time, and it strikes a lot of chords in me because it reminds me of where I was at, in large part, before I found my stride in things - frustrated, giving a serious shot at something and feeling like you're getting nothing for your efforts, even though society has gone out of its way to teach us that effort should merit reward. To write for an extended period of time and feel like your audience isn't growing and you're making no headway. I'm inclined to suspect that your asking this question here is an attempt at unraveling the riddle or figuring out where you're going wrong, based on what you've said in the past.
You mentioned 'even if we've recently had a fight...' and I was on the other side of the people that were arguing with you the other night, just this past week. With that kept in mind, I do want to say I think you're a very intelligent individual, who has a hard time conveying what's going on in your head to the people around you. This extends both to arguments and to certain points in your writing.
In terms of where you're struggling, and I'm saying this based on what others have shared and in terms of what I've read of your serial - I think you ran into the same issues 90% of us do (myself included) and struggled to find your stride for the start of your serial. Many serials, even acclaimed ones on WFG (and mine) run into this. I think you've then run into trouble with connecting with your audience. You move too fast and/or you go off on a tangent, and then you lose people, as Gavin said. It makes for a harder horse to climb atop and a sometimes hard horse to stay atop.
This isn't to say the work itself is bad, but there's a skill to be learned here in grounding things to touch base with the audience and remind them what is going on and, more importantly, why it's going on. Why does X matter to the audience? Coming at things from left field (which you're very good at) is a good thing when the audience is engaged and on board, but when they're already having trouble keeping up with the metaphorical horse they're on, that stuff can knock them from their unsteady seat on the saddle.
I don't know if you've been applying it, but foreshadowing will give them a heads up that X is coming, or at least tell them 'you ~could~ have seen that coming' after they get smacked in the head with it, so they don't think 'well, if I stopped I wouldn't have to get blindsided/smacked anymore'.
The tricky thing is that, in tending to this triangle that involves the author at one point, the text at another, and the audience at a third, it can be difficult to figure out how to unravel the audience's end of things when you don't have a strong core audience, yet when you can't unravel it, you can have trouble getting audience. The only thing you can do is what authors have done for generations - extrapolate from the feedback you do get, experiment, and write mindfully with the intent of helping your audience stay comfortably atop the horse.