I've always disliked the idea of "likable, human" antagonists. It feels manipulative to me, to some extent. "Look, the Dark Lord Nebolousity just destroyed the Eastern Seaboard and plunged countless millions into abject misery to serve the unfathomable will of an evil overlord, but hey, she was bullied once so her motives are understandable". Am I supposed to clap? Be mournful at the horrible consequences of myopic perspectives? Is this some sort of paean to our shared, common humanity?
If you're writing a story that has "villains", the least anyone can do is make them properly villainous ("Cower before me! Mwahahah!"). Else any claims and grounding for moral debate fall flat, lost in some generality about how we're all people deep-down. It doesn't work, or at least, if already claiming that this is a story where "villain" and "hero" apply, it seems to me a little strange to also want the villain to be "human and likable". After all, if one was just writing a story about people with different motives and means that would all go without saying, there'd be no reason to spend extra time dedicated to making the unreasonable seem reasonable. It's only when you're already sort-of-subtly hinting that unreasonable things are happening that it's even required. In a story where Lord Hawkstone tortures prisoners for information and Lord Stonehawk doesn't, and then they have a fight and eventually one dies both are human and their motivations for doing what they did can be discussed. That goes back to Shakespeare. In a story where Lord Hawkstone is evil and tortures prisoners for information because the noble is a villain and Lord Stonehawk let's everyone go without torture, any time spent making Lord Hawkstone seem more reasonable, likable and friendly is time spent undermining the premise that he's apparently also evil. The story is already set up in a such a way as to indicate one set of actions is "good" and the other not, so further time spent fleshing people out is coyly manipulating the reader.
I don't know, it feels as if it ties into the bedrock assumption of the story being told ("good and evil are things!") and then asks me to politely consider "burning orphanages" and "murdering people" somewhat less terrible because the person doing it might also have a family and a hobby, be weak flesh and full faith like me! A kinship bond of our common humanity! Well, I don't burn down orphanages anymore, I kicked that habit so you've already lost my sympathy. If you're writing and find yourself wondering: "Huh, is this making my antagonist too human and likable?" I guess I just wonder why that'd even be a problem, unless you're already setting up a situation where the the actions of the antagonist are not okay and you're using the protagonist to highlight their not-okayness so in order to justify the protagonists continued resistance you have to make them slightly less likable. The readers might lose sympathy for the protagonist if the villain's plans are revealed in full, or their background explorered too deeply? Why is that a problem? In that situation someone is already explicitly making universal judgement about right and wrong, and so, cloaking it in "oh, but they go home to their family and have tea" reads manipulatively to me. Jarring, anyhow - odd.
But then I come from a somewhat different perspective in that most of my antagonist writing has been for various shades of table-top scenarios, which involve setting up antagonistic organizations. Because of the assumption of those systems ("literally evil demons") I don't need to worry so much about "Likable", I just have to worry about "relatable" which strikes me as a better metric in general.
Antagonists with plans, purpose and plots are relatable because they're trying to achieve something, not necessarily because they're by any stretch likable. Ubersoft and Patrick Rochefort hint at that. If you can look at what they're trying to accomplish based on what has gone before, it makes interacting with the villains a lot better. Oh, so the Dark Lord Nebolousity was bullied and because of that she wiped Washington off the map so that she could rule the world from her Gloom-Fortress and monitor everyone 24 / 7 so no one could ever be bullied again.
Well, I don't agree with her motive or her methods, but at least I see where she's coming from. (in tabletop terms: "Oh, so *that's* why the X is doing Y, the fiends"). Lord Hawkstone was desperately in need of the information at hand and short on time? I don't agree with what he did, but I suppose I can see why he had to resort to finger-nail pulling and tickle therapy.
"Humanizing" can easily slip into accidentally "excusing" the terrible atrocities that have taken place ("But in the end, it was for love!"). Making the specific goals and aims for whatever antagonist in question explicit doesn't quite run the same risk, and in the cause of exploring that, you'll almost certainly also add a few characteristics to the creation in question. Not everyone can be in the Doomclock 24/7, at some point they have to take a break and play chess or have tea. Human moment! Now back to plotting the military take-over of the solar system.
I think there's more humanity in that. It doesn't cause a conflict with the implied morality of designating one group of characters as "heroes" and another as "villains", and if someone has a relatable plan for accomplising a set goal it's generally something that will make anyone reading it at least somewhat more sympathetic towards that character. I think most people sympathize with trying to achieve something, so merely explaining the background and the thought-process behind The Nebolon Corporation's #12 Step Programme for Optimizing the World helps make them seem more reasonable without undercutting their evil. If they're all incredibly affable, polite, conscentious, charitable and also psychopathic murderers, I just start to feel confused and wonder if perhaps I'm missing a joke somewhere.
That said, let's talk about "literally demons". At the risk of sounding incredibly silly, Dungeons and Dragons is a pretty cool tool for exploring deontological ethics as it applies to metaphysics. Devils in D&D are composed of actual 100 % Evil. Humans have cells, demons have Chaos and Disdain and Malice-chondria. By definition, they're irredeemable. (Well, in some versions anyway, let's not get into that)
As a story concept, that's actually kind of neat and I think adding the existence of something like that in a fictional universe can be an interesting tool. Well, this is the country of X and over here is the Principality of Y and this little spot of darkness at the corner of the map is objective, absolute Evil. You can't make something like that human. It's like Sauron, the lesser lieutenant of what'shisface, 100% proof evil, out to destroy you, no good. Hates puppies and rainbows. And those fellows can actually be pretty likable because being gleefully, irredeemably inhuman and into malice and the banality of small pain skips neatly past all the underlying ethical problems in thinking about whether antagonist XX is human or right to do what they did.
At least they enjoy their job, which is all-right. That serial killer slasher is just really into body-art. Art with your body, granted, and it's all of the "let's see if I can peel of your skin" varity, but at least they like doing it. That's neither human nor likable, but it is relatable and might make reading about them fun. At that point the conscientous, happy, polite, friendly and neighbourly psychopathic serial killers aren't humanized or near it, so reading about their antics takes on a escapist bent. There's a release valve I think gets flipped when that sort of stuff comes up, because instead of wondering about how many millions one must sacrifice to the meat-grinder to starve away the extinction of the human race ("Were they right to do what they did? WOULD I, knowing what I NOW KNOW perhaps not HAVE DONE THE SAME?! gasp horror shock"), you can just read about a road trip of twisted murderers who bumble along being unpleasant to everybody. It's something so essentially inhuman and horrifying that you can get away with it. At the end of the day, evil, actual capital E Evil is <i>unpleasant as all hell</i>, so humanizing those who do it too much makes the sheer banal atrocity of things too readily apparent.
. . . So I guess my response is "I don't try to" or "Gleeful serial killer".
But I guess also, like, having someone crack a joke might help.