“Now my own suspicion is, the Universe is not only queerer than we suppose, but queerer than we can suppose.” – J.B.S. Haldane, British biologist.
One of the greatest advantages of science fiction is that it’s the literature of ideas—that it’s not limited in ways that other types of fiction can be, especially those grounded in history or the here-and-now. But even so, it often doesn’t live up to the hype, especially when it comes to extraterrestrials: the hypothetical intelligent natives of other planets.
Too often they’re either exactly like us, or thinly veiled replicas of us with slightly different but recognizable bodies. In some cases, the authors simply use historical human groups (complete with their social mores and rituals) as sources for their aliens, or else create living mimics of a particular stereotype or idea… sometimes not even changing their names much. Often the result is intriguing, sometimes not. Jonathan Swift produced more interesting “aliens” for Gulliver’s Travels than many SF writers do today, and Gulliver stayed right here on Earth.
To be blunt, part of the limitation here is that of human psychology. It’s possible for a thoughtful writer to construct aliens so odd they’re completely incomprehensible; if I recall correctly, Jack Chalker invented (for one of his Well World novels) an alien that resembled a smear of paint on glass and had absolutely nothing in common with humans. In Anvil of Stars, Greg Bear describes a Ship of the Law inhabited by creatures made up of “cords” that form “braids,” which together are sentient—and suggests that they’re only about halfway along the continuum toward the most unusual known form of life compared to humans. At that, they’re too alien for the humans involved to accept, much less understand.
If a reader can’t understand an alien’s motivation or emotions, how can we possibly connect with them? How can we enjoy or learn from a story based on their lives?
The late Stephen Jay Gould pointed out that if evolution were somehow “rewound” and started over, there could be no way we’d end up with the humans again. Indeed, we might not end up with anything we could imagine, especially if that dinosaur killer hadn’t smashed into the Gulf of Mexico at Chicxulub 65 million years ago. Other biologists have challenged his assertion, suggesting that boundary conditions here on Earth would have produced similar lifeforms, though not necessarily a human-like sentient.
After all, there are other creatures here on Earth which seem rather intelligent that aren’t remotely humanlike, including raccoons, corvids (crows, ravens, and their relatives), and octopods. If dinosaurs had survived, something like a velociraptor might have evolved into intelligence, per Harry Harrison’s West of Eden. Any of those evolutionary lineages could have taken hold as sentients instead of hominins (our ancestors). Who knows what kinds of creatures might have evolved? Would there be sentients at all? Impossible to say. Just as it’s impossible to imagine a truly alien species, which is why we try to stick with life as we know it… more or less.
Even the Destroyermen in Taylor Anderson’s excellent series of the same name—WW II sailors on a WW I-era destroyer, transported to an alternate Earth where humans never evolved—encounter somewhat familiar mammalian allies and reptilian enemies. Though neither is human, the allied Lemurians may as well be; and the Grik, their reptilian enemies, embody all the worst in the Axis enemies the Destroyermen escaped, even to the point of allying with Japanese also swept into the new world.
TV/movie aliens have mostly been constrained by budget, though Star Trek has explained their aliens as being almost exclusively humanoid (and able to interbreed) because an early race of lonely humanoid progenitors seeded planets across the galaxy with their DNA. Some writers have made their aliens human because they were lazy (NOT the case for any of the fine writers mentioned here). But the real reason that most aliens in SF are thinly disguised humans is because it makes it easier to understand them, and to use to reflect our own differences.
This may also be why, in recent decades, we’ve seen the rise of human-only galaxies, where humanity is the only spacefaring species but has split into multiple warring factions; see, for example, the works of David Weber. There are few if any aliens, except in the “not from here” sense; all the drama is human, and therefore easily understandable. I also believe that this has (not entirely coincidentally) drawn a new audience to SF. I happen to like the subgenre, myself.
Those of us who read science fiction do so because it’s wonderful and weird, often exploring themes either underutilized or not previously considered. Since there’s nothing new under the sun, per Ecclesiastes, we have to get away from the sun to fully perceive the truly alien. And while that’s worthwhile, be careful with your aliens. In order to tell your story and get your point across, they can’t be so alien that we have no common ground with them, even if that common ground is only war. A few unexplained actions—like leaving a working Stargate in our solar system that allows access to anyone, per some of John Ringo’s and Howard Taylor’s work—is fine; ditto the odd stuff on a TV series like Falling Skies. Too much, though, becomes an exercise in frustration for the reader, who may not continue with you to the payoff at the end.