One of the things that every fan of fantasy and science fiction eventually will opine on is the exact and absolutely correct definition of these two; and since there is no agreement, then there will be blood and flames. What follows is not a well-considered and researched proposal for the ultimate settling of this problem, but rather something that popped into my head recently and felt like a possible way of imposing a superficial appearance of order on the teeming chaos of creativity.
Also, giant rambling incoherent post warning.
Now, let us say fantasy is the literature of that which does not exist; this is meant to be something beyond the fictionality of all, er, fiction. Fantasy, purposefully defined this loosely, is the literature of the really nonexistent.
After this definition, three well-formed but hazy-bordered subdivisions of fantasy can be called alternate history, science fiction, and “fantasy proper”. Each is an example of a bigger cluster of sub-categories of fantasy.
1) Alternate history is an example of “past fantasy”: its “that which does not exist” is a different historical event, and its expression is the different course of history after that altered event: time traveling South African racists give AK-47s to the Confederates, and wacky hijinks ensue. Or invasive aliens interrupt WWII; or a single letter doesn’t fall into enemy hands; and things veer off course. (As you can see, the nudge used to diverge things varies quite a bit.)
Alternate history is different from historical fiction, which inserts its fictions into the general flow of history, adds undercurrents, speculates about the dark parts, but does not alter the general flow. (This would place historical fiction very close to conspiracy fiction!) Historical fiction treads in the shadows where our knowledge ends; alternate history walks in the light with a crowbar. (Huge categorization pitfall: Depending on one’s knowledge of history, a dramatization of Däniken could be either.)
(Whether historical fiction is a subcategory of fantasy might be a too muddy question to answer; and on the folly of getting too hung-up on categories, remember the question of whether Pluto is a planet. Giant fuss because people want simple, universal categorization but don’t realize a lot of simplicity mean less universality, and vice versa. And, unlike with planets, one can probably construct/write pathological counterexamples for any proposed universal system of classification and demarcation.)
2) Science fiction is an example of “future fantasy”: speculation about future events, holding that those events will unfold “pretty much the same as the past”; that is, according to the known laws of nature or some superset of theirs.
Science fiction is not the only future fantasy there is — it is limited by (the appearance of) restraint: laws of nature, realism, explanation, certain widespread shared informed guesses of how the future will/can/should be. Some times the restraint is very much there (and this is called hard sf), and it both enhances the story and helps the suspension of disbelief; some times there is Star Wars. (Because, as Scalzi says, bad sf is still sf. And as science gets better answers, some formerly good sf becomes bad sf; but as long as it tries to look like it’s clinging to realism instead of vomiting up some random future scenario it’s sf in my books.)
Also, there’s the funny detail that in addition to people who make a hash out of the science there is, there are people who do justice to their view of science and reality — a view which in turn is wrong wrong wrong.
An example: There still are troglodytes with a magical view of the world — Christians and Muslims and other ha-ha people like that — who can create “science fiction” (or should it be called “personal world reality fiction” or some other hokey neologism?), proceeding from a supposed set of laws of nature that include a God, or “realism as they see it” — and you get the Late, Great Planet Earth and Left Behind. If you define science fiction as “speculating about the future, based on science and reality as the writer understands them“, then Left Behind is science fiction! And if you don’t, you have the problem of dated works: are Jules Verne’s works science fiction anymore because they surely speculate from outdated, false premises?
(Also gives rise to the possible category of “bad science science fiction” — Left Behind being the first inductee. A prizewinner would be — luckily, I don’t know if there is such a thing — a future speculation with homeopathy working as advertised!)
(This definition of science fiction needs the “future events” bit to differentiate itself from right-now realism, but is weak about it. How far is “the future”? A century? A decade? Ten seconds? Halting State is clearly science fiction (and a good one too), but what about Tom Clancy, whose works are widely thought reasonable and realistic near-future speculations? (About “likely” I have no idea.) The more immediate the future is, the more difficult it is to draw any sensible line or to claim this should be science fiction or even fantasy instead of just plain fiction.)
3) Fantasy proper (and this is a bogus category I just made up) is “timeless fantasy”: it discards this world of ours altogether, and conjures up another, with alien histories, gods, inhabitants and laws of nature. It is not speculation about the future, or the past on a different track; it is a world of its own, a clean example of “that which isn’t”.
Tolkien is the most well-known example of this, and there are very, very many others, because though the worldbuilding is time-consuming (provided you put some effort into it), it’s forgiving too: blank history, blank laws of nature, draw whatever you happen to draw, and let imagination instead of research guide you. (And the failures of this can be just as spectacular as those of science fiction; but they’re more failures of internal consistency than of fact.)
Fantasy proper gets confusing when it admits the existence of the real world (meaning some close shade of reality), and turns into a portal fantasy: your everyday everyman has a wardrobe malfunction, and he’s in Narnia, or gets a letter and ends up in Hogwarts — to say nothing of the instances like Pern when the different world of fantasy turns out to be a far-away future planet in some science fictional universe. Tends to happen when the thing being classified is something that thrives on being something different and new. (Well, a significant part of it does.)
So: Fantasy is the literature of that which isn’t.
Alternate history is the literature of what might have been, but wasn’t.
Science fiction is the literature of what could be, but hasn’t been yet.
Fantasy proper is the literature of that which is not related to us.
And that leaves all the numerous other forms and shades of fantasy out: you could say urban fantasy is that which isn’t happening right now, and so on.
This all leaves out the genre of horror — but horror’s not a classification that is in any way related to these. Horror is better defined by what it tries to evoke in the reader, rather than how it treats reality. (And from the above it should be clear that “treating reality” is a slippery approach since it’s the writer doing the treating, and all writers don’t live in the same reality.)
Footnote: Fun as vigorous handwaving like this can be, I’m still partial to the non-overlapping magisteria definition that sf is what might be; fantasy that which cannot be. (And the Star Wars prequels are that which should not be.)
(Sorry. Love Revenge of the Sith, by the way.)