Fiction is an illusion. It’s a made-up tale of something that never happened…and it’s the author’s job to get the reader to accept that illusion for the length of the story, however long the story is.
This basic unifying principle tends to get lost a lot, because so much fiction is mimetic – meant to imitate real life on some level. The more closely a piece of fiction imitates real life, the easier it is for the author to lose track of the fact that it’s still an illusion created with words, and moreover, that the illusion is a key aspect of every single component of the story.
Every time a writer breaks the illusion they’ve created, they give the reader a chance to break out of the story. It doesn’t matter what the break is – an inconsistent bit of characterization, an illogical-but-plot-necessary action, an inadequate bit of setup, wooden dialog, or even a line of narrative that sticks out because it’s clunkier (or smoother!) than all the prose surrounding it. Breaking the illusion gives the reader a chance to escape it.
Not every reader will take the chance when it is offered, of course. Some are less sensitive to certain types of illusion-breaks than others; some will ignore or not even notice illusion-breaks in one specific story-component. I have one friend who can happily remain immersed in a story that has gigantic plot-holes, so long as the prose is lovely and the characters charming, and another who will ignore the clunkiest prose as long as the plot hangs together.
That, however, is readers. Writers are another story. While it is a truism that no piece of writing will ever please everyone, most writers would really like to hang on to the maximum possible number of interested readers. (Note that I said interested readers. A writer may be very well aware that his or her stuff appeals only to a very small slice of the total reading public…but even so, they’d really like to grab and hang on to every single reader in that very small slice.)
One of the ways to hang on to readers is to give them very few places to escape the story, and that means maintaining the story-illusion on every level as much as one is capable of. Ideally, one would want to avoid both the clunky prose that would put off my first friend, and the giant plot-holes that would alienate my second. One doesn’t want to provide a really convincing and consistent illusion of a far-future war between the insectoid aliens and genetically-engineered humans, only to have a reader give up on the book because the dialog is wooden or the characterization inconsistent.
An that’s the first mistake a lot of writers make in this regard: they focus on one type of illusion-consistency (usually the worldbuilding or characterization) and forget that the plot, the prose, the dialog, the descriptions, the ways the characters interact, etc. also have to foster the overall story-illusion and be consistent with it (and, perforce, with each other). Writers will throw in really cool, well-worked-out worldbuilding details without stopping to think about how their characters would/could make use of them or what the effect would be on the plot if they did. They’ll work out an intricate and consistent political plot that doesn’t work because the characters they’ve created just wouldn’t do that (or rather, too many readers believe the characters-as-portrayed wouldn’t do it).
The second major error a lot of folks make is that they don’t have a feel or a plan for just what the story-illusion they’re creating is, so when they get to a tricky bit of dialog or plot or worldbuilding, they fall back on reality, even when reality is at odds with the illusion they’re trying to create. But not all fiction is mimetic, not even fiction that’s “present day” (or that was when it was written. Bertie Wooster and Jeeves wouldn’t be nearly as much fun if they weren’t comic exaggerations.)
Unfortunately for writers, reality is where we live. This means that it’s a whole lot easier to spot places where a flaw in the story-illusion also doesn’t match everyday real life than it is to spot places where imitating real life too closely is at odds with the story-illusion the author is trying to create. For instance: I read an SF story once in which the characters had really useful hand-held anti-gravity gadgets that they used for lots of plot-important stuff…but when they walked by the spaceport, the ships were being unloaded by workers using fork lift trucks. There are five or six really easy ways this could have been fixed…if the writer had noticed and/or thought about how a hand-held anti-gravity gadget would logically be used. But she didn’t notice, and I did, and it threw me out of the story.
Training oneself to notice this stuff is not easy. I learned a lot of it from being in a critique group with a bunch of very good writers who differed a lot in the types of illusion-breaking flaws they were sensitive to. Reading a lot of books of different types and differing quality with an eye toward noticing what does and doesn’t work also helps, but in my experience, it’s a lot easier if you have other folks to point insistently at the stuff you yourself don’t notice, until you get to the point where you do start noticing.