C.S. Lewis, in a passage I can’t now find, comments that there ‘surprisingness’ is a different thing from ‘being a surprise’. The latter is simply a question of reaction; it’s subjective and, by its nature, can only come once. But ‘surprisingness’ is a quality in the thing itself, the quality of being so structured as to be unexpected.
And as Prof. Lewis says, something that has the quality of surpringness will not lose its appeal the second time. On the contrary, a second go-round will be even better, because knowing what is coming, you will be able to appreciate how it fits into the whole. You will be able to see how this path doesn’t look like it is about to lead you to this view, and how artfully the final effect is concealed until the right moment. Knowing how the murder was done, you’ll pick up on the clues and realize how skillfully they were slipped in at just the right moments, so that had you been paying attention and thinking things through (and allowed for a rather elastic degree of probability), you might have known the truth all along.
This is one reason why I like re-reading detective stories. Some people don’t see a point to this; I enjoy going back and admiring a well-structured logical puzzle, seeing everything in its proper place and catching the inconsistencies that I ought to have noticed the first time. At the moment, I’m revisiting Busman’s Honeymoon (not exactly intentionally; I’d meant to only read the opening for the fun of the wedding accounts, but Dorothy Sayers is very hard to put down), and though the Peter Wimsey novels are most fun for their characters, it’s also delightful to look through the mystery and, once again, see how everything points to one non-obvious solution.
It doesn’t have to be a mystery either; the quality of unexpectedness can simply be a matter of utilizing something previously established in an unexpected way. One example that comes to mind is the end of Aliens, where Ripley comes out in the powerlifter to fight the alien queen. It’s been a good long time since we’ve seen the lifter, and with all that’s happened the average audience member probably forgot all about it. But then she walks out and delivers that fantastic line, kicking off an incredibly cathartic battle where she fights her worst nightmare on equal terms.Again, it remains a fantastically satisfying scene the second or third or tenth time you see it because of the artistry involved.
Now, the key word in all of this is ‘structured’. Any idiot can throw a random twist at you and claim that it’s brilliant because you never saw it coming. “Ho ho! The asshole admiral was the hero all along!” Except that doesn’t fit at all with her previous behavior. It may be surprising in the moment, the first time you watch it, but then if you (for whatever reason) decide to watch the film a second time, you’ll be fuming about how idiotically everyone is acting, knowing what we now know.
The surprise must flow, but flow in an unobtrusive way, not imposing itself upon your consciousness until the correct moment, at which point it not only reveals itself, but illuminates and alters everything that came before it, revealing a different overall effect. Like, for instance, when it’s revealed that two particular characters actually have a very different relationship from what we and the heroine were led to believe, illuminating certain past actions and casting both of them in a completely different light. It’s all there, but it seems to fit into the narrative as presented well enough that you don’t notice the first time. But then, on the second go round, you see the clues and hints, the inconsistencies and omitted details.
Don’t necessarily try to surprise the audience. Or rather, don’t sacrifice good storytelling or characters to try to grasp a surprise. Try rather to give your story that quality of ‘surprisingness’; of re-using and re-contextualizing earlier elements in new, but consistent ways. The reader may or may not be actually surprised, but he will appreciate it as a ‘surprising’ story.