Brad reached out on my Ask me Anything page to say,

“I’m trying to avoid writing the formulaic novel. terrified of it in fact, regardless of how it can put it on the NYT best sellers list. I understand the storygrid, and other methods/tecniques/etc. 
Any suggestions for my nightmare?
thanks.”

That’s a great question, Brad.

My number one suggestion is to write things that fall into the conventions you’re trying to avoid, because the most effective way to break a rule in a way that feels satisfying to the reader is to fully understand the rule you’re breaking. Why does this formula work? What’s satisfying about it? What are its strengths. Approach it at disdain at your peril, because that formula, whatever it may be, has been around longer than you have and will continue to succeed after you’ve gone. Writers write in it unconsciously because as readers they’ve internalized it. Formulas are the fabric of fiction. Save the Cat is a really great introduction to plot structure and mechanics.

Once you know it inside and out, then you can play with it, and that’s when the real fun begins. You know what the reader’s expect, you know how to deliver it, and using the knowledge of both, you can subvert those expectations in a way they never saw coming.

In order to do that, you’re going to first have to define what a formulaic novel is to you. It can mean a few different things to different people. If by formulaic, you mean that it follows a particular plot structure like the three act structure, that’s going to be harder (though not impossible) to avoid.

If you mean that it follows the tenets of a specific genre, spend a lot of time studying the genre in question. Genre is basically just a fancy word that encompasses the reader’s expectations when they pick up a certain type of book. Notable exceptions exist, but if I pick up a mystery, I expect there to be something to solve, clues along the way, and characters attempting to solve those clues. There are certain tropes, characters, key scenes, and tonal expectations that come with that genre. Study those. What elements do you want to remove, what do you want to add, what will you keep? You have to keep something. A mystery without a mystery isn’t a mystery. You’ve already mentioned The Story Grid, an excellent resource for breaking down the parts of a genre.

For the other meaning of formulaic novel, plug and play characters- books are basically indistinguishable from one another set ups, I’d identify those books and take some time to study them. Are they all part of the same genre? If so, is it one you plan to write in? Is it one you read a lot that you might have internalized? Is it your least favorite type of book in the whole world and you find yourself getting irrationally made every time you spot it on the shelf (I know that sounds crazy, but I hold grudges against books, I figure somewhere out there, other people do, too). In those cases, my advice to write it until you understand it stands a hundred fold, because to some extent, you’re going to have to get it out of your system. Every writers first couple of projects are subconscious homages and responses to what they’ve read. So not only will you be mastering your craft and learning the rules so you can break them, you’re also battling your reading and writing demons.

I hope that helps!