This is what I keep coming back to as I try to explain my disappointment in a novel I read recently. It was a very good, very nearly great novel. I don't want to get distracted by that particular book, because what I need to understand first is what matters in an ending.
We begin by setting the scene, meeting the characters. They begin to move around each other in ways that indicate an expanding network of relationships, expanding possibilities. A choice is made: Macbeth kills the king; Romeo and Juliet secretly marry, Faustus signs his contract with Mephistopheles. From that moment of climax the world spins out of control; things fall apart as we feel ourselves careening towards disaster. Finally, the hero attempts to make another choice, one he hopes will right earlier wrongs.
Of course, in the great tragedies, this last gambit fails. In failing, it becomes clear to the protagonist (and to us) how he or she has caused this last calamity. MacDuff announces to MacBeth that he was ripped from the womb. Juliet wakes to find Romeo has poisoned himself. Faustus realizes he has wasted his twenty-four years and tries to repent. That moment, the one that grips your heart in your chest, that is the denouement.
All the great tragic books have a moment like that. Sometimes the structure is different. Humbert Humbert admits to murder in Lolita's second paragraph, while in The Old Man and the Sea Santiago never reflects that his greed in trying to bring back the entire marlin costs him his prize. But, no matter the structure of the narrative, the denouement is there.
And a book like the one I read, deftly written and intricate, that manages only the first four acts (introductions, the expansion of possibilities, the choice and the mounting consequences) to then resolve itself peacefully and civilly in it's final act... I'm afraid that book will always be a disappointment.