I am a big fan of System.Transactions. It is by far one of the coolest additions to the .NET framework. It was what introduced me to the power of ThreadStatic variables (use with caution).
Since being introduced to System.Transactions, I've been using it to create transactional business objects (volatile resources in System.Transactions lingo). This has worked very well for me, however, I recently (read: in the past two weeks) discovered some nuances in implementing IEnlistmentNotification that might not be readily obvious or documented (as far as I could tell).
Nuance 1: Do not throw exceptions from the Prepare(PreparingEnlistment pe) method
If you need to indicate that your resource cannot prepare for the transaction to complete, you must call the ForceRollback(...) method on the PreparingEnlistment object passed into your implementation of Prepare(). If you want to include additional information on why your resource was unable to prepare for the transaction to complete, there is an overload of ForceRollback() that accepts an object that derives from System.Exception. Once all resources have rolled back, the System.Transactions infrastructure will throw a TransactionAbortedException. Any custom exception that you've provided to the ForceRollback() method will surface as the InnerException on the TransactionAbortedException thrown by the System.Transactions infrastructure.
You should NOT throw an exception from Prepare(), and you should wrap all code within the Prepare() method with a try/catch block - the catch block should catch all exceptions and pass them on via the ForceRollback() method. Throwing an exception from Prepare() (or allowing one to bubble up thru Prepare()) will short-circuit the System.Transactions infrastructure from calling Rollback() on enlisted resources - not something you want to happen.
Nuance 2: Rollback() is not called on your resource if Prepare() votes to rollback
If you call ForceRollback() within your implementation of the Prepare() method, the Rollback() method will not be called on your resource. Any clean up that typically occurs in Rollback() will need to occur within Prepare() IF your Prepare() method votes to rollback. For example, if your Prepare() method acquired any locks (e.g. "row locks", "table locks", etcetera) you will likely want to release them in the Prepare() method right before you vote to rollback.
I hope that this is useful to you - I haven't seen this behavior documented anywhere, but that could be my own fault.
I have no doubt that I will find additional implementation nuances when it comes to System.Transactions. When I do, I'll be sure to document them here.