Dear Al,
No research papers (to my knowledge) have been written yet on multiple testing for the hierarchical linear model, so I can't refer you to any literature.
Use "separate" if you want to get the same results as topTable(). The great advantage of this method is that you'll get the equivalent results regardless of which set of contrasts you test together. This is fine for each contrast but does not do any multiple testing adjustment between contrasts. Another disadvantage is that the raw p-value cutoff for any adjusted p-value threshold can be very different for different contrasts. Use this method if you have only a few contrasts and want to use the simplest method.
"global" is the simplest and obvious choice if you want to do multiple testing across all the probes and all contrasts simultaneously. The raw p-value cutoff is consistent across all contrasts. However you have to be careful that you do not include spurious contrasts in your test set because they will affect the results of the others. There is no theorem which says that "BH" will hold correctly for combinations of negatively correlated contrasts. However simulations suggest that it is pretty safe.
I suggest you avoid "hierarchical" as it is still experimental and its properties are not yet well understood with "BH".
Most multiple testing methods tend to underestimate the number of probes which are simultaneously significant for two or more contrasts. In my experience, "nestedF" gives good results when you want to focus on probes which respond to several different contrasts at once. However this method is virtually undocumentated as I have not found time to write the method up properly. It also provides formal fdr control at the probe level only, not at the contrast level. Do not use it if either of these things bothers you.
Hope this helps
Gordon