Recent work on image classification and segmentation has shown that incorporating evidence from multiple scales is an effective strategy. The natural model for sharing evidence at multiple scales is a tree-structured conditional random field.  We develop three refinements to these tree-structured CRFs: First, we introduce a noisy-or factor that is more appropriate than standard pairwise factors.  Second, we show how to allow factors to account for variations in graph structure between images.  Third, we show how to do multi-class learning and inference efficiently, as opposed to one-against-all classifiers.