This paper presents a framework for image parsing with multiple label sets. For example, we may want to simultaneously label every image region according to its basiclevel object category (car, building, road, tree, etc.), superordinate category (animal, vehicle, manmade object, natural object, etc.), geometric orientation (horizontal, vertical, etc.), and material (metal, glass, wood, etc.). Some object regions may also be given part names (a car can have wheels, doors, windshield, etc.). We

more »

... mpute co-occurrence statistics between different label types of the same region to capture relationships such as "roads are horizontal," "cars are made of metal," "cars have wheels" but "horses have legs," and so on. By incorporating these constraints into a Markov Random Field inference framework and jointly solving for all the label sets, we are able to improve the classification accuracy for all the label sets at once, achieving a richer form of image understanding.