In this paper, we introduce an annotation scheme for body movement based on participants' understandings in interaction. In previous studies of annotation scheme for building a dialogue system, labels which labelers put on a software were determined in advance. In our annotation scheme, we use a form of sentence to describe body movements and set two kinds of hierachical-related tiers; upper tier for interactional practice and lower tier for psysical practice. And we set meta-level tier in

more »

... to observe a relationship between modalities. We believe that it is useful information to detect similar expressions and sequential pattern of multimodal interactions.