V. Establishing Interrater Agreement [chapter]

Understanding Body Movement  
In expression psychology, a number of experiments on the interpretation of expressive behaviour have been conducted. These experiments unanimously reveal that naïve raters may agree in their judgements but that these judgements, however, do not correlate with an objective assessment or even more, that they are wrong. Eisenberg and Reichline (1939) had naïve raters judge a person's dominance by observing the person's gait. The naïve raters agreed more among each other than with the dominance
more » ... ng based upon a questionnaire (Eisenberg, 1937). Similar findings were reported by Mason (1957). 24 untrained raters were asked to judge leadership qualities of 75 candidates based on photographs. As an external criterion, the candidates' leadership qualities were assessed during a field test which required cooperation and problem solving competence by three trained raters, who applied a semi-objective checklist (interrater agreement r = .91). In addition, the candidates assessed each other after the field test. The judgements of the trained raters correlated high with those of the candidates (r = .78), i.e., the external criterion was reliable. The naïve raters, who evaluated the applicants' leadership competence based on photographs, did not agree well (r = .30), and their assessments correlated even less with those of the trained raters and of the candidates (r = .18). In a study by Wallbott (1989), 20 naïve raters were presented without sound the videotaped movement behaviour of psychiatric patients in clinical interviews. The raters were asked to estimate, whether the videotaped behaviour was from an admission interview or from a discharge interview. It turned out that the naïve raters' admission/discharge attributions were totally invalid. Hand movements that were intensive, expansive, soft, round, not nervous, and not coarse (according to descriptive scales) and that correlated with the computer parameters large circumference, long waylength, and high velocity were systematically associated by the raters with the discharge interview. This attribution, however, was wrong. To summarize, the results of the studies by Eisenberg and Reichline (1939), Mason (1957), and Wallbott (1989 evidence that naïve raters deviate from external objective criteria when assessing personality traits or psychopathology based on movement behaviour. In particular, Wallbott demonstrated that the na-Hedda Lausberg -9783653042085 Downloaded from PubFactory at 05/07/2020 03:23:09PM via free access Self-perception is crucial for objective movement behaviour assessment Further evidence for the necessity to establish a regular training in movement analysis derives from research on self-perception and social perception. There is long-standing knowledge in dance and movement therapy that therapists are better in understanding the client's movement behaviour, if they have an objective perception of their own movement behaviour. Empirical support for this practical intuition has been provided across different scientific disciplines. In expression psychology, Wolff (1943) reported that the gait of another person can only be objectively described by a rater whose self-assessment of the own gait is objective. Likewise, the performance in the identification of the own voice correlates positively with that in the identification of other persons' voices (Sackheim
doi:10.3726/978-3-653-04208-5/9 fatcat:hv43cd6mivcetds7cpkb74kaoe