Coding the conversation from the movie “12 Angry Men” was both a rewarding and insightful experience. While it was mostly aligned with what I expected, some aspects turned out to be noticeable barriers to accurate observations.
First, I would like to point out that the coding process did not pose a significant challenge. I was able to grasp the main principles and detect the characteristic phrases rather quickly, and closer to the second half of the paper, I could assign a code much faster than during the initial phase. My partners were similarly successful in the activity, which can be derived from the feedback received during the conversation following the initial submission of the paper. While I cannot conclude whether I was successful in mastering the fundamentals, it is a practical certainty that the procedure is less challenging than I expected. The received feedback confirms this suggestion since it indicates no need to resubmit papers and only contains a reminder about the conversation between the partners.
The inter-rater reliability brought up in the conversation requires special attention. As was expected, we had several differences in coding specific phrases. However, two aspects must be pointed out. First, the rate of disagreement was higher than anticipated, with three different opinions in some cases. I attribute it to the novelty of the activity and expect this metric to go down once y partners and I become more experienced on the matter. Second, and, perhaps, more importantly, the final segment of the conversation was subject to much less disagreement than the rest of the text. With few exceptions, the entire scene was unanimously coded as consisting of fight statements. The most likely explanation, in my opinion, is the prominent use of non-verbal devices by the characters in question. As most of them could easily be linked with an aggressive stance, the team coded the segment uniformly. However, I believe that while in this particular situation, it was a helpful detail, in many other instances, such predisposition may open up the possibility of bias. Therefore, such unanimity of statements is to be approached with caution and considered an illustration of the possible weakness of the technique.
Overall, the exercise was helpful in understanding the necessity of procedures meant to ensure inter-rater reliability. The conversation we had after the initial submission revealed several controversial points on some of the phrases and coding suggestions. Most notably, counter-dependent statements were often coded in disagreement between the peers (possibly because they are less intuitive than more familiar fight and flight ones). I also want to bring up the issue of the possibility of bias at this stage. While on most occasions it is necessary to consider the arguments of the peers and adopt the most reasonable point of view, at least on some occasions, such intention will lead to accepting the wrong position for reasons such as the authority of the partner or the perceived lack of competence. Therefore, the increase in concordance from 46% at the beginning to 61% after the conversation between all participants may be at least partially attributed to such effect.
In conclusion, I can say that such exercises are necessary not only to build up the experience and skill of coding but also (and, perhaps, more importantly) to approach the process from a critical standpoint and reflect upon its weak points and the capability to address them by concordance.