intercoder reliability in qualitative research

Conceptually, it is very difficult to separate the factors that are related with relativistic thinking and the factors that are related with a commitment to relativistic thinking. The demonstrations of relativistic thinking and commitment are closely intertwined in the transcript. This choice was based on the fact that a “commitment” means a commitment to the style of relativistic thinking. The code for the Relativistic Thinking-Factor was chosen to code the factors that were determined to be related to both relativistic thinking and commitment to relativistic thinking.

The inter-coder reliability was calculated for the first, second, and third level codes, respectively, resulting in agreement of 100 % for the lead coder and 92.8 % for the external coder (first level codes), 96.8 % for the lead coder and 100 % for the external coder (second level codes), and 75.9 % for the lead coder and 67.2 % for the external coder (third level codes). The inter-coder reliability was calculated to compare the codes by the lead coder and the external coder to the codes agreed upon after the discussion between them. Codes are modified and/or refined during and after the auditing process. An auditing procedure was conducted to discuss the results of the coding. One graduate student researcher who was experienced with qualitative research but not familiar with this research was invited to code these quotes using the initial codebook independently. This procedure resulted in 26 different quotes derived from all four transcripts (some codes link to the same quote). Quotes were chosen arbitrarily by using a random half of the codes’ corresponding quotes. After finishing the coding of the first four transcripts, there were 66 different codes in the initial codebook.

Intercoder reliability in qualitative research