Coding Open Ended Responses

Coding Open-Ended Responses: Concerns and Challenges

Procedure | Process Coding Scheme Creation

Procedure | General Coding Scheme Structure

Overarching Theme	Subthemes	Descriptor	Examples	Variable Name in Datafiles

Procedure | Link to Latest Coding Scheme Structure Concerns and Challenges

Procedure | Coding Scheme Application

1. Individual Coding Set-up: Coding Sheet Template
   1. Create an ‘Individual Coding’ folder
   2. For each coder, create a folder labeled with their name that includes two subfolders: (i) one folder that includes coding sheets that need to be coded and (ii) one folder that includes coded sheets. Once a coder has completed a coding sheet from their “to code” folder, they can simply drop it into the “coded” folder

2. Training Coders

   1. Provide the scheme for all coders involved in the project. This scheme should be discussed in detail and all initial questions addressed.
   2. Provide sample data, with which coders can practice the coding scheme. These initial codes should be discussed thoroughly with the coding team, and the exclusion and inclusion process should be detailed. At this stage, the coding scheme is often revised to reflect common questions put to the researcher by coders in order to provide documentation of any decisions made in the early training stages
   3. Importantly, provide space for coders to voice any questions, confusion, concerns about applying the coding scheme

3. Establishing Interrater Reliability

   1. Upon reaching an understanding of the coding scheme, the coders are assigned a small portion of the dataset for coding (e.g., 50 responses) to establish Interrater Reliability
   2. Provide space for coders to voice any questions, confusion, concerns about applying the coding scheme
   3. Calculate IRR (one possible template can be found here). Highlight any discrepancies in coders’ responses
   4. Discuss any inconsistencies, discrepancies, and questions - if necessary, modify the coding scheme to adequately represent the universe of likely responses
   5. If necessary, assign more (of the same) responses to be coded and reconvene to discuss discrepancies and questions
   6. Recalculate IRR (Interrater reliability is often deemed to be satisfactory if agreement extends above 85-90%

4. Individual Coding

   1. Once raters are sufficiently trained, they can be assigned cases to code independently. Ideally, periodic reliability checks throughout the coding process are done to prevent coder drift, which is the tendency for raters to veer away from each other in their interpretations of the coding manual over time

Notes:

• A single student response can be coded for more than one concern.
• We differentiate between students who explicitly mention ‘no concerns’ versus students who skip/do not respond to the question. For the latter, label the response as missing (e.g., NA, -99 etc.)