In A Word: A Guide to Coding Open-Ended Survey Comments (September 2012)
There is much contained in a word, a phrase or a comment. Holding onto not only its meaning, but also its tone, undercurrent, or even its insight around the topic—that is the art of coding. It is an art which requires a unique skill and ability to quickly grasp a wide range of survey topics.
DataStar is often contracted to code open-ended questions as part of data processing. What does this mean? When the heart of the research is in respondents' comments regarding what they do or do not like about a new product, their employer, etc., it is often necessary to quantify what the respondent is saying qualitatively. Coding is essentially a categorization process for analysis of open-ended text.
We begin by reviewing verbatim responses for 10-20% of a given sample for each open-ended question to be coded. During this review, we note recurring responses or themes as a series of words and phrases. This becomes the foundation for the "codebook," where each recurring string of phrases or words is assigned a numeric value. Once the numeric values or "codes" are assigned, the code-frame or codebook is ready to be used for actual coding of the responses.
Before beginning to code, we must determine how many codes per response will capture the essence of the comment provided. Typically we code up to three (3) mentions per response, though it can be more, depending on the nature of the question and client preferences.
Next, with codebook in hand, we systematically review each verbatim comment and assign one or more codes to the response. These codes are then captured during the data entry phase along with the closed-ended data.
If we are working with an electronic file of verbatim responses, we start the coding process by sorting the responses alphabetically and adding column(s) that will ultimately hold the coded data. By doing this sort, we can speed up coding many of the single word responses and some of the code-able phrases. The Coder will then have to read through and interpret the remaining responses to assign the appropriate code(s) suited to each response.
Careful balance plays a major role in the coding process. The "art" of coding comes into play when the Coder can read a word or phrase that is not exactly listed as such and manage to resist "stuffing" the response into an inappropriate category. This will sound like an oxymoron, but the skills needed for a good Coder are: good subjective comprehension of what is being said and clean objectivity when selecting the correct code.
Long ago a rule was laid down stating - "When in doubt... Hold out." Very seldom will a codebook be built that works like a charm for every response. Often, a new theme will emerge that deserves its own unique code and thus should be added to the codebook. Our rule of thumb is if three percent (3%) or more of a "new" mention is cropping up, then an additional code should be added to the codebook. A catch-all "all other" code is usually included at the bottom of each question's code-frame to capture those comments that don't fit into any of the codes in the codebook and don't warrant the addition of a new code. We sometimes provide our Clients with a "list out" of what has been coded as an "all other" so that they can see what mentions have been coded as such. We also suggest that our Clients read through many of the verbatim responses to gather good anecdotal material for their report and presentation.
DataStar has years of experience processing paper surveys and flipping through pages to code open-ended questions. We have stream-lined the coding process to make the per-piece expense of coding less cost prohibitive, yet, still, it remains a very manual process. It has been our experience that there is no getting around "human coding." Most text savvy "content analysis" tools that we have explored do little more than give back word or phrase counts with little or no elaboration around that elusive "essence." However, as new technological advances are made, we will continue to explore automated systems which show future promise. Until that time comes, we see coding as a task to be performed in-house by our carefully trained staff.
Turning qualitative responses into quantifiable data can be achieved accurately, objectively and efficiently. The coding process is part art, part science, and with practice, can be utilized effectively and add great value for nearly any survey project.