Validity/Reliability Issues
Am I just seeing what I want to see?
Once the sample has been defined and generated, the next big step is establishing a coding scheme and actually coding the content of the sample. Developing the coding scheme itself --- deciding on what things to look for in the content --- involves a great deal of decision-making. Depending on the nature of the content and how much research has been conducted on your topic in the past, one advisable way to increase the validity of your coding scheme is to consult existing research on the topic. How have other studies coded for race in analyses of children’s books, for example? It is acceptable and useful to draw from other studies’ coding schemes as your develop your own, especially if you intend for your study to contribute to and build on that literature. Coding schemes are developed both by consulting existing literature and through open coding of the data itself. Open coding refers to the process by which you read/view the contents of your sample (or a representative sub-sample) to get a sense for the concepts or patterns that exist within it.
Once your coding scheme and coding rules are established, code a sub-sample of your content (e.g., 15 of your sample of 150 magazine articles) to check how well they “work.” In the case of multiple coders, you will want to see how aligned you are with the other coder(s). If a lot of discrepancies come up or too many coding questions arise, it’s a good indication that your coding scheme and/or rules need refining. Once adjustments have been made, code another sub-set of articles to assess alignment and continue this process until you have reached a satisfactory level of agreement. After coding rules have been confirmed, you will calculate an inter-coder reliability rate, which will eventually be reported in your write-up. If you are the sole coder on the project, you will need to enlist the services of a colleague to code a sub-sample of the content. Whether you are coding on your own or as part of a team, you will need to report an intercoder reliability rate as an indicator of reliability in your write-up.
As with sampling procedures, our most important piece of advice for coding procedures is to be systematic and consistent. Define your codes and coding rules as precisely and exhaustively as possible (see examples of codebooks here) and then adhere to those definitions and rules as carefully as possible while coding each text. Doing so will go a long way to improving the validity and reliability of your study.
Once the sample has been defined and generated, the next big step is establishing a coding scheme and actually coding the content of the sample. Developing the coding scheme itself --- deciding on what things to look for in the content --- involves a great deal of decision-making. Depending on the nature of the content and how much research has been conducted on your topic in the past, one advisable way to increase the validity of your coding scheme is to consult existing research on the topic. How have other studies coded for race in analyses of children’s books, for example? It is acceptable and useful to draw from other studies’ coding schemes as your develop your own, especially if you intend for your study to contribute to and build on that literature. Coding schemes are developed both by consulting existing literature and through open coding of the data itself. Open coding refers to the process by which you read/view the contents of your sample (or a representative sub-sample) to get a sense for the concepts or patterns that exist within it.
Once your coding scheme and coding rules are established, code a sub-sample of your content (e.g., 15 of your sample of 150 magazine articles) to check how well they “work.” In the case of multiple coders, you will want to see how aligned you are with the other coder(s). If a lot of discrepancies come up or too many coding questions arise, it’s a good indication that your coding scheme and/or rules need refining. Once adjustments have been made, code another sub-set of articles to assess alignment and continue this process until you have reached a satisfactory level of agreement. After coding rules have been confirmed, you will calculate an inter-coder reliability rate, which will eventually be reported in your write-up. If you are the sole coder on the project, you will need to enlist the services of a colleague to code a sub-sample of the content. Whether you are coding on your own or as part of a team, you will need to report an intercoder reliability rate as an indicator of reliability in your write-up.
As with sampling procedures, our most important piece of advice for coding procedures is to be systematic and consistent. Define your codes and coding rules as precisely and exhaustively as possible (see examples of codebooks here) and then adhere to those definitions and rules as carefully as possible while coding each text. Doing so will go a long way to improving the validity and reliability of your study.
