Survey Data: Reliability and Validity? Are they Interchangeable?

Have you found yourself in a discussion about the “reliability” or “validity” of data that comes out of surveys such as course evaluations or employee engagement surveys? Why does the reliability or validity of survey data matter?

Flawed data can lead to wrong conclusions. Particularly when stakes are high, we need to be sure that we are gathering the right data. What this implies is that there are good surveys, and there are bad surveys. Good surveys produce accurate data and critical information, providing salient windows into the core of the topic under exploration. Conversely, bad surveys produce flawed data. In other words, data that is unreliable, irreproducible, or invalid, leading to the wrong conclusions and actions.

The term “survey” reflects a range of research objectives – target populations and sampling frames, recruitment strategies, survey instrument designs, survey administration methods, data processing, and statistical adjustment –to ensure a high-quality survey process and outcome. Given the wide range of options in conducting a survey, it is imperative for the consumer/reader of survey findings to understand the potential for bias as well as the strategies and techniques used for reducing bias, so that appropriate conclusions can be drawn from the data.

Good surveys produce accurate data and critical information, providing salient windows into the core of the topic under exploration. Conversely, bad surveys produce flawed data. In other words, data that are unreliable, irreproducible, or invalid, leading to the wrong conclusions and actions.

There are many elements involved in survey design that affect the quality of data that comes out of a survey – the time and effort it takes for survey respondents to complete a survey, order of questions, number of points on the rating scale, order of question-answer options – to name a few. So, what are the key factors to consider to create surveys that gather high-quality data? While validity and reliability are commonly discussed in the field of psychometrics, it is often assumed that they are present without validation. But we need to be sure that the answer to the question, “Is the data that comes out of this survey reliable and valid to use?” is yes. This is to avoid drawing the wrong conclusions, particularly when stakes are high such as survey results impacting one’s promotion decision or an investment decision on where to spend time and money for improvement.

What are survey reliability and validity? Do they mean the same thing? Even though it is not uncommon to see these two words used interchangeably, they are two very different concepts in the data research field. How are they different? Why does it matter to know the difference?

Survey reliability vs. Survey validity

Validity and reliability are two key factors to consider when developing and testing any survey instrument for use in gathering data. Attention to these considerations helps to ensure the quality of your survey instrument and the data collected for analysis and use.

Imagine that you are designing the employee engagement survey, and you consider asking the question “I am engaged” in the questionnaire.

In assessing the quality of a survey, we must consider the following:

• Are we asking the right question? In other words, to what extent is this question actually measuring what is supposed to be measured? ( i.e., engagement)
• How well (or poorly) will this question evoke the same interpretation to yield the same kind of information? (i.e., a consistent response each time it is asked)

The first question considers the validity, and the second question the reliability. As you can see, they don’t concern the same issue.

Reliability does not imply validity. Survey reliability on its own doesn’t effectuate/establish validity and vice versa.

• A valid measure that is measuring what it is supposed to measure does not necessarily produce consistent responses if the question can be interpreted differently by respondents each time asked. In other words, an employee engagement survey can have high validity but low reliability.
• Also, an employee engagement survey can be designed to have high reliability – consistent responses each time asked – but low validity if the wrong questions are asked. Asking how strongly a respondent agrees to “I am engaged” can be valid, depending largely on your survey objectives. If your survey objective is to get a sense of % of employees who are engaged, this may be a valid question. However, if your survey objective is to identify the attributes that impact employee engagement, the validity of this question is arguably questionable.

Let’s summarize

• Validity looks at the extent to which a survey instrument measures what we want to measure. For example, a survey designed to explore employee engagement, but which actually measures customer satisfaction, would not be considered valid.
• Reliability considers the extent to which the questions used in a survey instrument consistently elicit the same results each time it is asked in the same situation on repeated occasions. Reliability is a statistical measure of how reproducible the survey instrument’s data is. A survey instrument is said to have high reliability if it produces similar results under consistent conditions, and any change would be due to a true change in the attitude, as opposed to changing interpretation (i.e., a measurement error).

Although distinctly different, survey validity and reliability are inextricably linked. Reliability does not imply validity, but it does place a limit on the overall validity.

There are various types of reliability and types of validity and test methods to estimates them. Below are a few good resources.

• Reliability and validity assessment by Carmines and Zeller (1979) is probably one of the most frequently cited books, where they define and discuss the two interlinked concepts.
• Psychological testing: Principles and applications (6th ed.). Upper Saddle River, N.J.: Pearson/Prentice Hall. ISBN 0-13-189172-3.)
• Survey quality indicators can be found in the European Commission Eurostat to help assess the quality of survey data.

The evidence of validity and reliability is a prerequisite to ensure the integrity and quality of a survey instrument, as we draw conclusions from the data that comes out of a survey for proper application. This is the reason why it matters to know the difference between the two.

Did you find this article helpful to you? Question? Feedback? Let us know.