Evaluating Data as Sources

Evaluating data for relevance and credibility is just as important as evaluating any other source. As with other information sources with data there is never a 100% perfect source. You’ll have to make educated guesses (inferences) about whether the data are good enough for your purpose.

To evaluate data, you’ll need to find out how the data were collected. If the data are in another source, such as a book; web page; or newspaper, magazine, or research journal article, evaluate that source in the usual way. If the book or newspaper, magazine, or web page got the data from somewhere else, do the same evaluation of the source from which the book or article got the data. The article, book, or web page should cite where the data came from. If it doesn’t, then that is a black mark against using that data. The data in a research journal article are often the work of the authors of the article. But you’ll want to be sure they provide information about how they collected the data.

If the data are in a research journal article, read the entire article, including the section called Methodology, which tells how the data were collected. Then determine the data’s relevance to your research question by considering such questions as:

• Was the data collected recently enough?
• Is the data cross-sectional (based on information from people at any one time) or longitudinal (based on information from the same people over time)? If one is more appropriate for your research question than the other, is there information that you can still logically infer from this data?
• Were the types of people from whom the data was collected the same type of people your research question addresses? The more representative the study’s sample is of the group your research question addresses, the more confident you can be in using the data to make your argument in your final product.
• Was the data analysis done at the right level for your research question? For instance, it may have been done at the individual, family, business, state, or zip code level. But if that doesn’t relate to your research question, can you still logically make inferences that will help your argument? Here’s an example: Imagine that your research question asks whether participation in high school sports in Minneapolis schools is positively associated with enrolling in college. But the data you are evaluating is analyzed at the state level. So you have data about the whole state of Minnesota’s schools and not Minneapolis in particular. In this case, ask yourself whether there is still an inference you can make from the data.

To evaluate the credibility of the data in a research journal article you have already read, take the steps recommended in Evaluating Sources, plus consider these questions:

• Is the article in a peer-reviewed journal? Look at the journal’s instructions for authors, which are often located on the journal’s website, to see if it talks about peers reviewing the article and asking for changes (revisions) before publishing. If it is a peer-reviewed journal, consider that a plus for the article’s credibility. Being peer-reviewed doesn’t mean it’s perfect; just more likely to be credible.
• Do the authors discuss causation or correlation? Be wary of claims of causation; it is very difficult to determine a causal effect. While research studies often find relationships (correlation) between various variables in the data, this does not equal causation. For instance, let’s return to our example above: If the study of Minneapolis high school students’ sports participation showed a positive correlation between sports participation and college enrollment, the researcher cannot say that participation caused college enrollment. If it were designed to show cause and effect, the study would not have resulted in a correlation. Instead, it would have had to have been designed as an experiment or quasi-experiment, used different statistical analyses, and would have supported or not supported its hypotheses.