Just another Edublogs.org site

Establishing the Trustworthiness of Qualitative and Quantitative Research Studies

Trustworthiness as related to research design indicates that the results of a study and the researcher’s conclusions are credible and therefore have some value to the consumer.  Because qualitative and quantitative research designs are so different they cannot possibly be held to the same criteria for establishing trustworthiness. Qualitative research stresses natural settings, meaning based on verbal narratives, observations, and flexible research design; quantitative emphasizes numerical data, measurements, deductive logic, and experiences.   The same terms and methods are not appropriate to use interchangeably between both research traditions.

Qualitative (Naturalist)

Quantitative (Positivist)

Establishing Credibility

To what extent are the data, analysis, and conclusions accurate and trustworthy? The following represent what a researcher should do to establish credibility during their research.

1. The researchers should engage in activities that increase the probability that their findings will be credible.

Prolonged engagement. Researcher must invest long periods of time within the context/culture they are studying, as qualitative research on any phenomenon cannot occur without also studying the context that surrounds it.  The long observation period also allows the researcher the opportunity to identify any distortions in the data and build trust with the participants. There is a risk of “going native,” where the researcher becomes completely embedded in the culture, over-identify with the participants, and loose perspective.

Persistent observation. The researcher must be aware and open to a variety of influences and factors in order to decide what is relevant. They take in data, tentatively label, and adjust when necessary.

Triangulation.  Essentially researchers must compare their POV to others’ by using a multiple sources  (such as multiple copies of the same source or different sources of the same info), using multiple investigators (working on a team of researchers), and/or using multiple theories.

2. The researcher should expose themselves and their research for peer analysis and review, which serves multiple purposes.

  • The researcher is kept honest by forcing themselves to respond to questions concerning their biases, meanings, and interpretations.
  • Provides a chance to test working hypotheses that are developing in the researchers mind.
  • Allows the researchers the opportunity to develop and test additional steps in the research design.
  • The researcher also has the chance to rid themselves of any feelings or emotions that may be clouding their judgment or good sense.

3. The researcher should perform a negative case analysis in order to refine his or her hypothesis to reflect all (or most) instances of the observed phenomena, essentially eliminating outliers.

4. The researcher should utilize referential adequacy, or the use of recordings to establish a benchmark that findings can be compared to (a record that can later be referred to and reanalyzed to confirm the conclusions).

5. The researcher should submit his or her research for member checks, where the data, interpretations, and conclusions are shown to the original participants for their reaction and confirmation.  This should occur continuously as an assessment of intentionality, a chance to correct errors quickly, and summarization.

Establishing Internal Validity

To what extent are the results credible? Did the intervention produce the result or are extraneous and confounding variables responsible?  In order for a researcher to establish internal validity they must design controls for each type of variable if it is a plausible threat.

To determine if a threat is plausible, researchers must ask themselves:

1. Does the factor influence the dependent variable?

2.  Does the factor differ in amount and intensity across levels of the independent variable?

Possible threats to internal validity include:

History. Threat from uncontrolled events that may occur during the study or before, and affect the dependent variable.

Selection. Threat from the characteristics of participants.

Maturation. Threat from the changing of participants over time.

Pretesting. Threat from the effect of taking the pretest.

Instrumentation. Threat from unreliability or changes in measurement.

Treatment Replications. Threat from insufficient replications of treatments.

Subject Attrition.  The risk of research participants dropping out of the study.

Statistical Regression. Threat from change of extreme scores to those closer to the mean.

Diffusion of Treatment. The threat from treatment affect on one group affecting other groups.

Experimenter Effects. Threat from characteristics of expectations of the researcher.

Subject Effects, Threat from effects of awareness of being a subject, or a participant changing their behavior as a result of their involvement in the study.

Establishing Transferability 

To what extent can the data be appropriately applied to other contexts?

By providing an accurate and detailed description of the context of the study, the researcher has done their part to ensure that others can possibly use the data in other contexts as well.

Establishing External Validity 

To what extent can the results be generalized to other subjects and settings?

To generalize appropriately, usually from a sample to a population, the researcher must be sure to carefully follow specific procedures in conceptualizing and implementing the intervention and use of measures, as well as have record of participants demographic information (age, race, sex SES, etc).

Establishing Dependability

Although no context – therefore no qualitative study – can be totally and completely replicated, to what extent is the study affected by the inherent unreliability of what is being studied?

Part of this question can be answered in that the qualitative researcher admits to the changing nature of their “instrument” – human beings – but it must be embraced due to the ever-evolving nature of their research design and hypotheses.

Because there can be no credibility without dependability, proving research credible may be considered sufficient to prove dependability.

An inquiry audit where an outside agent examines the process of the research, and if deciding it credible attests to its dependability.

Establishing Reliability 

Even though no participant – therefore meaning no quantitative study – will be totally consistent, to what extent are participant scores free from error?

Researchers can empirically determine error through estimates of reliability through several methods.

Stability. Giving the same instrument twice to the same group after a specified amount of time in between.

Equivalence. Administering two forms of the same test to a group and then correlating the two scores.

Equivalence and stability. Giving two forms of the same test to a group after a specified amount of time.

Internal consistency. Indicates the degree to which participants’ answers to items measuring the same trait is consistent.

Agreement.  Giving the same instrument to the group of individuals twice with no time in between.

Establishing Confirmability 

How is the data established to be confirmable?

The confimability audit.  The records created during the process of research are examined, including: raw data (written notes, videotape), data reduction and analysis products (summaries of field notes, condensed notes, working hypotheses, ect.), data reconstruction and synthesis products (themes, definitions, inferences, relationships), process notes (methological notes, trustworthiness notes, audit notes), materials relating to intentions and dispositions (research proposal, personal notes, and expectations), and instrument development information (pilot forms, schedules, etc). The auditor looks at the appropriateness of decisions and methodological changes during the research process, researcher biases, the scope and accountability of data, and if the researcher sought negative as well as positive data.  The overall design of the study is also considered.  If satisfied the auditor writes a final report of their findings (letter of attestation).

Establishing Objectivity 

How is the data established to be objective?

If multiple observers can agree on a phenomenon, their collective judgment is thought to be objective.

Posted by on October 29, 2012 at 5:11 pm | Comments & Trackbacks (1)

One Response to “Establishing the Trustworthiness of Qualitative and Quantitative Research Studies”

  1.   Susanne Says:

    Nice comparison! Well done. :)

Leave a Reply