14 - burns

14 Mixed Methods Research Jennifer R. Gray Quantitative research and qualitative research have different philosophical foundations. Because of these differences in philosophy, researchers do not always agree on the best approach with which to address a research problem. The convergence of technology with health disparities and the complexity of the healthcare system have given rise to several research problems that cannot be answered completely with either type of research (Morgan, 2014; Sadan, 2014; Shneerson & Gale, 2015; van Griensven, Moore, & Hall, 2014). As a result, researchers combine quantitative and qualitative designs into one study, with increasing frequency, using the methodology called mixed methods research (Creswell, 2014; 2015). Using mixed methods offers researchers the ability to use the strengths of both qualitative and quantitative research designs (Creswell, 2015) to answer different stages or parts of a complex research question. Although some research experts (Munhall, 2012) have argued that using two qualitative methodologies in a single study is mixed methods research, for this chapter we will be describing only designs in which quantitative and qualitative methods are combined. This chapter begins with a description of the philosophical foundation of mixed methods research and continues with descriptions of three mixed methods study designs, with an example of a published study for each type. The challenges of conducting a mixed methods enquiry will be discussed, followed by criteria by which mixed methods studies can be evaluated. Philosophical Foundations The philosophical underpinnings of mixed methods research and the paradigms that best fit these methods continue to evolve. At the foundation of the differences between qualitative and quantitative studies are philosophical differences regarding the question, “What is truth?” A philosophy's ontology (What is? or What is true?) shapes the epistemology (how we can know the truth), that then influences the methodology (research design) (Morgan, 2014). Over the last few years, many researchers have departed from the idea that one paradigm or one research strategy is superior, and instead have taken the position that the search for knowledge requires the use of all available strategies. Researchers who hold these views and seek answers using mixed methods may have exchanged the dichotomy of positivism and constructivism for the “epistemological middle ground” of pragmatism (Yardley & Bishop, 2015, p. 1). However, the interpretations of what pragmatism is, as applied to mixed methods research, have differed (Bishop, 2015). For our purposes, pragmatism refers to the researcher's consideration of the research question and the knowledge needed for the discipline (desired outcome) before selecting a methodology. The desired outcome guides the selection of a methodology that is most likely to address questions within a problem area (Florczak, 2014; Morgan, 2014). As discussed in previous chapters, the process of developing a study design is iterative and reflexive. Decisions are made tentatively about the question and the design and then reconsidered as each phase is developed. Because an in-depth analysis of pragmatism as a philosophy is beyond the scope of this chapter, we are basing our discussion on the goal of pragmatism, which is solving the problem by “choosing the appropriate design for the research aim” (Yardley & Bishop, 2015, p. 2). With mixed methods designs, the researcher can allow the

strengths of one method to compensate for the possible limitations of the other (Creswell, 2015). Stated in a more positive way, mixed methods research allows the strengths of each method to interact in a complementary way with the other. Overview of Mixed Methods Designs The focus on problem-solving or answering the research question means that a mixed methods research design is selected based on study purpose, timing of the quantitative and qualitative elements, and emphasis on one element over the other. Table 14-1 provides a description of mixed methods designs classified by the researcher's reason for combining methods. The purpose of combining two methods may result in a classification based on the order in which quantitative and qualitative elements of the study are implemented (Table 14-2). Another way to label mixed methods designs is according to which element is emphasized. In this classification, the emphasized element is noted in uppercase letters (QUANT or QUAL) and the other element in lowercase font (quant or qual). Table 14-3 provides an overview of this classification. TABLE 14-1 Mixed Methods Classified by Purpose Label Description Exploratory Qualitative methods are used to explore a new topic, followed by quantitative methods that measure aspects of what was learned qualitatively Explanatory Quantitative methods are used to establish evidence related to incidence, relationship, or causation. Then qualitative methods provide a more robust explanatory description of the human experience aspect of the quantitative results. Transformative Quantitative and qualitative methods are used with a community-based research team to address a social problem in the community. Advocacy Quantitative and qualitative methods are used, guided by feminism, disability theory, race/ethnicity theory, or other approach to providing information to raise awareness of the needs of a specific group; aspects of advocacy research may overlap with transformative designs. Data from Creswell, J. W. (2015). A concise introduction to mixed methods research. Los Angeles, CA: Sage; and Bishop, F. (2015). Using mixed methods research designs in health psychology: An illustrative discussion from a pragmatist perspective. British Journal of Health Psychology, 20(1), 5–20. TABLE 14-2 Typology of Mixed Methods Designs Based on Timing of Quantitative and Qualitative Elements Label Description Sequential Either the quantitative or the qualitative phase may be implemented first. Results from the first phase of the study are used to inform the specific methods of the second phase. Concurrent Qualitative and quantitative elements are implemented at the same time through the study. Findings are integrated at interpretation. Data from Creswell, J. W. (2015). A concise introduction to mixed methods research. Los Angeles, CA: Sage; and Bishop, F. (2015). Using mixed methods research designs in health psychology: An illustrative discussion from a pragmatist perspective. British Journal of Health Psychology, 20(1), 5–20. TABLE 14-3

of the phenomenon of interest. In additive studies, data may be collected sequentially but could just as easily be collected concurrently, because integration of all data occurs during analysis. Mixed methods studies in which data are collected concurrently are called parallel designs by some research experts (Creswell, 2014, 2015), because convergence does not occur until interpretation. When convergence occurs at interpretation, each phase could stand alone as a separate study and may be published separately (Morgan, 2014). Concurrent mixed methods designs can also have multiple points of convergence with both types of data being examined throughout data collection and analysis. In this chapter, models of the three mixed methods approaches and examples of each are provided to expand your understanding of these designs. Exploratory Sequential Designs The exploratory sequential design begins with collection and analysis of qualitative data, followed by collection of quantitative data. Often, findings of the qualitative data analysis are used to design the quantitative phase (Figure 14-1). This approach may be used to design a quantitative tool (Morgan, 2014). For example, focus groups may be conducted with members of a target population and items for the quantitative tool developed using phrases and content generated qualitatively. Another reason to use this strategy is to collect data about patients' perspectives concerning an issue or problem, so that their point of view is represented. With this input, an intervention can be developed or refined, incorporating the patients' perspectives about the intervention. An example would be a research team planning to implement an educational intervention and seeking input from members of the target population to gain the patient's perspective concerning the content to be taught. Morgan (2014) noted also that qualitative findings may generate hypotheses for the quantitative phase. FIGURE 14-1 Exploratory sequential mixed methods. Exploratory sequential designs may be selected for reasons other than shaping of quantitative methods by qualitative findings. Exploratory sequential strategies also may be indicated when a topic has not been studied previously, and qualitative data are collected first so that participants will not be biased by the content of the quantitative instruments. Ladegard and Gjerde (2014) provide an example of an exploratory sequential study in which the qualitative findings along with the literature were used to determine the hypotheses and outcomes of a theory-based leadership coaching intervention. “A two-phase exploratory sequential design (Creswell & Clark, 2011) was chosen to address different research questions: What generic outcome criteria should be used to assess the effect of leadership coaching? Does leadership coaching have a positive effect on these outcome criteria? To what extent do differences in facilitative coach behavior influence this effect? An additional reason for choosing this research design was that it enables a more comprehensive account of leadership as a leadership development tool.” (Ladegard & Gjerde, 2014, pp. 632, 635) The qualitative phase of the study was a focus group to address the first research question. Through the focus group with five experienced leadership coaches, Lardegard and Gjerde (2014) identified the outcome to be assessed for the quantitative phase of the study. From the qualitative findings, the researchers integrated existing theory into two hypotheses. “Two valuable and appropriate outcome criteria for evaluating coaching effectiveness stood out from the focus group discussion: confidence in one's ability to be an effective leader, and

confidence in subordinates' ability to take on responsibility.” (Ladegard & Gjerde, 2014, pp. 632, 635) Ladegard and Gjerde (2014) placed their qualitative findings into a theoretical context and recognized that confidence in one's ability to be an effective leader was the same concept as self-efficacy. The researchers examined the literature related to self-efficacy in leadership roles and, based on their review, hypothesized that leadership coaching would “positively influence leader role-efficacy” (Ladegard & Gjerde, 2014, p. 636). The relational aspects of the leadership role had been articulated clearly in the literature, allowing the researchers to identify the concept to be measured as “trust in subordinates” (p. 636). The second hypothesis was that leadership coaching would positively “influence leaders' trust in subordinates” (p. 636). Based on their qualitative findings and examination of the literature, Ladegard and Gjerde (2014) proposed three additional hypotheses. “Hypothesis 3. A leader's increased trust in his/her subordinates is associated with (a) an increase in the subordinates' psychological empowerment and (b) a decrease in their turnover intentions … Hypothesis 4. Facilitative coach behavior will affect leader role-efficacy positively… Hypothesis 5. Facilitative coach behavior will affect trust in subordinates positively.” (Ladegard & Gjerde, 2014, pp. 637–638) Ladegard and Gjerde (2014) described the quantitative portion of their study as a field experiment. They described the sampling and the intervention given to the treatment group. “The second part of this study was a field experiment chosen to test the propositions and hypotheses developed in the first part of the study. The objective was to reveal the effect of coaching on LRE [leadership role-efficacy] and LTS [leader's trust in subordinates] compared with a control group (between-group analysis) and whether changes in trust had any effect on subordinates, and to test whether facilitative coach behavior would predict variation in the two leader outcome variables (within-group analysis). We collaborated with a small coaching company that invited coaches from their network into the project … The leader questionnaire developed during the first part of the study was distributed to the 34 participants one week before the coaching sessions started … a follow-up questionnaire was sent to the 30 participants who replied in the first round. Of these, six did not respond, and the final sample included 24 participating leaders, which represents a response rate of 73% … From the participating organizations, we received 192 email addresses to subordinates, to which we distributed a questionnaire at the same points of time as we did to the leaders. We then matched the subordinates to their leaders, a process that shrank the sample considerably … The resulting final sample of subordinates comprised 80 respondents, of which 63 belonged to the coaching group of leaders. The number of subordinates per leader in the final sample ranged from two to seven, with an average of 2.7 per leader.” (Ladegard & Gjerde, 2014, p. 638) The results of the quantitative data analysis supported all five hypotheses. Ladegard and Gjerde (2014) noted the practical and theoretical implications of their findings, as well as the study limitations. “Our study adds to the knowledge base of both formative and summative evaluation, and argues that leadership coaching is a valuable leadership development tool. The strength of our study lies in our use of a mixed methods design combining qualitative and quantitative methods, providing us with opportunities for expansion and development. Our combination of methods and data sources should give a more complete picture of the effects of leadership

more manageable and less restrictive. The consequences of less than strict adherence caused uncomfortable symptoms and complications, some resulting in hospitalizations. After the first 6 months, “participants began to secretly experiment with an easy-going approach to adherence” (Lam et al. 2014, p. 912) and developed their own adherence profile, which the researchers labeled as sustained adherence. As they experimented, the participants worked through a process of letting some aspects of the regimen “slip” followed by monitoring the effects of the change. The participants made “continuous adjustments to live as normal a life as possible” (Lam et al., 2014, p. 912). This phase lasted 3 to 5 years. Long-term adherence emerged as the participants assimilated to a new way of life that became normal (Lam et al., 2014, p. 914). They selectively made modifications that had fewer negative consequences by knowing their physiological limits. The dynamic process of adherence emerged from the qualitative data because the selected participants had been maximally diverse: male and female, different ages, and on dialysis for different lengths of time. The researchers selected this type of sample because of the results of the quantitative phase of the overall study. Convergent Concurrent Designs The convergent concurrent design is a more familiar approach to researchers. This type of design is selected when a researcher wishes to use quantitative and qualitative methods in an attempt to confirm, cross-validate, or corroborate findings within a single study, using a single sample. Convergent concurrent designs generally use separate quantitative and qualitative methods as a mechanism to allow the strengths of the two methods to complement each other. Therefore, quantitative and qualitative data collection processes are conducted concurrently. This strategy usually integrates the results of the two methods during the interpretation phase, and convergence strengthens the knowledge claims, whereas the lack of convergence identifies areas for future studies or theory development (Figure 14-3). Great researcher effort and expertise are needed to study a phenomenon with two methods. Because two different methods are employed, researchers are challenged with the difficulty of comparing the study results from each arm of the study and determining the overriding findings. It is still unclear how to best resolve discrepancies in findings between methods (Creswell, 2014). FIGURE 14-3 Convergent concurrent mixed methods. Njie-Carr (2014) conducted a convergent concurrent study on the topic of interpersonal violence (IPV) with African American (AA) male perpetrators and AA women who were HIV- infected and had experienced or been threatened with IPV in the past 12 months. Njie-Carr described the research problem as being the need for gender-specific interventions to decrease women's vulnerability to IPV and the lack of research on “men's perceptions of their roles in violence against women” (p. 376). Especially noted was the lack of a concurrent approach to study this common problem. The researcher argued that female and male perspectives were needed to develop “effective and sustainable prevention interventions tailored to the unique needs of AA women who are survivors of IPV” (Njie-Carr, 2014, p. 377). To obtain a multifaceted view of IPV, Njie-Carr identified one study purpose related to factors of IPV in HIV-infected women, another purpose to explore the self-perceptions of abusers' roles as perpetrators of IPV, and a final purpose to determine the implications of triangulating the data.

“… IPV is a critical component of HIV risk and infection … integrating information gained from understanding male perpetrators' roles in propagating violence against women is critically needed to ensure effective, culturally relevant, and sustainable interventions.” (Njie-Carr, 2014, p. 378) Njie-Carr (2014) used Fishbein's (2000) integrative model as a conceptual framework for the study. The integrative model combines concepts of the theories of planned behavior, health belief, and social cognitive theory. The integrative model itself offered multiple perspectives that supported the various aspects of the study design. Figure 14-4 is a diagram of the study design that Njie-Carr (2014) provided in the article. In the diagram, the quantitative and qualitative arms of the study are identified as remaining separate until results were obtained from each, followed by triangulation of the integrated results and finally critical interpretation of those triangulated results. Triangulation is a metaphor taken from navigating ships and surveying land. In these fields, a location is determined by obtaining measurements from two perspectives. The point of intersection between the two perspectives determines the location of a distant object. In this study, triangulation was the process used to integrate data from two samples (men and women) and two methodologies (quantitative and qualitative). FIGURE 14-4 SRPS, Sexual Relationships Power Scale; ABI, Abusive Behaviors Inventory; HAKABPQ, HIV/AIDS Knowledge, Attitudes, and Beliefs Patient Questionnaire. Study design. (Adapted from Njie-Carr, V. [2014]. Violence experiences among HIV-infected women and perceptions of male perpetrators' roles: A concurrent mixed method study. Journal of the Association of Nurses in AIDS Care, 25[5], 379.) “A concurrent Mixed Method study design was used …to adequately capture multiple dimensions of male and female participant experiences by comparing and contrasting qualitative and quantitative results. The qualitative component was guided by Giorgi's method. This phenomenological descriptive approach was thought to be appropriate because it would help gain a better understanding of AA women's lived experiences of abuse and AA men's perceptions of their roles as perpetrators of violence (Dowling & Cooney, 2012) … In this study, it was important to capture unique contributions of each methodological approach in the context of the participants' cultural and social relationship experiences in order to triangulate the findings.” (Njie-Carr, 2014, p. 378) Njie-Carr (2014) specified inclusion and exclusion criteria for study participants. Quantitative and qualitative data were collected from “15 AA male and 15 AA female participants” who were recruited from different sites (Njie-Carr, 2014, p. 377). The women were recruited from the clinic where they received HIV care. The men had been arrested for domestic abuse and mandated by the court to attend a rehabilitation program that focused on developing their skills in anger management and in conflict management. The men in the study were in a situation in which signing an informed consent for a study on this topic could be viewed as an admission of guilt, so they provided verbal consent. The women signed consent forms. “To ensure consistency across the research team (project investigator and research assistants), a data collection guide was included as a cover sheet that itemized the sequence of activities during the data collection process: (a) introductions and brief overview of the study, (b) consent with either a signed form (female) or verbal agreement (male), (c) personal data form/review of

education (r = .743, p ≤ 0.001), physical abuse and social beliefs (r = .718, p = 0.003), and psychological and physical abuse (r = .845, p ≤ 0.001). Expected negative relationships were also found in that higher levels of psychological abuse were linked to lower levels of control in their dyadic relationships (r = −.750, p ≤ 0.001). An unexpected finding was that strong social support had a statistically significant positive relationship with high incidence of psychological abuse (r = .718, p = 0.003) and high incidence of physical abuse (r = .718, p = 0.003). Njie-Carr (2014) explained this by noting that women who are being abused may be more likely to seek support from their networks. The triangulation of Njie-Carr's (2014) quantitative and qualitative data did not occur until both sets of data were analyzed and interpreted. The researchers first triangulated the qualitative results for the women and men and provided a side-by-side table with themes and exemplars from either group. “When female and male data sources were triangulated, data convergence was noted, with similar themes expressed by male and female participants. Both groups shared the perception that males dominated relationships, resulting in power imbalances … a similar theme was patriarchal ideology and the need to control and institute power … most of the male and female participants reported childhood abuse. When asked how their experiences as children impacted adulthood, participants reported that negative childhood experiences might have resulted in the use of substances and alcohol, and for males being abusive to their female partners.” (Njie- Carr, 2014, pp. 384, 386) Both groups also identified what they believed could be done to prevent abuse in the future. Men and women provided different views, however, of the motivations for abusing women. “Female participants noted that a partner's level of education, inability to deal with stress, and drugs may have contributed to her vulnerability to abusive experiences. Male participants reported that they were stressed and frustrated in their efforts to make a living, which resulted in abusive tendencies.” (Njie-Carr, 2014, p. 386) Triangulation resulted in convergence across quantitative and qualitative results. The convergence was expected because the items on the quantitative tools guided the development of the interview questions. “Specifically, the contribution of relationship power on the psychological and physical abuse experiences of female participants, as noted in the quantitative analyses, were significant. Furthermore, similar findings were found with substance and alcohol abuse, childhood abuse, and increased risk for HIV infection from abusive experiences. These results demonstrated that variables and themes were cross-validated by using two data sources and two methodological approaches.” (Njie-Carr, 2014, p. 386) One limitation of the study noted by the researcher was “the small sample size for the quantitative component” (Njie-Carr, 2014, p. 389). Njie-Carr (2014, p. 389) did make the case that her study generated “important preliminary evidence.” The researcher was committed to triangulating the results in parallel, thus requiring use of the same sample for both arms of the study. Although the modest sample size increased the risk of Type II error due to low power, conducting the qualitative analysis with data from a larger sample would have made the study unwieldy and likely unfeasible. Nonetheless, statistical significance was achieved for the quantitative tests, implying that the small sample size, despite the author's observation, was not a true limitation and not representative of Type II error. Other limitations were the use of

self-report instruments and the researcher's lack of access to the male participants' medical records to ascertain their HIV status. The low reliability of the Power Scale for the men’s group indicated an unacceptable level of measurement error, making these data uninterpretable. Self- report instruments may produce inaccurate data due to social desirability, but self-report may be the only way to operationalize the relevant concepts. Implications for future research and health services were identified. “Interviewing females and their partners as a dyad may have provided a stronger methodological approach, but concerns for the women's safety precluded undertaking such a design in this study … additional research studies identifying contextual and structural causal pathways are needed to clarify critical factors that substantially contribute to HIV infection in the context of IPV … AA female participants reported their hesitancy to access medical care and treatment as a result of negative experiences with healthcare providers. This finding shows the need to educate healthcare workers about effective approaches to care for women survivors of violence.” (Njie-Carr, 2014, p. 389) The extensive data collected from each person, the triangulation of findings across the different groups, and the types of data obtained resulted in a robust study. As noted, the sample size was small, but the findings represent a solid foundation for additional studies by this researcher and others interested in the topic of IPV. The study involved concurrent data collection, but integration across data did not occur until the analysis and interpretation of each type of data were completed. Other concurrent convergence studies may show more evidence of data integration during the data collection and analysis such as Goldman and Little's (2015) study of Maasai women's empowerment in Northern Tanzania. Challenges of Mixed Methods Designs Combining Quantitative and Qualitative Data Limited guidance is available concerning how to combine data that are collected using two different research approaches (Östlund, Kidd, Wengström, & Rowa-Dewar, 2011). Historically, methodological triangulation (Denzin, 1970) was what mixed methods studies were first called. However, the process whereby integration of findings occurred was not well defined. In research, triangulation may be the use of more than one research design or multiple sources of data, to allow the researcher to approximate “truth” more precisely. Figure 14-5 displays triangulation as simple convergence. Östlund et al. (2011) describe triangulation as empirical findings integrated into one theoretical proposition, with the triangulation occurring between the grounded or empirical findings and the more abstract or theoretical implications. Figure 14-6 is a visual representation of empirical-theoretical triangulation. The authors also provided diagrams of other types of integration of data between the empirical findings and theoretical propositions. Östlund et al. (2011) describe using theoretical propositions to guide the development of mixed methods studies and seeking convergence between the empirical findings and the theory. In Figure 14-6, the arrows down from the theoretical level toward the empirical indicate theory concepts and propositions guiding the study design. The arrows from the empirical to the theoretical indicate the findings being integrated at the theory level. FIGURE 14-5 Triangulation with convergence.

Convergent concurrent Identify similarities (convergence) and difference (divergence) between the two types of data Matrix/graph of points of convergence and divergence Matrix/graph: x-axis is the quantitative findings by question or variable; y-axis is the themes or qualitative findings. Where findings converged, mark the point with a plus sign; where findings diverged, mark the point with a negative sign. Adapted from Creswell, J. W. (2015). A concise introduction to mixed methods research. Los Angeles, CA: Sage. TABLE 14-6 Example Display for an Exploratory Sequential Study: Developing an Instrument Quotation from a Participant Resulting Item on Instrument (Respondents Select Five Options from Strongly Disagree [1] to Strongly Agree [5]) “When I looked in the mirror and saw how fat I looked, I knew I had to stop eating junk food and eat healthy food.” My appearance motivates me to eat healthier. “Some of my friends are real health nuts and it is easier to exercise and eat right around them. Other of my friends think exercising is texting their friends.” My health-related behaviors are influenced by whom I hang out with. “I tried going to the gym and there wasn't anyone my age who was there. The music they used during classes was really old-school.” I want to exercise in a safe place with other people my age. When I exercise, I want to listen to my favorite music. “I have a job at a fast-food restaurant. I can eat for free, but there isn't much on the menu that is healthy. I can't afford to bring fruit and healthier snacks.” I eat healthier when in a place with many healthy foods on the menu. The cost influences my food choices. Data from a mythical study to develop an instrument to measure “Intent to Change Health Behaviors among Adolescents.” TABLE 14-7 Example Display for an Exploratory Sequential Study: Expanding Perspectives Quotation From a Participant Related Quantitative Finding “When I looked in the mirror and saw how fat I looked, I knew I had to stop eating junk food and eat healthy food.” M = 4.5 (SD = 0.8) on the Body Image Scale r = 0.4 (p = 0.001) between body image and healthy food choices “Some of my friends are real health nuts and it is easier to exercise and eat right around them. Other of my friends think exercising is texting their friends.” r = −0.28 (p = 0.01) between sensitivity to peer pressure and healthy food choices “I tried going to the gym and there wasn't anyone my age who was there. The music they used during classes was really old-school.”Response to open-ended question about reasons for not exercising: “No gyms where my age goes” “I have a job at a fast-food restaurant. I can eat for free, but there isn't much on the menu that is healthy. I can't afford to bring fruit and healthier snacks.”Subjects with lower incomes scored lower on Healthy Food Choice Scale than subjects with higher incomes did (t = 8.3, df = 1, p = 0.05).

Data from a mythical study to provide an expanded perspective on the intent of adolescents to change their health behaviors. TABLE 14-8 Example Display for an Explanatory Sequential Study: Follow-Up Results Joint Display Quantitative Results Qualitative Results Integration Low scores on self-efficacy related to healthy eating “I never know what to eat at a party.” “I usually eat what everyone else is eating.” Lack of knowledge may contribute to low self-efficacy related to healthy eating. Significant difference in knowledge of healthy foods between adolescents with higher incomes and adolescents with lower incomes “There is no grocery store in my neighborhood, only a convenience store on the corner.” “I've read about nutritious fruits like kiwi and cantaloupe but I don't even know what they are. No one eats that kind of thing where I live.” Adolescents living in lower income neighborhoods may have limited access and exposure to healthy foods. Integration of data from a mythical study to explain adolescents' intent to change their health behaviors using a mixed methods study. TABLE 14-9 Example Display for a Convergent Concurrent Study: Matrix of Interpretation of Convergence and Divergence Qualitative Themes QUANTITATIVE FINDINGS Body Image Self-Efficacy Knowledge Environment Behaviors Desire to fit in Being accepted in my neighborhood Inner beauty Positive view of self Knowing I can do it Strong belief in self Healthy behaviors require commitment Access to healthy foods Without access, hard to know Neighborhood makes a difference “Cool” place to exercise No mirrors but great music Easier to exercise in an adolescent-friendly place Note: Cells contain findings that result from the integration of that theme and variable. Integration of data from a mythical study to explain adolescents' intent to change their health behaviors using a mixed methods study. TABLE 14-10 Example Display for a Convergent Concurrent Study: Matrix Graph of Points of Convergence and Divergence Qualitative Themes Desire to fit in (−) Inner beauty + (−) + Knowing I can do it + + + Access to healthy foods (−) (−) + + “Cool” place to exercise (−) + Body image Self-efficacy Knowledge Environment Behaviors Quantitative Results

The quality standards by which to appraise mixed methods designs continue to evolve (Creswell, 2015). Pluye, Gagnon, Griffiths, and Johnson-Lafleur (2009) conducted a systematic review of the literature to identify or develop quality standards for mixed methods reviews. Their conclusion was that each component of a mixed methods study could be appraised separately followed by a three-question assessment of the quality of the data integration. The Office of Behavioral and Social Science Research at the National Institutes of Health (NIH) convened a panel of experts to develop best practices for mixed methods research (Creswell, Klassen, Clark, & Smith, 2011). Part of the panel's charge was to identify criteria by which applications for NIH funding could be evaluated. For this text, we have synthesized standards across sources, resulting in a concise set of quality standards for mixed methods research (Table 14-11). TABLE 14-11 Criteria for Critically Appraising Mixed Methods Studies Study Characteristic Questions Used to Guide the Appraisal Significance 1. Was the relevance of the research question convincingly described? 2. Was the need to use mixed methods established? Expertise 3. Did the researcher or research team possess the necessary skills and experience to rigorously implement the study? 4. Were the contributions or expertise of each team member noted? Appropriateness 5. Were the study purposes aligned with the mixed methods strategy that was used? 6. Did the mixed methods strategy fulfill the purpose or purposes of the study? Sampling 7. Was the rationale for selecting the samples for each component of the study provided? 8. Were study participants selected who were able to provide data needed to address the research question? Methods 9. Were the methods for each component of the study described in detail? 10. Were the data collection methods for each study component appropriate to the philosophical foundation of that component? 11. Was protection of human subjects addressed in the study? 12. Were the reliability and validity of quantitative methods described? 13. Were the trustworthiness, dependability, and credibility of qualitative methods described? 14. Were the timing of data collection, analysis, interpretation, and integration of the data specified? Findings 15. Was the integration of quantitative and qualitative findings presented visually in a table, graph, or matrix? 16. Was the integration presented as a narrative? 17. Were the study limitations noted? 18. Were the findings consistent with the analysis, interpretation, and integration of the qualitative and quantitative data? Conclusions and implications 19. Were the conclusions and implications congruent with the findings of the study? Contribution to knowledge 20. Was the study's contribution to knowledge worth the time and resources of a mixed methods study?

Synthesized from Creswell, J. W. (2015). A concise introduction to mixed methods research. Los Angeles, CA: Sage; Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches (4th ed.). Los Angeles, CA: Sage; and Creswell, J., Klassen, A., Plano Clark, V., & Smith, K. (2011). Best practices for mixed methods research in health sciences. Retrieved from http://obssr.od.nih.gov/mixed_methods_research. Building on your knowledge of quantitative and qualitative methods, learning how to critique mixed methods studies extends your capacity as a scholar. These standards of quality displayed in Table 14-11 provide a systematic method for critically appraising mixed methods studies. Using the quality standards proposed, a critical appraisal of a mixed methods study conducted by DuBay et al. (2014) is provided as an example. Summary of the Study DuBay et al. (2014), a team of 13 researchers, examined decisions by African Americans (AA) to become organ donors. The convergent concurrent mixed methods design included qualitative data that were collected through six focus groups and quantitative data that were collected through a survey administered to focus group participants. The Theory of Planned Behavior (Azjen, 1991) guided both components of the study. During the integration and interpretation of the findings, qualitative data were quantified using frequency of responses and were displayed side-by-side with the quantitative findings for comparison and confirmation. Significance AAs are underrepresented among registered organ donors and overrepresented among persons on waiting lists for transplants (DuBay et al., 2014). The study was socially and clinically relevant because the need for organ donors is increasing and the number of persons registered to donate organs is inadequate to meet current needs. The only reason that DuBay et al. (2014, p. 274) gave for using a mixed methods design was that the design had been “previously used in community health research to address health disparities” (Kawamura, Ivankova, Kohler, Perumean-Chaney, 2009; Ruffin et al., 2009). A more compelling reason for using mixed methods would have strengthened the study description. Expertise The research team was comprised of three physicians, two of whom also held master's degrees in public health; nine PhD-prepared researchers; and a baccalaureate-prepared employee of an organ center. The first author and the majority of the team were affiliated with the Division of Transplantation at the University of Alabama at Birmingham. DuBay reported the funding received from NIH that supported implementation of the study. The clinical expertise of team members and their educational preparation in research were noted, indicating the team's ability to implement a rigorous study. Two team members coded transcripts because of their experience in qualitative research; information about the specific contributions of other team members was not provided. Appropriateness The study purpose was stated to be identifying “factors (beyond those already identified) associated with AAs choosing to become a registered organ donor” (DuBay et al., p. 274). Guided by the Theory of Planned Behavior (Azjen, 1991), the qualitative data provided a deeper, contextual description of barriers and facilitators related to organ donation. The quantitative data provided the opportunity to compare and contrast the barriers and facilitators described by participants who were registered organ donors with those identified by the participants who

Qualitative data and quantitative data were collected in a manner congruent with their respective philosophical foundations. Human subjects protection was not described thoroughly, but the researchers indicated that the study was approved by the institutional review board (IRB) of University of Alabama at Birmingham (DuBay et al., 2014). IRB approval is an indicator that the study followed the standards of ethical research. The content and construct validity of the quantitative instruments were established by the researchers' report of the iterative process used to develop items consistent with the theory that served as the study framework. No information was provided about assessment of the reliability of the survey or its subsections. Although not identified by the researchers as indicators of rigor, the description provided of qualitative data collection, and analysis included measures used to increase credibility, specifically the level of agreement between the independent coding done by two researchers, use of a qualitative software program that included an audit trail for the process, and the inclusion of quotations in the research report that were consistent with identified themes. The NVivo 10 software used for coding allows researchers to explore various combinations of codes, in their search for themes. The software created an audit trail that documented and provided the rationale for the researchers' decision- making process. Evidence to support the credibility and dependability of the qualitative data collection and analysis was documented using these methods. The researchers specified the timing of the data collection and analysis for each phase. “Mixed methods data analysis and integration of the quantitative and qualitative results were performed at the completion of the separate analyses of the survey and focus group discussion data.” (DuBay et al., 2014, p. 275) Findings The integration of DuBay et al.'s (2014) quantitative and qualitative results was described and displayed in a table. “Qualitative themes and categories, organized according to the constructs of the Theory of Planned Behavior, were compared with quantitative survey items in a joint display matrix…the number of text references for qualitative categories were compared with the statistical test probability values for quantitative survey items to identify consistency in the participants' viewpoints about becoming a registered organ donor.” (DuBay et al., 2014, p. 275) The sample consisted of 87 AAs, 22 of whom were registered organ donors. With a mean age of 50 years, the participants were primarily female (DuBay et al., 2014). Study limitations were specified. “Underrepresentation of males may be especially important, as studies have demonstrated that non-donation attitudes are more likely to be related to medical mistrust in African American males than in African American females (Boulware et al., 2002)…The self-developed items on the questionnaire were not subjected to construct validity testing because of the small sample size…despite attempts to (prospectively) include items on the questionnaire that would measure each qualitative theme discussed during the mock focus group, some new themes emerged during the focus groups (and thus after the questionnaire was developed) for which there were no matching quantitative items. This situation is consistent with the inductive nature of qualitative research and its ability to yield more in-depth exploration of the phenomenon of interest and thus may also be a strength of the study (Lincoln & Guba, 1985).” (DuBay et al., 2014, p. 282)

Because of the matrix display and the description of the methods, the reader can feel confident that the findings were consistent with the collection, analysis, and integration of the data. Conclusions and Implications DuBay et al. (2014) identified a previously undocumented finding, which was the “emergence of a self-perception that organs from AAs are often unusable because of the higher prevalence of health issues compared with the prevalence in other races” (p. 281, 282). The implication for practice is that there is a need to include facts related to the usability of organs in community education programs. The findings validated common barriers to organ donation found in the literature such as fear, financial impact on the donor's family, the lack of a proper burial for the donor, and disfiguration of the donor's body. In keeping with AA culture, potential donors would benefit from discussing their decision with family and friends. Familial notification should be incorporated into donor registration, so as to increase the likelihood that a donor's wishes are supported at the time of death (DuBay et al., 2014). Conclusions and implications were congruent with findings. Contributions to Knowledge The convergent concurrent mixed methods study conducted by DuBay et al. (2014) uncovered novel insights about organ donation decisions of AAs. Critical appraisal of this mixed methods study supports its rigor and contribution to knowledge. “Using a mixed methods approach helped not only produce more rigorous conclusions, but allowed better capturing of the nuances that may account for differences in the intentions to become or not to become a registered organ donor. Results from this study suggest new content and motivational messages to include in campaigns to increase African American donor registration.” (DuBay et al., 2014, p. 282) Key Points • Mixed methods approaches most commonly combine quantitative and qualitative research methods. Data are collected either sequentially or concurrently. • The philosophical motivation for many mixed methods studies is pragmatism. • The three mixed methods approaches usually implemented in nursing research are (1) exploratory sequential designs, (2) explanatory sequential designs, and (3) convergent concurrent designs. • Exploratory sequential designs may be used when the researcher wants to expand on what is known about a phenomenon and the researcher does not want the content of the quantitative instruments to bias data collected qualitatively. These designs are used when the researcher needs insight into participants' perspectives prior to finalizing the quantitative component: they represent explanation of a phenomenon, followed by quantification. • When using an exploratory sequential strategy, the researcher collects and analyzes qualitative data before beginning the quantitative component of the study. Results from the qualitative component are used to plan or refine the methods of the quantitative phase. • Explanatory sequential strategies are used to provide additional insight into the topic being studied by providing multiple viewpoints. • When using the explanatory sequential strategy, the researcher conducts the quantitative component of the study before beginning the qualitative component. After the quantitative data are analyzed, the researcher finalizes the questions for the qualitative phase for the