Program_Evaluation_Theory_and_Practice_A_Comprehen..._----_(Part_I._The_Landscape_of_Evaluation)
pdf
keyboard_arrow_up
School
Queensland University of Technology *
*We aren’t endorsed by this school
Course
572
Subject
Philosophy
Date
Apr 3, 2024
Type
Pages
14
Uploaded by CaptainElectron9905
35
CHAPTER TWO
Framing Evaluation
Paradigms, Branches, and Theories
If you do not know much about evaluation theory, you are not an evaluator. You may be a great methodologist, a wonderful philosopher, or a very effective program manager. But you are not an evaluator. To be an evaluator, you need to know that knowledge base that makes the field unique. That unique knowledge base is evaluation theory.
—
W. R. S
HADISH
, Presidential Address at the American Evaluation Association (1998, pp. 6–7)
Copyright 2010 by Simon Cousins. Used by permission.
O
ne might assume that an evaluation begins when an evaluand is identified. However, an evaluand can be identified in many ways, and decisions about appropriate strategies begin with the evaluators’ beliefs about themselves and their roles, as well as their world-
views. Evaluators naturally construct explanations for everything that exists, and they live with their own preconceptions about how the world works. They can believe that others perceive life experiences in the same way as they do, or (ideally) they can appreciate that people make claim to different realities from their own. Evaluators can step back and broaden their worldviews by learning from previously published research and practice. For Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
36 THE LANDSCAPE OF EVALUATION
example, in planning an evaluation for an after-school program teaching middle school children about using social media, evaluators might fi
nd some relevant information and guidance about how other middle schools have used similar strategies (e.g., antibullying education, education on appropriate information to share, managing time online). Pro-
gram theory or social science theory can enhance evaluators’ planning as they discern in which paradigm they will work and which evaluation model or approach they will use.
Paradigms
as sets of philosophical assumptions and theories of evaluation, programs, and social science are discussed in this chapter as ways of framing more extensive discus-
sion of theorists and their approaches. (See Figure 2.1.)
Paradigms and Theories
In this chapter, the discussion of historical and contemporary approaches to evaluation is framed as an extension of Guba and Lincoln’s (1989, 2005) framework for understanding major worldviews (or paradigms). Paradigms are broad metaphysical constructs that include sets of logically related philosophical assumptions. Theories provide frameworks for thinking about the interrelationships of constructs and are more limited in scope than paradigms. Hence a variety of theoretical perspectives can be associated with a particular paradigm.
Theory plays multiple roles in evaluation (Donaldson & Lipsey, 2006). There are theo-
ries of evaluation, program theories, and social science theories that inform our work. The concept of evaluation theory is discussed in this section as an exploration of what we say we do when we do an evaluation. Social science theories are inclusive of such areas as development, learning, motivation, and social change; and identity-based theories such as Evaluation
Models and
Approaches
Evaluation Theory
Social Science Theory
Program Theory
Paradigm
Paradigm
Paradigm
Paradigm
Figure 2.1. Paradigms; program, evaluation, and social science theories; and evaluation models and approaches.
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
Framing Evaluation
37
“Paradigm” in Plain English
Paradigms can seem very complex and confusing, but let’s take a moment to con-
sider this concept in plain English. Paradigms are made up of four sets of assump-
tions that address people’s perspectives of what is ethical, what is real, what is con-
sidered to be valid knowledge, and what is considered to be appropriate evaluation
methods. For example, through your life experiences, you have constructed a world-
view that leads you to understand reality differently from the way I do. What you think
is true may not be true for me. What you think is ethical, I may not. Neither of us is
wrong (although each of us may think the other is). For example, Lisa lives in a state with the highest capital punishment rate in the United States. She might argue that
it is appropriate if a court decides to sentence a woman to death for murdering her
boyfriend. To her, it’s black and white: If you murder someone, you should die, too. Martina is from a country that does not have capital punishment; to her, an important variable to consider would be the fact that the woman’s boyfriend was beating her.
Finally, Kent, who is of the same minority group as the accused woman, might argue that she was not given the appropriate protection from the police or social services when she contacted them about her fears of violence.
Lisa, Martina, and Kent have each created their own metaphysical construct of the world and how they see it. If you could magically look through each of their eyes, you might be taken aback at how varied their understanding of the world around them is and perhaps better understand how and why they act as they do. Each was instilled with morals about what is right and wrong from their environments while
growing up. Each decided which values were important enough to live by and sup-
port. Their metaphysical constructs of their worlds differ depending on who they
are and what they have learned and experienced in their lifetimes. These constructs
guide the ethical decisions they make daily in life. Consider the sharp divisions in the
United States during the 2016 presidential campaigns. Although Americans come
from one country, their philosophical beliefs on how to and who could improve it, were starkly different.
We can see how different ethical principles philosophically guide Lisa, Martina,
Kent, and Americans to logical connections for what they accept as real and how they each understand what is real. For Lisa, it’s real to her that the woman who killed her
boyfriend is guilty of murder. Martina believes that extenuating circumstances need to be considered before guilt or innocence can be determined, and that the death sentence is not an ethical punishment. And Kent believes that ethically, larger soci-
etal forces need to be considered to interpret what really happened. Your philosophi-
cal assumptions, of which you may not yet be aware, will determine which of the four
paradigms you work in most comfortably as an evaluator. Before we introduce the four paradigms, we first need to understand the philosophical assumptions of each paradigm. We know there are four assumptions related to the nature of ethics (axiol-
ogy), reality (ontology), knowledge (epistemology), and systematic inquiry (method-
ology). As you continue reading this chapter, you will begin to see how we can frame our different philosophical assumptions into specifi c paradigms in the different ways that characterize how each person thinks.
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
- Access to all documents
- Unlimited textbook solutions
- 24/7 expert homework help
38 THE LANDSCAPE OF EVALUATION
feminist, queer, and critical race theories. These are used both to inform decisions about evaluation practice and to inform programmatic decisions. Program theories help explain the mechanisms believed to influence the achievement of the desired program outcomes. This is sometimes referred to as the theory of change—in other words, what needs to be done with which resources by whom in order to achieve the desired change.
Paradigms
Guba and Lincoln (1989, 2005) use the term “paradigm” somewhat differently than Thomas Kuhn (1962) did in his book The Structure of Scientifi c Revolutions,
in order to bring clarity of thinking to the assumptions that underlie research and evaluation. They have characterized paradigms within this context as metaphysical constructs made up of four sets of philosophical assumptions (see Box 2.1). Shadish (1998) notes that many of the fundamental issues in evaluation reflect differences in basic philosophical assumptions: Most debates in the evaluation field are about “epistemology and ontology, about what assumptions we make when we construct knowledge, about the nature of many funda-
mental concepts that we use in our work like causation, generalization, and truth” (p. 3).
Box 2.1. Four Sets of Philosophical Assumptions in Paradigms
Philosophical assumption
Guiding question
As experienced in life
Axiology
What is the nature of ethics?
Your societies, cultures, and religion taught you morals of what is right and wrong. You developed your own values, which are the principles or ideas that you believe in and are willing to support and defend. Both guide your ethics in how you conduct your lives. Lisa, Martina, and Kent all morally object to murder. Lisa values justice, so she ethically supports capital punishment. Martina values a person’s right to self-defense, so she ethically denies capital punishment. Kent values equal rights of appropriate social services for all, so he ethically rejects capital punishment.
Ontology
What is the nature of reality?
Is there one reality that I can discover? Or are there multiple realities that differ, depending on the experiences and conditions of the people in a specifi c context? Lisa’s brother, a prison guard, tells her stories of the awful behavior of the prisoners. Martina’s brother received addiction rehabilitation while in prison and now has a job and family. Kent has seen documentaries and read articles about the cruel treatment of prisoners in mass incarceration in the United States. Whose reality is real?
Epistemology
What is the nature of knowledge, and what is the relationship between the knower and that which would be known?
Another wording for the epistemological question might be: How should the evaluator relate to the stakeholders? Do you, as the evaluator, objectively stand apart from the stakeholders, or do you engage with them in deep conversation and in their activities? How could you learn what constitutes reality in the case of a woman murdering her boyfriend?
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
Framing Evaluation
39
Philosophical assumption
Guiding question
As experienced in life
Methodology
What are the systematic approaches to gathering information about what would be known?
Do you need to compare two groups, or can you document progress by intensively studying one group? Should you use a quantitative, a qualitative, or a mixed methods approach? Do you send surveys to the prison, go interview the woman, and interview pertinent workers at social service agencies? Do you collect data about domestic violence and gather information about the local and state laws?
Four paradigms provide a useful structure for examining different worldviews that are functioning in today’s evaluation world: the postpositivist, constructivist, transfor-
mative,
and pragmatic paradigms.
1
Each paradigm is described in Box 2.2 (later in this chapter), and is explained with approaches and sample studies in Part II. The boundaries between these paradigms and the evaluation approaches associated with them are not clear-cut. Rather, each paradigm can be regarded as placing different emphasis on differ-
ent philosophical assumptions, but overlap among the paradigms through the permeable boundaries that defi
ne them is still possible.
Evaluation Theory
Shadish (1998) has described evaluation theory
as “who we are” (p. 5), in the sense that it gives us the language we use in this transdiscipline to describe what we do uniquely as evaluators. The uniqueness of evaluation is “our willingness to attack value questions by studying merit and worth, our understanding about how to make results more useful than most other social scientists can do, or the strategies we have developed to help us choose which methods for knowledge construction to use depending on the needs of the evalua-
tion client” (p. 5).
Smith (2008) centers his description of evaluation theory on the purposes of evalu-
ation. Evaluation theory is that aspect that reflects “our thinking about how and why we engage in evaluation. Is the purpose of evaluation validation, accountability, monitoring, or improvement and development?” (p. 3). Theories provide guidance in determining the purposes for evaluations, as well as in defi
ning what we consider to be acceptable evidence for making decisions in an evaluation.
Criteria for a Good Evaluation Theory
Shadish et al. (1991) have suggested that evaluation theories need to meet the following criteria:
Knowledge: What do we need to do to produce credible knowledge?
Use: How can we use the knowledge we gain from an evaluation?
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
40 THE LANDSCAPE OF EVALUATION
Valuing: How do we construct our value judgments?
Practice: What do we evaluators actually do in practice?
Social programming: What is the nature of social programs and their roles in solv-
ing societal problems?
Stufflebeam and Shinkfield (2007) state that a program evaluation theory should have six components: “overall coherence, core concepts, tested hypotheses on how evaluation procedures produce desired outcomes, workable procedures, ethical requirements, and a general framework for guiding program evaluation practice and conducting research on program evaluation” (pp. 63–64). The evaluation profession has many core concepts and ethical principles, which are discussed in Chapter 1. Theorists in evaluation have also developed many different approaches that provide guidance for the conduct of evalua-
tions. However, the evaluation field has not yet produced a core body of research on evalu-
ation to support suggestions that specific use of particular procedures leads to desired outcomes (Alkin, 2013). Stufflebeam and Shinkfield (2007) conclude that the evaluation profession:
has far to go in developing overarching, validated theories to guide the study and practice of program evaluation. The program evaluation literature’s references to program evaluation theories are numerous, but these references are often pretentious. They usually denote as theories conceptual approaches or evaluation models that lack the comprehensiveness and validation required of sound theories. (p. 68)
Models and Approaches
Alkin (2013), in agreement with Stufflebeam and Shinkfield (2007), has acknowledged that what are commonly referred to as evaluation theories might more appropriately be called “
approaches
or models
” (p. 4, emphasis in original). We agree with him on that point, and therefore we discuss models and approaches developed by theorists. Models can be thought of as “a set of rules, prescriptions, and prohibitions and guiding frameworks that specify what a good or proper evaluation is and how it should be done” (Alkin, 2013, p. 4).
The historical roots of evaluation have been depicted in many different ways, due in part to the interdisciplinary nature of the field.
2
Christie and Alkin (2013) use a metaphor of a tree, with its roots represented as social accountability, social inquiry, and epistemology. To illustrate the historical and contemporary theoretical perspectives in evaluation, he depicts method-, use-, and value-based theories (approaches/models) as three major branches of the tree. This tree is useful in some respects; however, it is limited in that it primarily reflects the work of white Western evaluation theorists
3
and is not inclusive of evaluation theorists who are feminists, people of color, persons with disabilities, members of the lesbian/gay/bisexual/transgender/queer or questioning (LGBTQ) community, communities in economically poor countries, or members of Indigenous groups. In a rough way, Christie and Alkin’s (2013) three branches can be mapped onto three of the major paradigms listed earlier in this chapter (and described further in Part II of this book): postpositivist (Chapter 3), pragmatic (Chapter 4), and constructivist (Chapter 5). The Methods Branch maps onto the postpositivist paradigm, the Use Branch onto the pragmatic paradigm, and the Values Branch onto the constructivist paradigm. We propose Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
- Access to all documents
- Unlimited textbook solutions
- 24/7 expert homework help
Framing Evaluation
41
social justice as a fourth branch that maps onto the transformative paradigm (Chapter 6). As its name indicates, this branch focuses on furthering social justice, and it includes evaluators who develop theoretical frameworks based on cultural responsiveness; race/
ethnicity; on human rights, feminist, disability rights, deafness, postcolonial/Indigenous, and queer theories; and on other marginalized communities. See Figure 2.2 as an example of a tree that includes four branches of evaluation. We want to emphasize that we agree with Alkin (2013) when he stated that the branches of the trees are not exclusive. He wrote that it is not “that only one model believes in methodology and others do not. Rather the category system is based on the relative emphasis within the various models” (p. 7).
The depiction of evaluation models can thus be linked to our earlier discussion of paradigms. Box 2.2 lists the four paradigms introduced at the beginning of this chapter and shows their relationship to the major evaluation branches of Methods, Use, Values, and Social Justice. One quirky thing about evaluation history is that evaluators do not seem to leave their past behind. Rather, they seem to hold on to the various paradigms, theories, and approaches that served them in the early years, even as they recognize newer developments and evolutions in these paradigms, theories, and approaches. For example, postpositivists are still in business and advocating for randomized control trials;
pragma-
tists continue to write about strategies to increase the utilization of evaluation fi
ndings; constructivists add to the discussion about use of qualitative methods to capture the com-
plexity of reality; and transformative evaluators bring to light the voices of those who were not included in the earlier history of evaluation.
Critics have wondered whether a tree is the best metaphor for depicting the theoreti-
cal perspectives in evaluation. Patton (2004) has suggested that instead of tree branches, the various perspectives in evaluation should be depicted as branches in a river, where Figure 2.2. A four-branch tree of evaluation approaches.
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
42 THE LANDSCAPE OF EVALUATION
the water flows over the banks and opportunities exist for intermingling of ideas. We think that a water metaphor might be even more inclusive; perhaps the streams that lead into rivers and rivers that flow into oceans would provide a complex-enough metaphor to capture evaluation’s rich history and current status. There are places where freshwater and saltwater mix, and these are indicators of the permeable borders of theories and approaches.
As an alternative metaphor, the major theoretical perspectives might be thought of as ocean currents. An ocean current is a movement of water created by the force of the wind, temperature, salinity, tides, the magnetic pull of the sun and moon, depth contours, and shoreline configurations (National Oceanic and Atmospheric Administration [NOAA], 2013). The ocean’s currents flow across vast distances and play a large role in determining climate differences in the world. The ocean currents flow at a shallow level; however, there is another deep-water circulation system that is called the “global conveyor belt.” Its route covers most of the world: It travels through the Atlantic Basin around South Africa, into the Indian Ocean, and on past Australia into the Pacific Ocean Basin. (See Figure 2.3.)
This metaphor not only allows for intermingling of waters; it also demonstrates that many forces come into play to determine the nature and effects of different ocean cur-
rents. Although we do not want to push this metaphor too far, we do suggest that it is useful to demonstrate the complexity of evaluation theories, methods, and approaches.
4
Now we are going to add another piece of the puzzle: the role of social science theory in the development of the evaluand and the evaluation plan.
Social Science Theory
Social scientists have generated theories to explain human development, learning, motivation, literacy development, and changing behaviors, among other domains. Evaluators can use these theories to help guide program developers. Donaldson and Box 2.2. Major Paradigms in Evaluation
Paradigm
Branch
Description
Postpositivist
Methods
Focuses primarily on quantitative designs and data; may use mixed methods but quantitative methods dominate.
Pragmatic
Use
Focuses primarily on data that are found to be useful for stakeholders; advocates for the use of mixed methods.
Constructivist
Values
Focuses primarily on identifying multiple values and perspectives through qualitative methods; may use mixed methods but qualitative methods dominate.
Transformative
Social Justice
Focuses primarily on viewpoints of marginalized groups and interrogating systemic power structures through mixed methods to further social justice and human rights.
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
Framing Evaluation
43
Lipsey (2006) suggest that the use of social science theories
can contribute to the development of interventions that reflect collective wisdom, and hence can reduce wasted time on treatments already known to be ineffective. A caveat that must be mentioned is the difficulty of transferring social science theories across cultures. As obvious as it seems, evaluators must be cognizant of unique cultural factors that might limit their ability to generalize from a theory developed on a white, middle-class group of people from the United States to other populations and contexts.
Bledsoe and Graham (2005) evaluated a family literacy program, based on child development and cognitive learning theories, which postulates that parent–child bonding provides an opportunity for cognitive stimulation through reading and interactive activities. Thus, the program included several components, such as the use of literature and music to support the development of preliteracy and school readiness skills; the provision of health services and parent education programs; and a family literacy component that encouraged in-home reading between parents and children. Campbell, Greeson, and Fehler-Cabral (2014) used feminist theory to evaluate a counseling program for adolescents who had experienced sexual violence. The use of a feminist theoretical framework allowed them to address systemic power inequities in order to provide a safe context for the adolescents to participate in the study.
Program Theory
Program theory,
which began with the work of Chen and Rossi (1983), is discussed in more detail in Chapter 3. In its early days, it was largely based on attempts to identify and quantify those variables that would have an impact on program outcomes. Through the years, other evaluators have used program theory to develop ways of describing the evaluand, known as logic models
or log frames
(discussed in Chapter 7). These are graphic depictions of the inputs, resources, assumptions, activities, outcomes, outputs, Methods
Values
Use
Social justice
Metaphor for Evaluation Branches
Figure 2.3. The global conveyor belt as a metaphor for evaluation branches. Source:
Adapted from NOAA (2009).
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
- Access to all documents
- Unlimited textbook solutions
- 24/7 expert homework help
44 THE LANDSCAPE OF EVALUATION
and impacts of a program being evaluated. Donaldson and Lipsey (2006) have contributed greatly to the use of program-theory-driven evaluation by extending this approach to dif-
ferent methodologies in evaluation. Bledsoe and Graham (2005) demonstrate this mixing of program theory and multiple approaches in their combination of the program-theory-
driven approach with an inclusive, transformative approach that also has elements of a utilization approach and empowerment evaluation. Program-theory-based approaches are discussed further in Chapters 3 and 6.
Trochim (1998) believes that evaluators rely too much on social science theory and too little on the perceptions of stakeholders who are closest to the programs. Social science theory can be useful to provide broad guidance in such areas as literacy development or motivation; however, it does not reflect the specific contextual needs of stakeholders. He has developed mechanisms to help elicit the program theory that the stakeholders have about what is needed for a program to be successful. He calls these the “implicit theories” that the people closest to the program hold.
Evaluators’ Roles
Ryan and Schwandt (2002) connect evaluation theory with evaluators’ roles quite directly: “The concept of the ‘role of the evaluator’ is central to the theory and practice of evalua-
tion” (p. vii). If we accept the description of evaluation theory as what we do and who we are as evaluators, then this connection seems important as part of the theoretical explo-
rations in this chapter. Each major branch of evaluation theorists and approaches within those branches discuss roles of the evaluator that emphasize different aspects of what evaluators do and the roles they assume.
Skolits, Morrow, and Burr (2009) suggest that linking evaluators’ roles with specific approaches does not do justice to the complexity and dynamic nature of the roles that evaluators play in practice. They argue that during the course of an evaluation, evaluators play many different roles, depending on the stage of the evaluation and the demands of the situation. They examine what they consider to be a generic list of activities that most evaluators are called upon to pursue in an external evaluation, and then describe domi-
nant and supportive roles for an evaluator. Although these roles change from the begin-
ning to the middle to the end of an evaluation, one role is dominant throughout: the role of manager. Evaluators need to be able to manage the complex process of planning an evaluation, implementing it, and bringing it to closure. Skolits et al. also identify specific roles that occur more commonly during the planning, implementation, and postevalua-
tion phases. (See Figure 2.4.)
E X T E N D I N G Y O U R T H I N K I N G · · · · · · · · · · · ·
Evaluators’ Roles
1. Which evaluators’ roles have you had experience in fulfi
lling in your life?
2. In which roles do you think you will do well?
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
Framing Evaluation
45
This generic depiction of evaluators’ roles is useful to see how evaluators adopt different roles throughout the course of a study. Theorists associated with different branches of evaluation emphasize different aspects of these roles. Hence in the discussion of these branches in Part II, the roles of evaluators are examined in specific theoretical contexts. The different paradigms offer food for thought in terms of concepts that provide a more nuanced picture of evaluators’ roles by including/emphasizing values, political contexts, objectivity, distance from or involvement with stakeholders, bias, and/or advocacy.
Because evaluation occurs in political contexts, the roles of evaluators necessitate attention to the social relations that occur in an evaluation study (Abma & Widdershoven, 2008). Skolits et al.’s (2009) model suggests the importance of political context in an evalu-
ator’s role as diplomat. Abma and Widdershoven emphasize the pervasive need for evalu-
ators to be aware of their roles in terms of social relations with stakeholders. The quality of the relations established and maintained throughout the study determines, in part, the Use advocate: Facilitate use of evaluation results.
Learner: Identify areas for improvement in the evaluation (meta-evaluation).
Diplomat: Work with stakeholders to build trust and rapport.
Researcher: Collect and analyze data. Use valid and reliable measures.
Judge: Use evidence to make judgments for program improvement.
Reporter:
Disseminate findings to stakeholders.
Detective: Find out about the context, what is needed in terms of evaluator skills.
Designer: Develop a plan for evaluation that includes a feasible design.
Negotiator: Reach agreement on contractual issues in terms of time, money, and other relevant factors.
Dominant Role: Manager
Planning Phase
Implementation Phase
Postevaluation Phase
Figure 2.4. Evaluators’ roles throughout the course of an evaluation. Source:
Based on Skolits, Morrow, and Burr (2009, pp. 282–283).
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
46 THE LANDSCAPE OF EVALUATION
quality of the results. Skolits et al. extend the discussion of evaluators’ roles with these points:
Part of an evaluator’s role is to establish social relations with stakeholders and monitor those relations throughout a study. This includes being aware of and responsive to possible power differences that might result in confl
icts. Confl
ict is not necessarily bad; it can be an opportunity to address important issues if it is handled appropriately.
An evaluator is often called upon to play a leadership role in terms of making possible confl
icts visible and providing space for discussions about the situation to occur.
Evaluators also play an important role as communicators throughout the life of the study. They need to be effective communicators not just when they write the fi
nal report, but from their fi
rst contacts with clients and through the many interim contacts with various stakeholders.
Evaluators work as facilitators of change, especially if a project includes a phase of formative evaluation in which the intent is to improve the program during the course of the study.
Evaluators also provide insights in terms of the program’s context and the con-
straints and opportunities that the evaluation encounters. For example, if there are limited funds or negative stereotypes that need to be addressed in the discussion of program changes, an evaluator can propose these as variables to be included, in addition to evaluation data on process or outcomes.
Abma and Widdershoven (2008) summarize the roles of the evaluators for the four major evaluation branches. A graphic version of their summary is displayed in Figure 2.5.
Moving On to Part II
This chapter’s summary of the status of philosophy, theory, approaches, methods, and roles prepares you for Part II of this book, in which specific branches of evalua-
tion are examined in more detail. In particular, you will be using the terms and ask-
ing the questions listed below for the remainder of the book. Figure 2.6 is an illustra-
tion of this list; it will be a helpful guide as you read Part II.
1. The axiological belief system asks: What is the nature of ethics?
2. The ontological belief system asks: What is the nature of reality?
3. The epistemological belief system asks: What is the nature of knowledge, and what is the relationship between the knower and that which would be known?
4. The methodological belief system asks: What are the systematic approaches to gathering information about what would be known?
Part II also addresses the philosophy, theories, and approaches of the four major eval-
uation branches: Methods, Use, Values, and Social Justice. Specific approaches that Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
Your preview ends here
Eager to read complete document? Join bartleby learn and gain access to the full version
- Access to all documents
- Unlimited textbook solutions
- 24/7 expert homework help
Framing Evaluation
47
are commensurate with the philosophical assumptions associated with each branch are explained. In addition, extensive examples of the components of evaluation stud-
ies are provided to give you a context in which to place the discussion of theoretical positions. Thus each chapter presents examples of evaluation studies in summary form, with the various components identified. Many of the evaluators who conducted these studies also provide commentary on the process they used and the challenges they faced in the conduct of the study.
Method
Evaluator as objective,
neutral party
Use
Social relations manager to
facilitate use
Values
Communicator who engages in meaningful dialogue
Social Justice
Relationship builder based on trust and cultural respect; investigator of structural inequities
Figure 2.5. Evaluators’ roles by major evaluation branches. Source:
Based on Abma and Widdershoven (2008).
I see gray.
Axiology
What is ethical?
Methodology
How to collect data?
Ontology
What is reality?
I see light and dark.
The EVALUAND
I’m not sure.
I think I am.
Epistemology
How to interact with stakeholders?
Close? Distant?
Figure 2.6. The four philosophical belief systems in evaluations and the questions associated with them.
Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.
48 THE LANDSCAPE OF EVALUATION
Notes
1. The transformative paradigm as conceptualized by Mertens (2015a) is inclusive of Indigenous evaluators. Indigenous evaluators have produced recent scholarship in which they argue for a separate Indigenous paradigm (Chilisa, Major, Gaotlhobogwe, & Mokgolodi, 2016; Mertens & Cram, 2016; I do not include the Indigenous paradigm in this book as a separate chapter, but I will keep my eye on developments in the Indigenous communities).
2. Fitzpatrick, Sanders, and Worthen (2004) describe five major approaches to evaluation (objectives- oriented, management-oriented, consumer-oriented, expertise-oriented, and participant- oriented). Stufflebeam and Shinkfield (2007) focus on eight recommended approaches that they situate in four categories: questions and methods (objectives-based evaluations, experimental design, and case study); improvement and accountability (context input–process product model and consumer-oriented evaluations); social agenda and advocacy (responsive/client- centered evaluations and constructivist evaluations); and eclectic (utilization evaluations). We have chosen to use an adaptation of Christie and Alkin’s (2013) roots model because it fits more comfortably with the concept of paradigms that enable evaluators to clarify the assumptions that guide their work.
3. Critics of Christie and Alkin’s (2013) evaluation tree have also mentioned the lack of attention given to evaluators who work in private industry and nonprofits, such as the Brookings Institute, the Rand Corporation, the Urban Institute, and Westat. The evaluators on the original tree are primarily academic sociologists and psychologists. Few, if any, economists or statisticians are listed on the tree.
4. If you want to know more about the ocean’s currents, you can consult the NOAA website in the References. There are five major ocean currents: in the North Atlantic, South Atlantic, North Pacific, South Pacific, and Indian Oceans (NOAA, 2013). The Gulf Stream in the North Atlantic is the largest and most powerful, and it is fed by a current from North Africa to the West Indies. When the Gulf Stream reaches Latin America, it splits into the Caribbean Current (which goes into the Caribbean Sea and the Gulf of Mexico) and the Antilles Current (which flows to the West Indies). The Caribbean Current flows up the eastern coast of the United States. It turns off south of Newfoundland, producing eddies that flow toward the United Kingdom and Norway. Mertens, D. M., & Wilson, A. T. (2018). Program evaluation theory and practice : A comprehensive guide. Guilford Publications.
Created from qut on 2024-03-05 08:42:57.
Copyright © 2018. Guilford Publications. All rights reserved.