Juvenile Justice Program Evaluation

○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Program
#1
Juvenile Justice
Program Evaluation
Evaluation
An Overview
Briefing
Second Edition
Series
Juvenile Justice
Evaluation Center
Justice Research and Statistics
Association
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Office of Juvenile Justice and
Delinquency Prevention
○
○
○
○
○
○
○
○
○
○
○
○

---

○
○
○
○
○
○
○
○
○
○
○
○

---

○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Juvenile Justice Program Evaluation
An Overview
Second Edition
The JJEC, which is
supported by the
This is the first in a series of briefings prepared by the Justice
Office of Juvenile
Research and Statistics Association under the Juvenile Justice
Justice and Delin-
quency Prevention,
Evaluation Center (JJEC) project. This briefing was revised in
provides evalu-
June 2003 to reflect changes in the JJEC approach to evaluation.
ation information,
training, and
The purpose of the briefing series, which comprises six briefings
technical assistance
to date, is to provide juvenile justice program managers with infor-
to enhance juvenile
mation that will help them evaluate their programs. Each briefing
justice evaluation in
the states. For more
addresses a topic that is of particular interest to juvenile justice
information about
program managers who are trying to determine the effectiveness
the JJEC project,
visit our Web site at
of the programs they operate.
http://www.jrsa.org/
jjec, or e-mail us at
jjec@jrsa.org.
Juvenile Justice Evaluation Center
Justice Research and Statistics Association
777 North Capitol Street, N.E.
Suite 801
Washington, D.C. 20002
(202) 842-9330
http://www.jrsa.org/jjec
○
○
○
○
○
○
○
○
○
○
○
○

---

Acknowledgments ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○ ○
Stan Orchowsky, JRSA Research Director, prepared this briefing with
input from Taj Carson, former JJEC Project Manager, and Merideth
Trahan, JJEC Project Manager. Nancy Michel, JRSA Director of Publica-
tions, provided the editing. Eric Peterson, our Grant Manager at the Of-
fice of Juvenile Justice and Delinquency Prevention, has offered valuable
support throughout this project, for which we are extremely grateful.
June 2001
Second Edition June 2003
Mary Poulin, JJEC Project Manager, and Stan Orchowsky, JRSA Research
Director, prepared this revision.
This project was supported by Grant No. 98-RN-FX-0112 awarded by the Office of Juve-
nile Justice and Delinquency Prevention, Office of Justice Programs, U.S. Department of
Justice. Points of view in this document are those of the authors and do not necessarily
represent the official position or policies of the U.S. Department of Justice.
○
○
○
○
○
○
○
○
○
○
○
○

---

○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Table of Contents
Introduction ......................................................................... 1
Step 1: Define the Problem ................................................. 2
Step 2: Implement Evidence-Based Programming ............... 3
Step 3: Develop Program Logic .......................................... 4
Step 4: Identify Measures .................................................... 7
Step 5: Collect and Analyze Data ....................................... 9
Step 6: Report Findings ..................................................... 10
Step 7: Reassess Program Logic ......................................... 11
Summary ............................................................................ 12
○
○
○
○
○
○
○
○
○
○
○
○

---

○
○
○
○
○
○
○
○
○
○
○
○

---

○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Introduction
Evaluation is a key component in the process of program develop-
ment and management. Evaluation can be a useful tool for juvenile
justice program managers to identify what results they are trying to
produce and how the activities their staff members are engaged in
are geared toward producing those results. The purpose of this
briefing is to provide an overview of the program evaluation pro-
cess as it might be implemented by a local juvenile justice program
manager. Juvenile justice program managers need information about
what their programs are trying to accomplish, how their programs
are functioning, and what results they are producing. We propose
a seven-step process that juvenile justice program managers and
others can use to analyze and assess the functioning of programs.
These seven steps are as follows:
1
define the problem
2
implement evidence-based programming
3
develop program logic
4
identify measures
5
collect and analyze data
6
report findings
7
reassess program logic
Each of these steps is discussed in detail below.
1
○
○
○
○
○
○
○
○
○
○
○
○

---

Step 1
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Define the Problem
q
define 1
Programs are developed to address particular problems or needs. In
juvenile justice programming, these problems or needs are related to
the prevention, reduction, and elimination of delinquent behavior. Juve-
implement q
2
nile justice programs are developed to reduce delinquency or to affect
attitudes and behaviors that are related to delinquency. For example, a
q
develop 3
program’s purpose might be to reduce risk factors believed to be associ-
ated with delinquency, such as poor school performance or lack of
q
identify 4
conflict resolution skills.
There are many ways in which problems related to juveniles and de-
q
collect 5
linquency may come to the attention of the community. Local elected
officials, other community leaders, judges, those working with juve-
niles in the community, and concerned parents and other individuals
q
report 6
may identify problems that would benefit from juvenile justice pro-
grams. From the program manager’s perspective, however, it is critical
q
reassess 7
to collect data that demonstrate that there is a problem, and that
describe the magnitude and nature of the problem. These data may
come from a variety of sources, including law enforcement agencies
(crime and arrest rates for specific offense types), schools (grades,
drop-out rates), and community agencies (teenage pregnancy rates
and proportion of children living in poverty). For example, if pro-
gram managers or community leaders believe that programs to pre-
vent truancy are needed, they should be able to point to school data
that show a higher truancy rate than the statewide average or that of
neighboring communities.
As part of the problem identification process, program managers
should also be able to identify the characteristics of the juveniles they
are targeting for particular interventions. Some interventions may be
more effective when aimed at particular groups of juveniles. For ex-
ample, diversion programs may target young status offenders, while
other programs target chronic and serious juvenile offenders. Programs
that target specific populations but for one reason or another provide
services to other types of juvenile offenders may find that their effec-
tiveness is compromised. Programs that target first-time juvenile of-
fenders, for example, but end up serving more chronic offenders may
discover that their clients need intensive supervision and psychologi-
cal counseling, services that the program is not equipped to offer.
○
○
○
○
○
○
○
○
○
○
○
○
2

---

○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Step 2
Implement Evidence-Based Programming
define q
1
Once the problem has been identified, a solution, in the form of a
program or policy initiative, needs to be implemented. Ideally, the
proposed program should be demonstrated to be an effective solu-
implement q
2
tion to the identified problem. Research and evaluation studies of
juvenile justice programs have produced a great deal of knowledge
q
develop 3
about what works to prevent and reduce delinquency. OJJDP’s
Blueprints Project, which has identified 11 effective programming
q
identify 4
initiatives for juvenile justice, is an example of how accumulated
research and evaluation knowledge can be used to inform juvenile
justice programming (for more information, see the Blueprints for
q
collect 5
Violence Prevention Web site at: http://www.colorado.edu/cspv/blueprints/
index.html). While it is certainly possible that a proposed program
q
report 6
may be so innovative that there is little research related to the ques-
tion of its effectiveness, this is not often the case. Program planners
reassess 7
who can point to similar initiatives that have been demonstrated to
q
be effective are much more likely to see successful results from the
program they design. JJEC has begun to highlight exemplary pro-
grams with demonstrated efficacy on its Web site. Furthermore, as a
part of this effort, publications with sound evaluation design and
methodology are emphasized as well. (For more information, see
http://www.jrsa.org/jjec/programs.)
○
○
○
○
○
○
○
○
○
○
○
○
3

---

○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
○
Step 3
Develop Program Logic
q
define 1
Programs are defined by their activities. These activities are carried
out in an effort to solve the identified problem. Program managers
must be able to document what they and their staffs are doing and
implement q
2
how these activities will address the problem they are attempting to
solve. This may be accomplished by identifying goals, which are broad

q
develop 3
statements of what the program hopes to accomplish, objectives,
which spell out the specific strategies to be used to accomplish goals,
q
identify 4
and activities, which are the actions the program staff undertake to
accomplish goals and objectives. By specifying the logical connec-
tions between goals, objectives, and activities, the program manager
q
collect 5
creates a logic model for the program.
report 6
Goals should be expressed in a way that makes it clear that their ful-
q
fillment will lead to a solution to the problem. A goal is defined as a
broad statement of what the program intends to accomplish. It is also
q
reassess 7
the intended long-term outcome of the program. Here are some
examples of goal statements:
• To reduce the number of serious and chronic juvenile offenders.
• To divert nonviolent juvenile offenders from state juvenile
correctional institutions.
• To restore the losses suffered by the victims of crimes.
Each goal has one or more objectives associated with it, and each
objective has a set of activities with which it is associated. If the ac-
tivities are carried out successfully, they will hopefully lead to the
accomplishment of the program’s objectives, which will in turn lead
to the achievement of its goals. Program managers and staff should
understand and be able to explain how each activity helps achieve a
goal or objective, and how achieving these goals and objectives will
help solve the problem. (For example, if we teach juveniles about the
harmful effects of drugs, we will accomplish our goal of increasing
knowledge about drugs and their effects, which will ultimately result
in fewer juveniles abusing drugs.) If the program is conducting a
number of useful activities, but those activities cannot reasonably be
expected to lead to the achievement of its goals and objectives, then
the program will not be successful in solving the problem.
○
○
○
○
○
○
○
○
○
○
○
○
4

---