Planning the evaluation of Saskatchewan’s KidsFirst
early childhood intervention program
What is KidsFirst?
KidsFirst is a federally-funded, provincially-run intervention
program launched in 2002 that provides support and services
to vulnerable families with young children (aged 0-5) in
Saskatchewan. It is offered in nine areas of the province
that were identified as having high levels of need when the
program was established. There are KidsFirst programs in
Meadow Lake, Moose Jaw, Nipawin, Northern Saskatchewan,
North Battleford, Yorkton and selected neighbourhoods in
Prince Albert, Regina and Saskatoon.
KidsFirst uses intensive home visiting to support children
living in very vulnerable circumstances so that they can be
nurtured and supported by healthy, well-functioning families.
Home visitors work to build capacity in families, promote
healthy child development and facilitate goal achievement
for parents (such as returning to school or finding a job).
KidsFirst also connects families with mental health and
addiction services as well as early learning and childcare
programs. Home visitors focus on the positive, building on
each family’s strengths to build parents’ self-esteem and
confidence.
Saskatchewan KidsFirst Program Evaluation (2007-10): Evaluation Framework
Why evaluate KidsFirst?
This evaluation was designed to inform the emerging body of
knowledge on promising or best practices in the area of early
childhood intervention in Canada. By examining the research
literature, and discussing evaluation with KidsFirst policy
and program personnel, it was clear to the researchers that
there is a lack of specific knowledge about what makes early
childhood interventions successful, especially interventions
directed at vulnerable families in Canada. This information is
crucial in order to develop more effective policy interventions
to reduce maternal and child health disparities.
How was the evaluation planned?
In 2007, an evaluation process began, looking at how
effective KidsFirst has been in the short term at meeting its
four goals as well as objectives specific to each goal. This
evaluation was conducted as a partnership between the Early
Childhood Development Unit at the Ministry of Education,
which oversees the program, and the Saskatchewan
Population Health and Evaluation Research Unit (SPHERU), a
bi-university research unit at the Universities of Saskatchewan
(in Saskatoon) and Regina (www.spheru.ca). SPHERU
researcher Nazeem Muhajarine led the evaluation, which was
www.kidSKAN.ca/KidsFirst 1
funded primarily by the Canadian Population Health Initiative
and the Government of Saskatchewan.
The evaluation team started by working with program
managers of the KidsFirst sites to develop an evaluation
framework, to guide how the evaluation
would be conducted to determine
if KidsFirst is meeting its goals and
objectives (available at:
www.kidskan.ca/node/174).
The evaluation was conducted in three
phases:
1. the development of program logic
models of the program (that describe
how the program is intended to work)
and the development of an evaluation framework (that
describes a plan for evaluating whether a program is
working as intended, and why it is or isn’t working)
2. A core phase during which existing data were analyzed,
and new data collected and analyzed, to determine the
KidsFirst program’s effectiveness; and
This fact sheet is part of a series describing our three-year
evaluation of KidsFirst, funded by the Canadian Population
Health Initiative, and the Government of Saskatchewan, with
some additional support from MITACS, and the College of
Medicine at the University of Saskatchewan. The evaluation
was led by Nazeem Muhajarine, lead of the Healthy Children
Research Team at the Saskatchewan Population Health and
Evaluation Research Unit, in partnership with staff from the
Early Childhood Development Unit in the Saskatchewan
Ministry of Education, and KidsFirst program staff. A full list of
team members is found in the reports.
The following reports were produced as part of this evaluation:
• Evaluation Framework
• Community Profiles
• Focused Literature Review
• Using Theory to Plan and Evaluate KidsFirst
• Report of the Qualitative Study
• Report of the Quantitative Study
• Summary of Findings and Recommendations
Reports and fact sheets can all be read online or downloaded
from www.kidSKAN.ca, the Saskatchewan Knowledge to
Action Network for Early Childhood Development. Go to www.
kidSKAN.ca/KidsFirst, or click on “KidsFirst” under the Projects
menu on the front page. For more information, contact Fleur
Macqueen Smith, fleur.macqueensmith@usask.ca, 306-966-
2957, Knowledge Transfer Manager of the Healthy Children
Research Team in SPHERU.
2 Saskatchewan KidsFirst Program Evaluation (2007-10): Evaluation Framework
3. the integration of all evaluation findings, linking them
to program goals and objectives as well as the broader
context of provincial early childhood development.
As well as giving an idea as to how effective
KidsFirst has been, findings from this
evaluation add to the emerging body of
knowledge around the practices used
in early childhood community-level
interventions.
The evaluation’s key
principles and objectives
The evaluation framework describes
the process by which the evaluation was undertaken. It
provides an overview of Saskatchewan’s KidsFirst Program;
the KidsFirst Program Logic Model; the KidsFirst Evaluation
Framework (key evaluation principles, evaluation objectives,
and data sources and collection methods); the evaluation
data collection matrix; and a review of six KidsFirst assessment
tools.
The framework describes the key principles that guided the
evaluation:
• Collaboration among key stakeholders
• Reliability and validity of the findings
• Professional standards (according to the Joint Committee
on Standards for Educational Evaluation, on its utility,
feasibility, propriety and accuracy)
• Ethical considerations (that participation is voluntary and
that participants know the aims and methods of the study,
the nature of their involvement, and any potential risks).
It also sets out the evaluation’s objectives:
• To create a community profile for each KidsFirst site.
• To describe the implementation and operation of KidsFirst
at each site.
• To compare perinatal and early childhood health outcomes
of KidsFirst participants with those of the comparison groups.
• To determine whether KidsFirst participants show better
developmental outcomes than do children in the
comparison groups, and if so, to what extent.
• To determine whether KidsFirst parents show higher levels
of parenting confidence and knowledge than do parents in
the comparison groups, and if so, to what extent.
• To determine whether the quality of parent-child
interaction is better among KidsFirst parents than among
parents in the comparison groups, and if so, to what extent.
• To identify the site-specific processes, practices, and
policies that contributed most to positive short-term/
intermediate outcomes related to families and child health
and development.
www.kidSKAN.ca/KidsFirst
• To assess selected outcomes/impacts of KidsFirst between what the program is trying to achieve, what
participants (e.g., child abuse and maltreatment outcomes, activities it is doing to bring about these achievements, and
maternal mental health outcomes) in relation to the what actually happens in the program.
comparison groups.
• To assess the overall effectiveness of the KidsFirst Program. In order to better understand how the program operates,
• To evaluate whether the overall goals/objectives of the researchers developed the KidsFirst program logic model
program were met, and, if so, to what extent. through consultation with program managers and staff,
• To provide feedback to key program stakeholders to and by reviewing program documents. The logic model
facilitate the ongoing/future implementation of the summarizes the goals of Saskatchewan’s KidsFirst program,
KidsFirst Program. and acknowledges that services and supports vary among
the nine program sites, as well as from family to family.
Developing program logic models
The process started with a day-long workshop with the
A program logic model is a visual model—like a flowchart— program managers in Regina in January 2007. During
that outlines the goals and objectives of a program the workshop, the participants identified the objectives,
(objectives are specific items highlighted within the four outcomes and indicators for the following program
overall goals for KidsFirst), the activities undertaken to fulfill components:
these objectives, and the intended outcomes. A program • prenatal casefinding and in-hospital screening;
logic model helps identify the values and assumptions that • in-depth family assessment
underpin a program, as well as the resources available in • home visiting
the program. It provides a broad overview of the program’s • mental health and addictions services
functioning, so can be used as a communication tool. • early learning and child care
Another way to think of it is as an idealized, causal path

KidsFirst Program Logic Model - Overview

Please note: In this draft KidsFirst
of the PLM, Program Logic
KidsFirst goals Modelare
and objectives
Outcomes as follows: G#2-O#1 refers to Goal #2 – Objective #1.
– Overview
identified in the Intermediate

1. Children in very vulnerable situations are born and remain healthy.

2. Children living in very vulnerable circumstances are supported and nurtured by healthy, well-functioning families.
Program
3. Children living in very vulnerable situations are supported to maximize their ability to learn, thrive and problem solve
Goals
within their inherent capacity.
4. Children living in very vulnerable situations are appropriately served by the KidsFirst program and supports.

Program Program Component Key Short-Term Intermediate

Principles Component Objective Activities Outcomes Outcomes
yCo-operative, Collaborative, Shared Responsibility

Vulnerable families Family Family

are identified prior
Casefinding to, at the time of, or
after the birth of
their child. System System
Detailed PLM
on page 10 *

The most appropriate Family Family

In-Depth Family services and supports
Assessment for vulnerable
families are
identified. System System
yEquitable

Detailed PLM
on page 11*

KidsFirst families Family Family

Home are provided with
yComprehensive

Visiting intensive support and

yAccountable

mentoring.
System System
Detailed PLM
on page 12 *

KidsFirst families Family Family

yCulturally-Affirming

Mental Health & receive appropriate

Addictions mental health and
Services addictions support
yPreventative

and services. System System

Detailed PLM
on page 13*

Children’s social, Family Family

yChild-Centred

Early Learning & intellectual, and

yEmpowering

Child Care emotional

development is
supported and System System
enhanced. Detailed PLM
on page 14 *

* Page numbers refer toKidsFirst

pages in the full report
Program (available at www.kidskan.ca/node/174).
Evaluation-Phase I 9 Evaluation Framework
July 31, 2007
Saskatchewan KidsFirst Program Evaluation (2007-10): Evaluation Framework www.kidSKAN.ca/KidsFirst 3
An overview logic model was created for the whole program
(see page 3), as well as more detailed logic models for each
program component (see pages 9-14 of the Framework,
available at: www.kidskan.ca/node/174). The logic models
show objectives, key activities, short-term outcomes and
intermediate outcomes for each program component. After
the initial workshop, these models were created, reviewed, and
revised until consensus was reached.
For example, the logic model for the home visiting program
component shows:
Objective: KidsFirst families are provided with intensive
support and mentoring.
Key Activities: A dozen activities are listed at the family level,
including: engaging families in the home visiting program;
establishing a trusting and nurturing relationship with the
family; providing learning opportunities, and assessing
family vulnerability on an ongoing basis. At the system level,
the key activity is to advocate for families with appropriate
government agencies.
Short-term Outcomes: There are several activities listed at
the family level, including: the family’s acceptance of home
visits and their retention in the program; parents’ better
understanding of children’s growth and development; the
strengthening of parent-child interaction; and increased
family self-reliance. At the system level, the key outcomes
are that vulnerable families are better supported, and that
provincial policy is informed.
Intermediate Outcomes: Again, there are several listed,
including: families are able to provide nurturing and safe
care for their children; social support networks, housing, food
security, nutrition and education, employment and income
for families are improved; and families are satisfied with
KidsFirst services.
4 Saskatchewan KidsFirst Program Evaluation (2007-10): Evaluation Framework
Data collection, analysis and reporting
The evaluation framework describes collection methods
and data sources to be used to evaluate how well the
program has met its goals and objectives. These include:
literature reviews, document reviews, key informant surveys
and interviews, the KidsFirst data management system,
administrative data, comparative group data, case studies
and a cost-benefit analysis.
The evaluation framework includes a data collection matrix,
which links possible evaluation questions, indicators, and data
sources and collection methods with the objectives of the
evaluation, as well as the overall goals of the KidsFirst Program.
As part of the initial evaluation process, the researchers
reviewed the assessment tools used in the KidsFirst
program: the Larson Prenatal Screen; the In-Hospital Birth
Questionnaire; the In-Depth Family Assessment; the Ages
and Stages Questionnaire; the Ages and Stages
Questionnaire–Social-Emotional; and the Parent Satisfaction
Survey. This review determined the purpose of each tool,
what it measures, and its reliability and validity.
It is important to recognize that evaluating KidsFirst, a large,
multi-site program, is a complex and time-consuming
undertaking. Although it is a provincial program, it is
delivered entirely at a local level in nine communities. The
needs of multiple stakeholders, service organizations and
program administrators need to be addressed, and the
varying capacity and socioeconomic realities of program
sites recognized. Evaluators need to recognise the delicate
balance between the program’s service delivery mandate,
on the one hand, and the need for evidence of the program’s
effectiveness, on the other. Given the complexity of this
work and the many potential pitfalls that may compromise
its success, a carefully planned, policy-engaged, and patient
approach to evaluation is necessary.
Printed on the iGen3 Digital Press at
www.kidSKAN.ca/KidsFirst Printing Services Document Solutions & Distribution
University of Saskatchewan • 966-6639