Overview of Evaluation Design: Birth & Beyond Program

The evaluation includes both process and outcome components. The process evaluation component focuses on describing the program implementation including clients and services provided, as well as acknowledgment of the systems change and collaboration elements of the Birth & Beyond Program. The outcome evaluation component focuses on assessments of program effectiveness with respect to changes with knowledge, attitudes, and behaviors for parents/families, children, and the community.

(1) Process evaluation is designed to document implementation of the program, to provide advice in a formative manner to improve implementation and to track systems change. Collaboration at the county and site level will be examined as part of the process evaluation, identifying elements that facilitate program development, implementation, and service delivery. Periodic surveys of the Family Support Collaborative (FSC) and site level collaboratives will provide ongoing feedback regarding programmatic and systemic enhancements and barriers related to working in collaborative relationships.

Program and staff development will also be addressed in the process evaluation. Evaluation staff will meet with program staff periodically as part of the data collection process. In particular, there will be annual site visits to conduct interviews and focus group discussions with staff and to review case files and other documentation. Through this ongoing site visit process it will be possible to determine what is working, what is challenging, and identify issues for problem solving at both the site and the county collaborative level. In addition, policies and procedures and staff training will be tracked to determine how well sites maintain fidelity to the Birth & Beyond program model.

The process evaluation also includes descriptive information about services and clients, to help determine how the program is being implemented. Without this information it will be extremely difficult to link outcomes to program or client characteristics. Therefore, the process evaluation will generate demographic profiles of clients and home visitors; risk profiles of clients; home visitation services provided and levels of participation. Process evaluation reports such as monthly participation reports will provide valuable information to the sites, the county agencies, and the Family Support Collaborative. Monthly participation reports may include home visit "no show" rates, and other indications of engagement and retention of families. Baseline assessment scores and the Client Information Forms will provide basic descriptive information on each client. Client Activity Records and Home Visitation Records will provide information about services received by the client.

(2) Outcome evaluation is designed to discern the impact of the program on clients, family and community. In the absence of a formal comparison group or random assignment, the evaluation will focus on comparisons of client status, knowledge, attitudes, and behaviors pre- and post-program. For the Birth & Beyond program the primary client is the mother, for whom most pre- and post-assessment comparisons will determine outcomes. In addition there will be key outcomes associated with the well being of the infant, including immunizations.

The six required assessments for Birth & Beyond programs and the Client Intake Form will provide baseline information about the client on several dimensions of risk for child abuse or neglect. In addition the local evaluation will include a more comprehensive Family Assessment at six month intervals to measure progress along multiple dimensions related to the physical environment of the home, the caregiver(s), interactions between the caregiver(s) and the child, and health factors. In addition key health behaviors will be tracked to insure adequate prenatal and pediatric care as indicators of reduced risk for child abuse and neglect. With the assistance of the Public Health Nursing representatives on the Multi-Disciplinary Team (MDT), the evaluation will incorporate immunizations, well-baby visits, and linkages with a primary care physician to demonstrate preventative health maintenance (vs. crisis health care via emergency room visits).

Longer term outcomes will include indicators such as referral to Children’s Protective Services, completion rates of immunizations of toddlers by age two, changes in assessments, and children’s readiness for pre-school. Fidelity to the model will be a consideration in the outcome evaluation because of the necessity of linking program activities to outcomes in the data analysis.

Evaluation Overview: Research Questions, Indicators and Methods to Gather Information