At Cobblestone, we are wrapping up the first year of a K-12 curriculum study. Over the years, we’ve learned an important piece of conducting a curriculum efficacy study is measuring the implementation of the curriculum or treatment program by our participating teachers. This allows us to better determine if any differences that exist between treatment and control groups are a result of the curriculum. We hope to encourage other evaluators to measure fidelity to implementation by providing information about uses for implementation data, reasons for reduced implementation fidelity, and tips to encourage and measure implementation.
Uses of Implementation Data
One of the main reasons to collect implementation data is to provide more context for outcome results. For example, implementation context may help explain if no differences are found between treatment and control groups in student achievement. Other uses for implementation data include, among others, the opportunity to determine:
- If the program was able to be reasonably implemented in the classroom as designed
- If fidelity to all of the program components were necessary to see results
- If the program is appropriate for all classrooms/teachers/students
Reasons for Reduced Implementation Fidelity
Reasons for reduced fidelity from our participating teachers generally fall into one of the three categories below. While we are confident that other reasons exist for a lack of implementation, these three categories explain the majority of the situations we have encountered in K – 12 curriculum studies.
Implementation Requirements Too Demanding
We have found that a primary reason that a lack of implementation fidelity occurs is that teachers do not have enough time to implement the program as prescribed by the publisher/ program authors. Implementation guidelines are often considered too demanding in light of competing interests or activities within the school, district, and/or state. Teachers have to simultaneously meet the needs of their students by balancing their obligations while trying to incorporate a new curriculum. This can be a difficult challenge, especially during the first year of using a new program.
Belief that the Program is Not Appropriate
Teachers may choose to modify, supplement, suspend, rearrange, and/or omit portions of the program to meet the perceived needs of their students. We generally find that teachers feel a program is not appropriate for their students when teachers believe their students either lack certain skills; need additional practice with concepts; or have a lack of interest in program content.
Inability to Implement
Lastly, we find that implementation fidelity decreases because teachers have a perceived or actual inability to implement a new program. A perceived inability to implement the program usually occurs when a teacher feels unprepared to implement the program and, subsequently, reverts back to activities or lessons with which they feel more comfortable. Also, while infrequent, teachers sometimes lack the content knowledge or pedagogical skill to implement a treatment program without additional training that was beyond the scope of the study.
Tips for Managing Implementation Issues
If possible, the research group/ evaluation team should be involved when the program developers are establishing their guidelines for implementing the program. Developing the implementation guidelines is also the time to consider the biggest problem in implementation fidelity: teachers running out of time. We have found that teachers are more likely to implement programs with fidelity when the required components are kept to the absolute minimum.
Implementation may also be affected by the amount and type of training that is provided to teachers. For example, more training would most likely be needed if the program requires a change in pedagogy (e.g., strong emphasis on inquiry-based instruction). The training should be standardized to ensure that all participants receive the same instruction.
During the Study
During the study, implementation issues are more easily identified when teachers report frequently on their implementation. At Cobblestone, we use weekly or monthly implementation logs as the primary method of tracking implementation. While persuading teachers to complete logs can be a difficult task, we have found that response rates improve if the logs are brief (only a few minutes) and only the essential components of implementation are reported.
We have found these additional insights have proved valuable to our clients, especially in updating or modifying their curriculum; thus, we believe that identifying and managing implementation issues will allow us to provide our clients with more meaningful and useful results.