Functioning Effectively: A “How-To” Guide for Program Evaluation for Research Administrators |
How can we move beyond generalized assumptions and identify what’s actually working? Program evaluation can help professionals use both quantitative and qualitative data to determine the efficacy of existing programs, as well as what improvements need to be made. Evaluation does not have to be complicated and can be an effective tool for making decisions.
There are several approaches to program evaluation that can fit into the research administration environment, and we suggest exploring all options to see what fits best for your program and needs. In this article, we discuss the Kirkpatrick Model as the primary framework for evaluation. Although the Kirkpatrick Model is best known for training program analysis, it can also be creatively adapted as a tool for evaluating workflow changes.
In a nutshell, the Kirkpatrick Model helps break evaluations into four consecutive levels (Kirkpatrick & Kirkpatrick, 2016):
- Reaction: How participants react to the training, program, or process being evaluated.
- Learning: Whether the program or process was understood by the participants.
- Behavior: Did the behavior of the participants change (after the training was completed / process was updated)?
- Results: Did the program or process change have a positive impact (on the institution, business, or organization)?
Through two possible scenarios, let’s break down how to use the various levels of the model when assessing a new process or program.
Level 1: Reaction
First, ascertain the nature of stakeholder support regarding the project/product/process. (Kirkpatrick & Kirkpatrick, 2016). For this level, it is best not to wait until the end of the project or process roll-out; start at the beginning, then continue to implement a few more pulse checks throughout implementation.
Scenario A, Contract Review Process Update: You are rolling out a new contract review process over the course of the next six months. During the rollout, engage with the stakeholders around three months in to gauge stakeholder sentiment: What’s working, and what isn’t up to par? Essentially, receiving the initial reactions to the new process before full implementation occurs to provide time to make adjustments as necessary.
Scenario B, Certified Research Administrators (CRA) Study Group Implementation: You have started a CRA study group and are a few sessions into the program. At this point, you can garner initial opinions via staff satisfaction surveys, then use the feedback to refine the program.
Level 2: Learning
During evaluation, it is vital to determine the stakeholder’s learned knowledge, skill acquisition, attitude and confidence, or commitment to reach the desired outcome (Kirkpatrick & Kirkpatrick, 2016).
Scenario A, Contract Review Process: To assess learning in this scenario, a questionnaire to rate staff confidence in their knowledge of the process could be a viable option.
Scenario B, CRA Study Group: To assess learning, you could implement pre- and post-test questionnaires to gain insight into the learning outcomes.
Level 3: Behavior
Beyond what is learned, new behaviors must also be applied successfully. This involves measuring the application of what is learned and how the knowledge is being transferred into the work (Kirkpatrick & Kirkpatrick, 2016). This is the application phase of learned knowledge.
Scenario A, Contract Review Process: A strong method to assess behavior would be to analyze a sample of the contracts reviewed and see if staff followed the process accurately.
Scenario B, CRA Study Group: To assess behavioral change, you could reach out to supervisors of study group participants to identify any positive changes in their understanding of regulations, policies, workflow, etc.
Level 4: Results
This level measures whether the desired outcomes were reached based on what was implemented (Kirkpatrick & Kirkpatrick, 2016). Think of this goal at the organizational level.
Scenario A, Contract Review Process: If the desired result is to improve the overall time spent negotiating a contract, then tracking this metric incrementally throughout the process rollout and after full implementation can provide the data needed to assess results. This is where data can be compared and analyzed. At an organizational level, the desired outcome is for negotiations to move more quickly (Example: Average time to review decreased from 20 days to 8 days following the implementation of the new process). This would enhance the sponsor’s experience with the institution, and hopefully garner more activity in the future.
Scenario B, CRA Study Group: If the desired results of the CRA Study Group were for staff to obtain their certification to help improve the overall support at the organization, this is where data on how many staff took the exam and passed would come into play (Example: 85% of staff who took the exam passed). Another metric to assess could be changes in satisfaction levels with the resources provided by the institution, particularly those services or programs in which the study group participants were involved. This could show the impact of multiple aspects of the Study Group: the act of preparing for the exam, the impact of achieving the certification, and how the information learned may have been applied to positively influence institutional success.
Bottom line: The Kirkpatrick model is one of the many ways one can evaluate different programs or process updates. Carefully executing each level will provide you with the opportunity to assess the initial reactions, acquire knowledge, change behavior, and ensure that the result of the implemented project/product/process/etc. is the desired outcome.
References
Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. Association for Talent Development.
AI Statement
The content is written by the authors. AI was used to improve the structure of the article.
Authored by:
Julie Swaringim-Griffin, PhD
Assistant Vice President
Central Sponsored Programs Administration
Oklahoma State University
SRAI Communications Committee Co-Chair
Robert McTear
Director of Research Administration
New York University School of Medicine
SRAI Communications Committee Co-Chair
#Catalyst#October2026#Operations&WorkflowManagement