Blog Viewer

Volume L, Number 1

Utilization of a Paired Comparison Analysis Framework to Inform Decision-Making and the Prioritization of Projects and Initiatives in a Highly Matrixed Clinical Research Program

Authors

Marcus R. Johnson, MPH, MBA, MHA
Cooperative Studies Program Epidemiology Center-Durham, Durham Veterans Affairs Health Care System – Medical Center
Melissa Middleton, MSW, LCSW
Cooperative Studies Program Clinical Research Pharmacy Coordinating Center, New Mexico VA Health Care System
Mackenzie Brown, BS
Cooperative Studies Program Epidemiology Center-Durham, Durham Veterans Affairs Health Care System – Medical Center
Tara Burke
Cooperative Studies Program Coordinating Center-Perry Point, VA Maryland Health Care System
Tammy Barnett, MA, ACRP-CP
Cooperative Studies Program Coordinating Center-Hines, Edward Hines Jr. VA Medical Center
Grant Support
The research reported/outlined here was supported by the Department of Veterans Affairs, Cooperative Studies Program (CSP).

Abbreviations

CSP Cooperative Studies Program
ORD Office of Research and Development
VA Department of Veterans Affairs
VAMCs VA Medical Centers
HSS CSP Health Systems Specialist
PC Paired Comparison
CRPCC Clinical Research Pharmacy Coordinating Center
CC Clinical Trial Coordinating Center
EC Epidemiology Coordinating Center
SMART Site Monitoring, Auditing and Resource Team

Background

Decision-making and prioritization of projects and initiatives prove to be arduous tasks for any organization. In part, this is due to the need to simultaneously determine and evaluate the potential consequences and downstream effects of those choices during decision-making and prioritization efforts (Saaty, 2008; Simon, 1979). This observation is particularly salient in large, complex organizations with matrix management structures (Davis & Lawrence, 1978). The area of research administration is not immune to these challenges and in actuality, is notably impacted by them due to the nature of the field with regards to frequently changing priorities, matrixed management structures, and limited research funding opportunities (Thom et al., 2014). Clinical research administrators often must make decisions and prioritize projects under the aforementioned conditions and may be better prepared to do so by using a methodology that is structured, categorized, and inclusive of multiple stakeholder perspectives as these criteria have been demonstrated as critical components for decision-making in the context of the prioritization of initiatives (Carnero & Gomez, 2016; Mitton & Patten, 2004; Tromp & Baltussen, 2012). These factors are paramount to increasing the likelihood of an initiative’s completion and/or sustainability, and the overall operational efficiency of an organization. There is an abundance of literature on paired comparison analyses to inform decision-making but there is a limited amount of publicly available information on its use in research administration settings (Bradley & Terry, 1952; Ock, Yi, Ahn, & Jo, 2016; Torrens & Smith, 2013; Lorio, Martinson, & Ferrara, 2016).

The Department of Veterans Affairs (VA) is the United States’ largest integrated healthcare system and provides comprehensive care to more than 8.9 million Veterans each year (2017). The Cooperative Studies Program (CSP), a division of the Department of Veterans Affairs (VA) Office of Research and Development (ORD), was established as a clinical research infrastructure to provide coordination and enable cooperation on multi-site clinical trials and epidemiological studies that fall within the purview of VA (2018a). The first VA Cooperative Study was conducted in 1946 to evaluate the efficacy of various drugs, including the antibiotic streptomycin, in the treatment of tuberculosis for 10,000 Veterans with this condition; the results of this study revolutionized the treatment of tuberculosis and led to the development of an innovative method for testing the effectiveness of new drugs (2018b). Currently, the program consists of eleven coordinating centers that facilitate the execution of clinical trials and epidemiological studies through the provision of project management, statistical, drug and device management, and regulatory and compliance support, with each center having a primary focus on either clinical trials or epidemiological studies (VA Office of Research & Development, 2013). CSP also houses a pharmacogenomics laboratory that was created to support ongoing and future pharmacogenomics studies and clinical trials within the program, as well as a consortium of VA medical centers (VAMCs) that have teams (nodes) in place dedicated to enhancing the overall performance, compliance, and management of CSP multi-site research (VA, 2018c; Condon et al., 2017; Johnson et al., 2018). CSP Health Systems Specialists (HSSs) have the primary responsibilities of facilitating the alignment and coordination of activities across the program, leading initiatives to meet program goals, and communicating the larger CSP vision and direction to colleagues across the program. This position also makes recommendations on resource allocation and project identification and prioritization to leadership at both the CSP national program level and the Center level. Therefore, an organized, methodological approach to decision-making, prioritization of initiatives, and resource allocation is vital to those individuals serving in the HSS role, as well as to research administrators in other settings.

The purpose of this project was to determine the feasibility of utilizing a paired comparison analyses framework to inform decision-making in the context of prioritizing projects and initiatives assigned to or initiated by the CSP HSS group. The findings may inform individuals or groups in research administration and leadership roles seeking to develop, select, and prioritize projects within their organizations.

Methods

Participants

The participants in this study were nine HSSs that each represented one of the eleven VA CSP Coordinating Centers (three Epidemiology Coordinating Centers (ECs), five Clinical Trial Coordinating Centers (CCs), and one Clinical Research Pharmacy Coordinating Center (CRPCC)), as well as the CSP Program Manager. There were two ECs that did not have representation on the HSS group and subsequently, did not participate or have direct input during this exercise. The individual administering this study was the HSS facilitator and their primary role was to coordinate exercises whose objectives were to strengthen leadership and administrative skills among the participants.

Prioritization Tool

The primary instrument used in this initiative was the Paired Comparison (PC) Worksheet (2018d). This worksheet was selected to be used as a tool during the creation of a shared framework for comparing values and projects among the participants in order to translate prioritization and critical thinking behavior to daily work life. There are six steps associated with successful use of the PC worksheet and these were all followed during this process. Step one indicates populating the worksheet with all options targeted for comparison. After all options have been identified, step two specifies listing the options in the cells vertically and horizontally across the gridded worksheet so that there are two of each option. The structure of the worksheet is comprised such that any areas where an option would be compared with itself or with another option more than once is negated, therefore, each option is compared with all other options only once. Step three details comparing the rows against the columns and assigning the option that is of higher priority and importance to the blank cell. Steps four and five entail scoring and totaling the score for each option. The options were scored as “0 = no more important”, “1 = slightly more important”, “2 = moderately more important”, and “3 = significantly more important” in relevance to the other options. Once the options were tallied and scored, a clear ranking was derived. Step six allows time in the process for any adjustments to be made that the participant or facilitator deems necessary.

Prioritization Process

Prioritization of Values: The participants in this exercise came from diverse backgrounds with regards to their expertise and roles at the VA CSP Coordinating Centers. Considering this diversity, the HSS facilitator developed a list of values based on the HSS group’s role in the organization so that all projects could be evaluated in the context of a specified values framework. This approach also enabled the participants to become comfortable and familiar with the PC worksheet. The values chosen by the HSS facilitator included the factors that HSSs were most commonly faced with when evaluating the feasibility of executing a project. These factors are commonly associated with decision-making in research administration and leadership roles outside of the VA CSP setting, and include Risk, Feasibility, Cost/Time, among others (Deeming et al., 2018; Baskerville, 1991; Layard & Glaister, 1994; Morgan, Hejdenberg, Hinrichs-Krapels, & Armstrong, 2018; Kuruvilla, Mays, Pleasant, & Walt, 2006; Henderson, 2001). These factors were then populated into the rows and columns of an abbreviated version of the Paired Comparison worksheet in Table 1. This worksheet was then sent to the participants by email and they were instructed to compare the values against each other and to return the document to the facilitator. Completion of the values comparison resulted in a defined ranking of the most important value for each participant. After receipt of the completed worksheets, the facilitator tabulated the results and communicated them to the group during their next scheduled conference call. The results were discussed and the group agreed that the rating of the values was accurate. The members were then advised that they would be receiving a follow-up prioritization worksheet containing a list of HSS projects.

Prioritization of Projects: The facilitator populated the same PC worksheet template (Table 2) with projects that were listed as standing items on the HSS monthly call agenda. Projects were also selected based on feedback from individual HSS group members, suggestions by other groups or individuals from across the program, or communications from CSP leadership to the HSS group regarding program-level challenges that were of enough importance to request that they be addressed. The projects that were selected for the PC work were relatable to all participants so that meaningful selection could occur for each participant. The list of projects that were populated in the rows and columns of the PC worksheet included Document Review & Mapping, Improve Smart Communication, Document Writing Process, Facilitator Guidelines, Training Plan, Improve Internal Review (IR) Process, Determine Virtual Meetings, and CSP Publications. The document was then distributed to the participants by email and the participants were then instructed to follow the same methodology as previously executed. The PC worksheet was then used to compare all projects in the list against each other. Participants determined which projects were of higher priority and importance as compared to other projects, all while keeping the shared values and results from the previous exercise in mind and as a reference. The results were presented to the participants during a subsequently scheduled conference call and the ranking that was tabulated by the HSS facilitator was discussed.

Prioritization of Projects Based on Values: During the conference call that followed the Prioritization of Projects PC exercise, one group member provided feedback that indicated the PC worksheet was not sufficiently weighted to account for the differences between projects, based on values. This individual hypothesized that some values may or may not account for the differences seen in the rankings of projects between HSS individuals. Based on that feedback, the HSS facilitator created a grid similar to the PC worksheet template and it was used to compare the values against the top five projects using a Likert scale (Table 3). The values and projects were aligned across the rows and columns to simulate an experience comparable to the prioritization exercises that were completed with the first two PC worksheets. This grid was distributed to the group members by email and the directions remained the same for comparing and ranking the values and projects as before. Once responses were received, weighted results were tabulated by the HSS facilitator and communicated on the next scheduled conference call.

Results

Figure 1 shows the ranking of values from the first Values Paired Comparison exercise the HSS group was instructed to participate in. These values are ranked from most important (1) to least important (5), based on a scale from 1-5. The most important value chosen by the group during this exercise was “Risk”, while the least important value was identified as “Effort”.

The results from the second exercise, the Project Paired Comparison worksheet, are displayed in Figure 2. These results show the complete ranking of the projects across all nine of the HSSs and the CSP program manager. The Document Writing process was originally considered by the group to be a high priority project, but once the PC framework was applied, the results showed that it had fallen in importance to the fourth spot. Once these results were stratified across the two types of HSS employees, “EC HSSs” and “CC HSSs”, results showed that the EC HSSs placed higher value on the project “Training Plan,” which would provide additional training for their role, i.e., the HSS training plan. Additionally, the CC HSSs placed higher value on projects aimed at addressing issues that impacted the entire program, such as “Improving SMART Communication” and “Determine Meetings to Hold Virtually”. When all participant scorings were tabulated together, three projects aimed at addressing program-level issues clearly rose to the forefront.

The results from the last group exercise, ranking of projects by values, is shown in Figure 3. The projects aimed at addressing program-level issues rose to the top, with the highest-ranking project being “Determine Meetings to Hold Virtually”. This project had previously been given high-priority by VA leadership due to the significant costs (flights, lodging, per diem, etc.) of having large, in-person meetings. Additionally, the projects “Improve SMART Communication” and “Document Review and Mapping” were among the top three weighted results, which were two of the projects that would impact the entire program. The “Document Writing Process” fell in rank to the fourth spot. There was a 90% response rate for the distributed worksheets used by the facilitator for scoring during this program.

Discussion

Decision-making and prioritization of projects and initiatives are often challenging and complex responsibilities for organizations to undertake (Salihu, Salinas-Miranda, Paothong, Wang, & King, 2015; Simons, Benders, Bergs, Marneffe, & Vandijck, 2016). Strategies and tools that provide a structured framework for accomplishing these tasks can be beneficial in mitigating the burden and risks associated with these efforts. The collaborative nature of research is increasing in the form of research networks and other partnerships; therefore, research administrators will benefit from identifying and developing processes for group decision-making and other aspects of collaboration that warrant structured approaches for the prioritization of tasks (Adams, 2012; Bozeman, Gaughan, Youtie, Slade, & Rimes, 2016; Fagan et al., 2018). Even in the context of established research administration leadership and management teams, decision-making, the prioritization of projects, and resource allocation prove to be challenging tasks due to diminishing funding and rising expectations (National Science Board [NSB], 2012). This initiative demonstrates that the utilization of a paired comparison analyses framework to inform decision-making as it relates to the selection and prioritization of projects was effective in a highly matrixed clinical research program. Therefore, the use of this tool may be beneficial to other research administrators that are faced with similar challenges as they work to manage the execution of research studies and initiatives at their respective institutions. The use of this framework was also effective in prompting discussion that clarified ambiguity related to ill-defined project definitions and scope.

There are several publications that report the use of a paired comparison analyses framework to improve the selection and prioritization process of projects and initiatives in healthcare, business, and other settings (Canero & Gomez, 2016; Mennecke, Townsend, Hayes, & Lonergan, 2007; Ock et al., 2016; R. Subramoniam, Huisingh, Chinnam, & S. Subramoniam, 2013), but there is a limited amount of publicly available information on the use of the tool in a clinical research administration setting. We are therefore unable to compare the results of this project with previous initiatives but can address some common themes that occurred over the course of our work.

The use of this framework confirmed the need for more clarity around projects that the HSS group included as a part of this effort. It is critical that clear, specific project parameters such as the problem statement, scope, and timeline are established prior to beginning work on any project in order to increase the efficiency of its execution, as well as to eliminate ambiguity related to its desired outcome. For example, there was considerable ambiguity within the group surrounding the third ranked project, “Improve SMART Communication”. This project was initially suggested to the HSS group by others in CSP as being one that would be beneficial to the program to undertake. Within the HSS group alone, there were several interpretations of what the intended goals of this project were, including improving communication between research study sites and the CSP Site Monitoring, Auditing and Resource Team (SMART) group, as well as improving communication between Coordinating Centers and the CSP SMART group (2018e). Naturally, the differences in interpretation of the project goals led to confusion around what the project execution plan should entail. Utilizing the paired comparison framework necessitated that the HSS group reach back out to those in the program that requested the project for additional details on the problem that it was intended to solve and other specifics related to it. Research administrators are often faced with competing requests from multiple parties for resources, e.g. study sponsors, study sites, internal study teams, etc., for decision-making with regard to taking on projects, and for their expertise on how to manage projects (Sajdyk et al., 2015; Glasgow et al., 2014). This example demonstrates the tool’s usefulness in eliciting clarity and specificity of project parameters during the prioritization process for projects that are requested by either a group or individuals for another group to take action on.

It was also of interest to observe that when the results were stratified across the two types of HSS employees, “EC HSSs” and “CC HSSs”, they showed that the EC HSSs placed higher value on the project “Training Plan,” which would provide training for their role, i.e., the HSS position training plan. Although there are many similarities between the ECs and CCs in terms of considerations that are involved with executing research studies in their respective settings, there are also numerous differences between them due to the nuances that exist with conducting epidemiologic, observational studies, as opposed to randomized clinical trials. The CSP HSS group started with representation solely from the CCs and the EC HSS members have only been a part of the HSS group for a couple of years to date. Due to that sequence of events and the origin of the group’s roster, it is possible that EC HSS members desired additional training on the responsibilities of the HSS role and what it entailed given that the they had not functioned in the position previously. The addition of EC representation to the HSS group has been beneficial to the larger program because it provided another venue for input from the ECs in project decision-making and prioritization efforts for program-level initiatives that, prior to them joining the group, may not have been considered. This finding also had great significance because it highlighted a perceived need for supplementary training from members of the HSS group, which from an administrative perspective, is a critical area to be addressed. Competency-based training and professional development is vital to the success of the clinical research enterprise (Behar-Horenstein et al., 2017; Arango et al., 2016) and research administrators are often involved with staff development and must organize and coordinate various trainings for the staff that they oversee and/or work with. Utilization of the paired comparison analysis framework demonstrated that there was an additional benefit to its use, in the form of establishing that there was a desire for additional training from our personnel. Furthermore, the most important aspect of any organization is its staff, and administrators have an obligation to invest in staff development and provide adequate training to their personnel to increase the likelihood of their success and value to the institution (Sung & Choi, 2014; Gesme, Towle, & Wiseman, 2010; Elnaga & Imran, 2013).

Lastly, our use of the paired comparison analysis framework was an effective approach for acquiring the views and perspectives of a collective of research administrators in order to inform prioritization and project selection within a clinical research program. The expertise of the CSP HSS group spans across many disciplines including project management, compliance, quality management, law, and administration. Therefore, in the absence of using a structured, methodological process for prioritizing and making decisions on the projects that were assigned to or initiated by the group, the likelihood that decisions might have been made that favored a particular discipline might have been higher. CSP Health Systems Specialists (HSSs) have the primary responsibilities of facilitating alignment and coordination of activities across the program, leading initiatives to meet program goals, and communicating the larger CSP vision and direction to others across the program. Therefore, it is paramount that any decisions that the group makes concerning projects or initiatives are formed through “systems thinking” (Adam & de Savigny, 2012; Leischow et al., 2008) and have the potential to have the highest positive impact and greatest benefit to the entire program. The management of research activities, particularly those involving human subjects, is complex and inclusive of a variety of responsibilities including ensuring compliance with research regulations and policies, managing the diversification of funding portfolios, and facilitating collaboration amongst researchers (Bian et al., 2014; Falk-Krzesinski & Tobin, 2015; Zikos, Diomidous, & Mantas, 2012). This framework facilitates broad, high-level thinking by incorporating a variety of perspectives into decision-making and prioritization efforts and would likely be useful to research administrators in any setting.

There are several limitations of our work that may present challenges to its implementation in other settings. The first was that it was difficult to identify the individuals and/or groups who originally suggested the projects to the HSS group. This situation made it difficult to determine what the true intent of the requested projects were in terms of what they were intended to achieve. Ideally, an organization’s process for evaluating and prioritizing projects and initiatives should include “the voice of the customer” (Boll, Rubin, Heye & Pierce, 2017; Nazi, Turvey, Klein, & Hogan, 2018; Valdez et al., 2018). Another limitation was that this process did not involve all relevant stakeholders across CSP. For this effort, there was representation from nine of the eleven VA CSP Coordinating Centers; two Epidemiology Coordinating Centers (ECs) were not a part of the process. The PC framework may have yielded different results had those two centers participated. Furthermore, there are several functional subdomains in the program with distinct subject matter expertise (e.g. project management, biostatistics, finance, etc.) who were not included in this process. Although the CSP HSSs work closely with these groups in varying capacities, the subdomains did not have direct participation in this effort. Had they participated in the exercises directly, results may have differed. There are potential options that the HSS group could undertake to address the lack of full CSP representation in this process. The first would involve the HSS group working with the leadership teams at the two ECs that do not currently have representation on the HSS group to identify two individuals (one from each EC) to serve as HSSs. Since the HSS position is a funded position, there would also need to be support from CSP leadership to provide funding to those two ECs for them to be able to hire and fill those positions. Secondly, the HSS group could invite subject matter experts from the relevant CSP subdomains to participate in the decision-making and prioritization process for projects when appropriate. For example, if the HSS group is tasked with making decisions related to projects that involve finance, project management, and compliance, then additional representation from the relevant CSP subdomains could be requested for their participation in the prioritization process. The sheer size of the CSP (500+ employees) makes having the direct involvement of every CSP employee in this process impractical, but having additional representation from groups or individuals that would be directly impacted by the outcome of any decision made related to a particular project or initiative would improve the overall process. Another potential limitation of our work is related to the setting in which it was conducted and its possible impact on the results of this approach in this setting and others. This initiative was conducted in a clinical research program within a large, integrated healthcare system that is managed by the United States federal government. Therefore, the contributors in this exercise were not impacted by the influence of financial considerations in their decision-making efforts. In other settings, such as business or for-profit healthcare systems, the results of this framework may have been different due to the potential impact of profit and/or revenue on the participants during this activity.

Considering these limitations, our project demonstrated several notable strengths. This activity was novel in that there is a limited amount of publicly available information on the utilization of the paired comparison analyses framework in clinical research administration settings, as determined through a review of publicly available literature. Prior applications of this methodology have been employed extensively in business and healthcare environments, but not in this specific type of setting. Another strength of this project is that it facilitated the collection of input directly from representatives of the majority of our program’s Coordinating Centers. This cross-representation may have increased the likelihood of sustainability for the selected projects. Lastly, the diversity of perspectives and experience of the HSS group was an asset during the exercise and resulting discussions, as well as during additional assessments of the projects. The expertise of the group spans across several disciplines including project management, compliance, quality management, law, and administration. The variety of backgrounds in the group undoubtedly strengthened the paired comparison activity by increasing the breadth of perspectives that contributed to the exercise, and subsequently allowed for decision-making and prioritization that was inclusive of multiple viewpoints.

In summary, utilization of the paired comparison analysis framework was an effective strategy to inform decision-making for the selection, evaluation, and prioritization of projects and initiatives in a highly matrixed clinical research program. Additional work is needed to determine the effectiveness of this strategy in other research organizations, both within and external to the VA. Future work in this area should also involve a more extensive evaluation of the projects that are selected and prioritized when utilizing this framework, in terms of their sustainability and achievement of desired outcomes. The field of research administration is complex and demanding in nature, therefore, any potential tools and approaches that can be utilized to simplify and alleviate its associated challenges would likely prove to be valuable to individuals in these positions.

Author’s Note

This work was supported by the VA Cooperative Studies Program. The other members of the CSP HSS group are as follows: Jonathan Franz, RRT, MS, Lauren Harris, MA, Alysia Mafucci, JD, Lori Nielsen, Jennifer L. Sporleder, BS, and David Burnaska, MPA. We would also like to thank Grant D. Huang, MPH, PhD of the VA Cooperative Studies Program Central Office.

Marcus R. Johnson, MPH, MBA, MHA
CSP NODES National Program Manager
Durham VA Health Care System
508 Fulton Street (152)
Durham, NC, 27705, United States of America
Email: marcus.johnson4@va.gov

Melissa Middleton, MSW, LCSW
Health Systems Specialist, CSP Clinical Research Pharmacy Coordinating Center
New Mexico VA Health Care System

Mackenzie Brown, BS
Research Assistant, CSP Epidemiology Center - Durham
Durham VA Health Care System

Tara Burke
Health Systems Specialist, CSP Coordinating Center - Perry Point
VA Maryland Health Care System

Tammy Barnett, MA, ACRP-CP
Health Systems Specialist, CSP Coordinating Center - Hines
Edward Hines Jr. VA Medical Center

Correspondence concerning this article should be addressed to Marcus R. Johnson, MPH, MBA, MHA, CSP NODES National Program Manager, CSP Epidemiology Center-Durham, Durham VA Health Care System, 508 Fulton Street (152), Durham, NC, 27705, United States of America, marcus.johnson4@va.gov.