Volume LI, Number 2
Professional Development for Clinical Research Professionals: Implementation of a Competency-Based Assessment Model
Christine Deeter
Duke University School of Medicine, Duke Office of Clinical Research
Deborah Hannah
Duke University School of Medicine, Department of Dermatology
Camila Reyes
Duke University School of Medicine, Duke Office of Clinical Research
Michelle E. Mack, PhD
Duke University School of Medicine, Duke Office of Clinical Research
Marissa Stroo
Duke University School of Medicine, Duke Office of Clinical Research
Stephanie A. Freel
Duke University School of Medicine, Duke Office of Clinical Research
Rebecca J. Namenek Brouwer
Duke University
Heather E. Gaudaur
Duke University and Duke University Health System
Andrea L. Doughty
Duke University
Denise C. Snyder
Duke University School of Medicine, Duke Office of Clinical Research
Abstract
Aligning job descriptions at Duke University Schools of Medicine and Nursing with the Joint Task Force for Clinical Trial Competencies (JTFCTC) spurred additional related activities—namely, establishing professional pathways and career ladders for clinical research professional (CRP) positions. CRPs leave Academic Medical Centers for many reasons, but lack of career advancement opportunities is one of the top observations. Duke developed a competency-based tiering system for CRP job classifications. This process of Tier Advancement allows CRPs to advance their tier through demonstration of knowledge, skills, and abilities in a variety of clinical research competencies. Tier Advancement is point-based and permits CRPs to advance within a single job classification, removing the need for onerous reclassification. The point-based nature of the system allows flexibility, in that CRPs can advance by demonstrating a limited set of in-depth specialized competencies or a breadth of responsibilities across a wide array of competencies. Tier Advancement has provided benefit to individual employees by offering an opportunity for self-driven career development. It has also benefited the institution by increasing CRP retention rates and allowed Duke to better understand current competency levels of its workforce.
Keywords:
Career Development, Competency, Workforce, Workforce Development, Clinical Research, Professionalism
Background/Problem Statement
Clinical research careers have changed significantly concurrent with the increasing complexity of the research environment across all industries, including Academic Medical Center (AMC) sites (Califf, 2009). For clinical research professionals (CRPs) within academic research sites, the conduct of clinical research often involves a heavy regulatory burden, adoption of tools for study management and health system connections, data collection and sharing, and integrating internal and external processes (Gwede et al., 2005). Thus, the job of the clinical research professional has morphed, diversified, and in many cases specialized over time. While typical teams may include clinical research coordinators, research nurses, regulatory coordinators, and data coordinators or managers, there is significant variability of responsibilities and roles from program to program (Baer et al., 2011). This variability in jobs, and likely the background and credentials that are needed to fill those positions, has made it difficult to build an engaged workforce pipeline or promote clinical research as a profession. To address this challenge, Duke and others have worked toward the standardization of job descriptions and clinical research competencies for professionals (Sonstein et al., 2014). At Duke, existing clinical research staff were mapped into a defined, competency-based framework of positions (Brouwer et al., 2017). This institutionally-driven process of position mapping was successful in harmonizing clinical research competencies across a wide scope of clinical research units.
Standardizing job roles was only a first step toward clinical research workforce innovation. Skill development takes time, and true expertise in clinical research competencies can take even longer. In recent years, coordinators have opted to stay in jobs because they enjoy the interaction with participants; however, they are looking for opportunities for upward mobility (Getz, 2018). The clinical research coordinator (CRC) is at the center of patient care, safety, and research. The importance of having well-trained clinical research professionals is imperative to conducting rigorous, reproducible, quality research (Brandt et al., 2011). If institutions do not take notice, there is a great risk of loss of institutional knowledge should clinical research professionals leave for other opportunities. Employee turnover is known to be incredibly costly. It is suspected to be a minimum of $25K per employee (see Table 1 for formula). This formula was developed out of conversations with subject matter experts in the clinical research field and based on estimates of labor and cost of time. As with any profession, turnover of CRPs is costly to the institution, holds the workforce at a constant state of underdevelopment, and creates risk at the study level through participant retention (relationship with a CRC may drive participants back to study visits [Abshire et al., 2017]) and enrollment with protocol handoff (gaps in work or protocol suspension until replacement staff are identified). Along with salary and burnout, a primary reason CRPs leave their position is the perceived lack of career advancement opportunities (Center Watch Staff, 2015; Owens Pickle et al., 2017; Speicher et al., 2012). Retention of qualified staff depends on providing opportunities for career growth and empowering personnel to direct their own career advancement.
Historically at Duke University Schools of Medicine and Nursing, career advancement for CRPs often required “job hopping”—leaving one job in one part of Duke for another higher-level position in another part of Duke. Promotion decisions, while inclusive of the skill level of the employee, tended to weigh heavily on tenure, budgetary constraints, and existing needs for higher-level positions within departments or therapeutic areas. To the employee, career path opportunities were perceived to be limited to their own department, because there was not a uniform definition for CRP career advancement across the institution. These staff changes take a toll on research productivity, and such organizational turbulence leads to inequitable and subjective advancement. These inequities can compound across an organization, creating a workforce that is dissatisfied and unsettled, and encouraging internal competition due to lack of consistent definitions across departments (Breza et al., 2018). Ultimately, this culture can undermine employees’ sense of value within the workplace. A more objective competency-based advancement system could improve employee satisfaction and expand promotional opportunities if implemented across an institution (Center Watch Staff, 2015). National conversations indicate that few opportunities exist for employees to demonstrate their level of skill across clinical research competency areas.
While competency-based clinical research certifications are offered by independent organizations, such as ACRP and SOCRA, these are perceived by some as costly and may not directly influence career advancement. That said, certification does support professional development and should be pursued as one tool that staff may consider as part of “building their toolbox.” Due to a lack of an institution wide competency-based career advancement pathway, it was imperative to develop a track specific to clinical research professionals within the context of Duke’s defined job classifications (Brouwer et al., 2017). Duke’s Workforce Engagement and Resilience (WE-R) advancement initiative aimed to address the need for a clear, delineated career development pathway. This was satisfied by creating levels, or “tiers”, within each job classification, thus giving way to the name, Tier Advancement. The clinical research competencies and domains associated with Tier Advancement were developed in conjunction with subject matter experts and are founded on those developed by the Joint Task Force (Sonstein et al., 2014). Described here is the development and implementation of the WE-R Tier Advancement process, including: 1) defining the model, 2) developing assessment processes and tools, 3) launching the process, and 4) describing the impact of implementation across the Duke Schools of Medicine and Nursing.
Methods/Observations
Tier Advancement, an employee-driven, competency-based, career advancement pathway for clinical research professionals, was developed over several critical stages, each of which are described below. The documents and assessments that comprise the process can be found on the Duke WE-R website (https://medschool.duke.edu/node/97565).
Engaging the Research and Administrative Communities
The WE-R initiative grew out of the Clinical Research Professionals Working Group that is described extensively in “Using Competencies to Transform Clinical Research Job Classifications” (Brouwer et al., 2017). As with previous elements of the Clinical Research Professionals Working Group initiative, early buy-in and engagement from the Duke research community and executive leadership was essential in developing the Tier Advancement process. Early on, it was identified that the professional pathway should be: 1) transparent and achievable by the research community, 2) manageable for supervisors who may have large teams of staff and little time dedicated to each staff member’s development, and 3) understood and championed by those in leadership roles. To address these components, the WE-R initiative formed working groups composed of staff in the tiered roles, as well as in the roles that manage those classifications.
The working groups considered several elements important for an advancement model: relevance across the institution; fairness to employees; applicability of assessments; metrics of success; and burden on staff, managers, or others completing the central review. While the importance of a competency-based framework had been established, exactly how to measure competencies in clinical research had not (Sonstein et al., 2014). Therefore, working groups were tasked with defining a competency-based process for advancement. Workgroups considered several approaches to measure competencies, such as the use of study complexity scores, the number of studies, the role an individual held for each study, relevance of funding source and financial responsibility, critical elements of leadership, the use of interviews or committee review, and the best modes to measure various competencies. While many approaches were discussed, only some of these were retained in a final advancement model. Once the working group had a strong proposed model for advancement, the WE-R team offered town hall presentations to the research community. Based on town hall feedback, the model was optimized.
Addressing the Heterogeneous Academic Research Environment
Another consideration of the Tier Advancement process was Duke’s diverse research environment. Large academic medical centers conduct a variety of research, including Phase 1 trials, complex investigator-initiated research, community-based research, epidemiological studies, and retrospective chart reviews, among others. In order to support this array of research, individual CRPs need to have distinct skillsets, requiring strengths in a variety of competency domains. Initially, the WE-R Tier Advancement process aimed for a model with tremendous standardization, requiring a core set of competencies in which all staff, regardless of role, met at least a fundamental level. The difficulty with this model was that some CRP roles, such as that of the Regulatory Coordinator, require a depth of responsibilities across a limited, specialized set of competencies, whereas other roles require a breadth of responsibilities across a wide array of competencies. After recognizing that this model would seriously limit the opportunity for many staff to advance, the core-competency structure was eliminated. Instead, a point-based competency model was implemented. This standardized, yet flexible model provides all CRPs an opportunity to: 1) advance by gaining points in competencies with which they are experienced and 2) choose to grow professionally in competencies that more closely align with their research type and/or classification. Details of this point-based model are described below.
In addition to opting for a point-based model, based on a standard set of competencies, an ‘other’ competency was implemented. This ‘other’ allows for CRPs to demonstrate skills or knowledge in a competency that is not identified in any other existing assessments. To qualify for inclusion in the Tier Advancement process, this competency must significantly contribute to the science or the study team. The CRP submits a description of the competency and provides a short description of the value that the competency brings to the research team or group. The WE-R team monitors submission via an application process to ensure that the competency listed does not fit into one that is already part of the process.
Defining an Advancement Model
Contributions from the working groups and research community guided the WE-R team to develop a model that serves our diverse workforce. The model’s key objectives were to: 1) create a series of standardized competency-based assessments that would enable CRPs to demonstrate skill across clinical research competencies relevant to their day-to-day job and 2) develop a way to measure and ascribe levels of skill within each competency that accurately distinguish degrees of capability, independence, and leadership.
Levels. Four levels for the competencies were defined: Fundamental, Skilled, Advanced, and Expert. Individuals performing at the Fundamental level require some coaching, assistance, or direct oversight. An example of a Fundamental level of the competency associated with participant-level documentation would be a CRP who maintains participant-level documentation for at least one complex study with some supervision. CRPs at the Skilled level are expected to complete tasks independently, accurately, and be able to find and use available resources. Advanced level employees demonstrate a high level of skill and knowledge, have the ability to teach, mentor, or lead, and consistently apply critical thinking and problem solving when demonstrating the competency. Employees are considered Expert level in a particular competency when they have achieved the Advanced level and demonstrate leadership in a variety of unit, department, or division-level initiatives; they may oversee the work of research staff outside of their immediate unit. Due to the higher expectations for achieving the Advanced and Expert levels, these assessments often require additional documentation. The leveling structures and definitions that WE-R used in this model were further explored using the JTF competencies and published in 2018 by Sonstein et al.
Points. A flexible point-based system to allow for advancement within a heterogeneous research environment (Figure 1) was developed. A total of 37 competencies across Duke’s 5 clinical research competency domains (Research Operations, Safety and Ethics, Data, Scientific Concepts, and Site and Study Management) are available for assessment and point accumulation. The final assessments can be found on the WE-R website. Employees accumulate points to achieve each tier, and each tier above Tier 1 has an established threshold of minimum points (36 for Tier 2 and 84 for Tier 3). When employees demonstrate the Fundamental level within a specific competency, they accrue 1 point, demonstration of the Skilled level accrues 2 points, the Advanced level accrues 4 points, and the Expert level accrues 8 points.
Committee Review. In addition to accruing 84 points, successful advancement to the top level, Tier 3, requires a committee review of a submitted portfolio. Leadership and contribution are competencies that were not thought to be adequately captured within a standardized assessment. As these are both important pieces of higher-level professional development, a committee review was instituted to ensure these two concepts were applied fairly across all candidates seeking Tier 3. CRPs seeking Tier 3 were asked to provide documentation and letters of support that demonstrated leadership and contribution to the institution or research. These portfolios and collated assessments were reviewed by a committee.
Determining Assessment Mode
The working group of volunteer subject matter experts aimed to determine the best modes for assessing skill level within each competency area. They considered several important factors: the applicability and practicality of measurement modalities; the time investment on the part of the employees and their managers; and the feasibility of standardizing for regular and equitable assessment cycles. The working group believed the most applicable mode of assessment would be something that mimicked direct observation in a natural setting by an impartial expert. However, this would require a significant investment of time by many expert assessors, is incredibly difficult to standardize, and was deemed largely impractical. The least applicable, but most easily standardized mode, was a knowledge test that would assess employees’ understanding of a competency via a proctored exam. Knowledge tests require few central resources but do not assess application in the field and impose an increased time burden to employees. Further, it was recognized that managers are best able to assess their employees’ demonstration of competencies; however, creating an assessment that is both standardized and objective for managers to employ consistently is difficult.
Based on feedback from the working group, the WE-R team selected an assessment mode for each competency that would balance applicability, objectivity, standardization, and time burden for all parties. While most competencies required a single assessment to achieve the Advanced designation, others required a secondary assessment. This was particularly true for competencies assessed only with the knowledge tests, as these were felt to be the least representative of “real world” implementation of the competency.
A total of five modes were implemented across the 37 competencies:
- Knowledge tests contained approximately 20 multiple-choice questions.
- Case studies required that the candidate review scenarios of varying complexity and answer questions via free-text response.
- Self-report assessments asked employees to describe their level of responsibility with specific tasks associated with a competency, and provide free-text examples. Managers reviewed that description to attest to its accuracy and used predefined, competency-level criteria to determine the employee’s level for the competency (Fundamental, Skilled, Advanced). Clear scoring guides were embedded as part of the assessment.
- Direct observation assessments involved a manager watching an employee execute a particular skill and completing a checklist of objective measures.
- Centralized review involved a subject matter expert reviewing information submitted by the employee to an institutional application (e.g., electronic IRB submissions).
The above modes were used for testing employees seeking a Fundamental, Skilled, or Advanced designation. Achieving the Expert designation required an assessment that is somewhat different from the three lower levels. Any employee testing at the Expert level was required to first meet the Advanced criteria and then provide evidence of how they met four specific attributes of a given competency. The attributes are: 1) creating and overseeing unit-level systems related to the competency; 2) training others on this competency unit-wide and/or outside the immediate unit; 3) serving as unit or institutional expert for the competency; and 4) presenting as a go-to resource for multiple groups/staff. The manager and an independent third party must attest to the accuracy of the employee’s responses to the four attributes. All Expert competency submissions are reviewed by two subject matter experts to ensure equity across all units. Note, each employee could be assessed at the Expert level in no more than 3 competencies. This avoids the scenario in which a CRP could advance without demonstrating at least a Fundamental level in multiple competencies, which can create a too-narrow skillset. It also avoids overlap between high level tiers and the separate Senior job classifications (e.g., CRC, Sr), which are distinct job titles.
A great deal of early discussion centered on tiers for new employees, recognizing that employees may be hired with a broad range of prior experience. Ultimately, with significant input from HR, the decision was to assign all new employees to Tier 1. Differences in experience and education were analyzed against the classification salary range upon hire. New employees were then eligible for the next Tier Advancement cycle following completion of their 90-day new hire evaluation period. Employees would then receive the standard percentage salary increase if they successfully advanced to a new tier. This process allowed for equitable salary placement while maintaining advancement based on the demonstration of competencies. Employees, whether new hires or mapped, were not restricted from attempting advancement directly from Tier 1 to Tier 3. This was important to address new employees who were entering Duke with extensive experience and already had a strong competency portfolio.
Process
Piloting the Tiering Process and Assessments
The Tier Advancement process and assessments were piloted prior to the initial cycle, by testing the process and assessments with 33 employees and 29 managers. These staff members were from a variety of research units across the AMC. Duke’s clinical research structure is divided by therapeutic area. The WE-R group attempted to utilize participants from each clinical research therapeutic area to promote generalizability to all types of research. Managers and employees were asked to complete four assessments each. Leadership within the research unit ensured that participants were testing a variety of competencies.
Following completion of the assessments, staff were asked to fill out a brief survey about the process. The surveys collected data on: 1) who participated, 2) their perception of the comprehensiveness, fairness, and quality of assessment(s), 3) the competency level tested, 4) time burden, and 5) how the assessment affected employee confidence when considering electing for Tier Advancement. Feedback indicated that approximately 85% of participants thought the assessments were easy to understand, fair, and did not contain any critical mistakes. The WE-R team also evaluated each assessment and shortened those that were considered to be too time consuming. Unless significant changes were made to the assessment, employees were permitted to retain the assessments they completed during the pilot process in an official Tier Advancement cycle.
Preparing for and Tracking the Tier Advancement Process
Tier Advancement was launched in October 2017, and two cycles per year were offered to employees (Figure 2). Any changes in tier and salary based on achievement in the first cycle would go into effect in March 2018.
At the beginning of the first cycle, employees and managers were provided with an early prototype of what was later named the “Readiness Tool.” This tool was designed to guide conversations between employees and managers about whether employees were ready, or at an appropriate point in their career, for Tier Advancement. Employees and managers would complete this tool together by reading the summary expectations for each level of each competency. They would then mark whether each competency was part of the employee’s job and what level they believed the employee to be at. By tallying up the points at the end of the tool, the manager and employee could determine the employee’s readiness for Tier Advancement. This tool was further refined throughout the process and is shown in Figure 3.
On the WE-R website, a section for Tier Advancement was created and is publicly accessible so employees and managers can identify and obtain items needed for the Tier Advancement process, such as the Readiness Tool, Assessments, and Scoring Guides. Most assessments are available in their entirety in Microsoft Word format so they can be completed electronically or on paper. Scoring Guides were provided for most assessments to ensure transparency of assessment criteria to employees and managers. For knowledge tests, scoring guides were not available—instead, 5-10 knowledge objectives were published. The knowledge objectives describe what would be covered by each test with the intention that employees clearly understand expectations at each competency level. Assessments and associated scoring guides can be viewed on the Duke WE-R website (https://medschool.duke.edu/node/97565).
Prior to the Tier Advancement cycle, a single point of contact (SPOC) was designated by leadership within each clinical research unit (CRU), which is the staffing organization for all CRPs at Duke. The SPOC is responsible for: 1) ensuring staff are eligible for advancement, 2) submitting the staff’s information to WE-R when they are ready for Tier Advancement, 3) disseminating Tier Advancement assessment results to individual employees and their managers, 4) educating newly hired staff on the Tier Advancement process, and 5) ensuring that communication regarding changes and updates to the Tier Advancement process is shared within the unit. Regionalizing contacts within each unit allowed for appropriate assignment of assessments and streamlined communications about the cycles and process.
The WE-R team used a web-based data collection tool, REDCap (Research Electronic Data Capture [Harris et al., 2009]), for managing data related to the Tier Advancement process. Initially, the database housed data regarding results and assessments for managers and employees. Later, the information expanded to include the collection of nominations for Tier Advancement, dissemination of centrally scored assessments, and administration of proctored testing. The use of REDCap also assisted in streamlining notifications to managers and employees via automatic emails that could be disseminated based on programming within the system. It reminded employees and managers when assessments were due, included links to surveys that needed to be completed for data collection purposes, and allowed attachments of completed forms for employee and manager records.
Employee Testing and Manager Scoring
Once an employee officially entered into the Tier Advancement process, they and their manager were given a deadline by which all assessments must be completed, scored (if appropriate), and uploaded. Employees completed the self-report assessments and returned the information to their managers for scoring and determining competency level (Fundamental, Skilled, Advanced). Employees scheduled time with their manager to complete the appropriate direct observations of their competency demonstration. Assessments that were scored or handled centrally (proctored knowledge test, proctored case studies, and centralized review) had earlier deadline dates, as the WE-R team had to assemble experts for scoring. At the close of the cycle, managers electronically recorded the level achieved for each competency, and uploaded a single file of all assessments and scoring sheets into REDCap. A WE-R team member entered the level achieved for each centrally-scored assessment. Scores in REDCap were audited by the WE-R team to ensure they accurately reflected the documentation submitted.
Committee Review for Advanced Tiers
As mentioned previously, advancement to Tier 3 required an additional step—portfolio review by committee. In earlier cycles of the Tier Advancement process, WE-R requested portfolios without giving much guidance beyond asking for items that demonstrated leadership and contribution. Later, WE-R provided the employees with standardized guidelines for their portfolio packet submission. These guidelines were developed out of conversations with the committees about what content would best enable them to review information and make effective advancement decisions. The current portfolio guidelines require a standardized narrative template, an updated CV, full submission of any competency assessments, additional documentation that describes their contributions to their unit, division, department, or institution, and at least two letters of recommendation for advancement.
Portfolios packets were sent to committee members ten business days in advance of the committee meeting date. Committees were comprised of senior employees in clinical research across the Duke Schools of Medicine and Nursing. A WE-R team member attended to ensure the criteria for Tier Advancement was applied consistently and assist with interpretation of the criteria and the process. An HR representative attended to ensure that inappropriate considerations were not included in the discussion, such as performance ratings and informal character assessments. Each employee’s portfolio was assigned a primary reviewer, who reviewed the candidate’s portfolio and assessments in-depth, and made a verbal recommendation to the committee. The reviewer was typically assigned based on: a) being from a unit different from the candidate, b) having a role somewhat congruent with the candidate (e.g., Regulatory Coordinator Senior reviewing a potential Regulatory Coordinator, Tier 3 candidate), and c) having a research portfolio similar to the candidate. After presenting an overview of the candidate and recommending advancement (or not), the committee was able to ask questions and ultimately vote. If a candidate received a majority of favorable votes, they would advance to the tier they applied for. HR and WE-R members did not vote, unless one of the committee members had to recuse themselves due to a bias. If a candidate was not advanced to Tier 3, the committee worked together to provide constructive feedback as to how this employee might build a more competitive portfolio to be considered at a later date. This feedback was shared with the manager when results were disseminated.
Note that if employees did not meet the required 84-point threshold for Tier 3 applications, their portfolio packet was not reviewed by the committee during that cycle. However, it could be reviewed in future cycles after the point threshold was achieved.
Much debate has occurred within the communities involved on whether an appeals process should be created for those who did not successfully advance. To minimize the involvement of WE-R in performance issues, appeals have been placed entirely within the domain of HR and outside of WE-R or committee determination.
Collecting Feedback
After each cycle of Tier Advancement, but prior to sharing individual employee results of the process, WE-R collected feedback from managers and employees. After receiving survey responses, WE-R conducted focus groups to obtain feedback on how the process and assessments might be improved in future cycles.
Disseminating Results
Results were shared with each unit’s SPOC. For each candidate, SPOCs were provided with the outcome tier and a summary report indicating the resulting level (Fundamental, Skilled, Advanced, Expert) for each competency assessment. A depiction of the summary report can be seen below in Figure 4. If an employee did not advance to the desired tier, a WE-R team member had a one-on-one conversation with the manager prior to the dissemination of results. During this discussion, the manager was provided with talking points and was given an opportunity to ask questions related to the employee’s unsuccessful attempt. The School of Medicine Human Resources collaborated with central Compensation and Payroll to calculate and implement any salary changes due to the advancement, and communicated to business and departmental HR managers, who then disseminated information to managers and employees as appropriate.
Support and Communication
WE-R recognized that providing tools and navigation throughout the Tier Advancement process would be important for employees and managers. To that end, several opportunities were offered for employees and managers to ask questions, receive guidance, and discuss challenges. Examples include a step-by-step “welcome to the process” packet, weekly in-person office hours, and a central e-mailbox. In addition, WE-R offered relevant training sessions for managers and employees (e.g., “difficult conversations”) in cases where the manager may not feel the employee is ready for advancement.
To facilitate communication and transparency, an easily accessible website was created where managers and staff could access materials relevant to the process. A series of town halls was conducted to discuss the implementation of this process and allow for questions. Both of these components, as well as the implementation of a unit situated SPOC, have been critical in keeping the research community engaged and informed during this process.
Outcomes
Participation
Duke’s CRP workforce typically encompasses over 800 staff, with more than half of those staff in tiered positions. As of August 2019, four Tier Advancement cycles had been completed. During those cycles, SPOCs submitted 193 staff in CRC, Clinical Research Nurse Coordinator, or Regulatory Coordinator positions for participation in Tier Advancement. Of those, 149 CRPs completed Tier Advancement and 44 withdrew from the process. Table 2 indicates the number of employees in each position who completed Tier Advancement and notes what their target tier was.
Success Rates
As shown in Figure 5, 82% of Tier Advancement participants advanced at least one tier. One staff member in the CRNC job classification advanced from Tier 1 to Tier 3. Almost all with a target tier of 2 (89%) succeeded in achieving Tier 2 with a mean point total of 47 for all participants. Of those attempting Tier 3, 28 (70%) met the 84-point threshold with a mean point total of 86 for all participants and advanced to committee review of their portfolio; of those reviewed by committee, 63% advanced to Tier 3.
Overall, 23% withdrew from the Tier Advancement cycle after beginning the process. These withdrawal numbers reflect the inclusion of staff who left Duke, staff who were re-classfiied into other positions, as well as those who did not complete Tier Advancement. One commonly expressed reason for withdrawing from Tier Advancement was that the employee did not have enough time to complete the assessments during that cycle. Employees retain points from completed assessments from one cycle to another, so employees who withdraw do not have to repeat assessments unless the assessment changed significantly between cycles.
Revising Assessments and Processes in Response to Feedback
Following each cycle, feedback was gathered about the process via surveys and focus groups. In the survey, managers and employees who completed the Tier Advancement process were asked whether the process allowed them to exhibit their (or their employee’s) competence as a research professional using a rating scale of 1 (no confidence) to 10 (complete confidence). After the first cycle, employees demonstrated moderate levels of confidence (mean = 5.2) and managers expressed moderately high levels of confidence (mean = 6.1). After the most recent cycle, confidence levels for both groups increased slightly; employees’ mean confidence score increased to 5.4 and managers’ mean score increased to 6.8. Further process improvements showed a decrease in the average time spent by each employee completing assessments, as shown in Figure 6.
Participant feedback encouraged several changes, such as moving to shared cloud-based folders for managing assessments, scoring guides, and portfolios, creating online training modules about the Tier Advancement process, combining assessments and scoring guides into a single form to reduce unnecessary document management, and moving proctored testing administration into REDCap versus the Learning Management System. Due to a lack of attendance and the mobile nature of the coordinator job, office hours were changed from in-person to virtual, which allowed for greater accessibility and easier scheduling. Welcome Packets for managers and employees were made available via the website. These packets provide additional information on the process, responsibilities, scoring, and timelines for Tier Advancement. They were developed in response to feedback from managers and employees regarding a limited understanding of the Tier Advancement process steps.
By cycle four, the process was well-established and few new opportunities for optimization were presented. Optimizations made during the first three cycles included: 1) limiting each assessment to a single mode, except for a few Advanced and Expert-level assessments, 2) streamlining data collection and communication processes, and 3) harmonizing length for knowledge assessment tests.
As indicated previously, guidance regarding the portfolio has changed over time. These written guidelines, which better articulate the institution’s view of leadership and professionalism, have launched a series of professional development sessions hosted by the Research Professionals Network at Duke. The series covers strategies for portfolio development, mentorship, and career development.
Conclusions/Reflections/Future Steps
The design and implementation of a broad competency-based career advancement model was a lengthy, multilayered, dynamic process involving numerous stakeholders, from clinical research professionals to human resources. Without the involvement of these stakeholders from day one, this process would not have been successful.
Timing is Important
The Tier Advancement process was developed while still in the process of mapping Duke’s clinical research workforce into a standardized set of well-defined, tiered job classifications. Completing the job mapping prior to launching the Tier Advancement process was critical for two reasons. First, the concept of clinical research competency domains being used to define the aspects of CRP jobs needed to be socialized among the research community. Second, by mapping staff into appropriate classifications, each employee would proceed through advancement from a common baseline. Likewise, implementing a structured and objective advancement process after mapping was important to replace outdated seniority or familiarity-based advancement norms. While not all employees were satisfied with their mapped positions, the opportunity to drive their own advancement, shortly following mapping, ameliorated some of that dissatisfaction. Early in the process, two advancement cycles were offered each year to allow ample opportunities for all employees to attempt advancement. This has shifted to an annual cycle in 2020 to reduce the administrative burden on the WE-R team, managers, HR, and business managers. An annual cycle should also provide a more realistic pace for employees to complete assessments while still maintaining a frequency that allows sufficient opportunity to advance.
Culture Change Requires Significant Proactive Planning and Change Management
An employee-driven system of advancement based on structured demonstration of competency is entirely different from any system that had previously been in place within Duke University. Historically, advancement was determined by very individual factors within specific groups, based on education, seniority, budget, relationships, and local leadership opportunities. While investigators and leadership across the institution fundamentally understood the importance of equity and growth opportunities within the workforce, the specific impacts of these various advancement modalities in undermining equity and quality were not universally recognized. Therefore, it was crucial to have early and frequent conversations with staff, managers, and research leadership about movement and progression. Multiple town halls were held throughout the process, both during development and implementation of Tier Advancement. Planning committees included faculty, staff, and administration, all with unique viewpoints that proved critical in developing the process and in generating buy-in across these stakeholder communities.
Difficult Conversations and Manager Support are Needed
It is common for managers to struggle with having difficult conversations with employees. In the context of this project, managers struggled to address readiness for advancement. HR was engaged to provide regular difficult conversation training with managers during each cycle. The Readiness Tool allows managers to objectively walk employees through advancement expectations, which alleviates some of the anxiety related to having these difficult conversations. A training module was created, and is available to all managers, to assist in standardized scoring of self-report assessments. Lastly, individual conversations are held with each manager when an employee does not meet the requirements for advancement before general results are released. All of these tools provide managers opportunities to build skills and ask questions that may assist them in having these conversations. The continued development of manager-facing training sessions on specific aspects of the process are crucial to help them navigate discussions with their employees and more easily address their assessment requirements.
Performance and Education are Separate from Demonstration of Competency
The idea of “competencies” is not a new concept but has not been widely used among the clinical research workforce prior to the release of the Joint Task Force recommendations in 2014 (Sonstein et al., 2014). While competency-based clinical research certifications, conferences, and presentations are a consideration, they may not have direct impact on competency demonstration. There is a fine balance between attendance and takeaway. Staff are encouraged to not only attend professional development courses but take the knowledge and apply it to the research being conducted, thus demonstrating competence. However, the recognition of time and effort that goes into earning and maintaining a directly applicable professional certification has been added as a part of the currently operating Tier Advancement process.
The WE-R team struggled with the perceived conflation of “doing my job well” through performance and demonstration of specific competencies. While work has been done to socialize the ideas of competency domains through town halls and inclusion of resources on the WE-R webpage, the distinctions between job performance, educational pursuit, and competency demonstration continues to be a challenge. Moreover, the idea that one can be performing their job extremely well, yet still not be performing work commensurate with a high tier level is another confounding factor that can lead to dissatisfaction with results. Limits have been set on the advancement process so that employees who are struggling with performance, as indicated by a “needs improvement” during their most recent performance evaluation, are not eligible for Tier Advancement until these performance issues have been resolved. Managers are encouraged to discuss Tier Advancement during their annual performance conversations and use the Readiness Tool to guide goal setting. The goal is to make competency evaluations as clear and objective as possible so that the process is not used to address performance issues.
Tracking Results Can Lead to Better Training and Resources
In the past, clinical research training has been focused primarily on the principal investigator (Calvin-Naylor et al., 2017). Now, the Tier Advancement process has provided a wealth of information that can be utilized to continually improve Duke’s workforce and create effective training opportunities that advance competency among our clinical research staff (Association of Clinical Research Professionals, 2018). There is now transparency into strengths and weaknesses across competency areas, as demonstrated by Figure 7. The figure displays all staff members who tested for a specific competency within the Research Operations domain, submitted the assessment, and scored either Fundamental, Skilled, or Advanced. By continually evaluating assessment results, new educational opportunities, experiences, and training to address weak areas within the workforce can be created. Competency profiles across divisions and therapeutic areas can be tracked and shared with leadership in those groups to both share strengths or address weaknesses before large problems develop. It is too early to measure the effect of this targeted training on the clinical research community. Future plans to measure how these trainings have affected site quality, such as enrollment rates, avoidable protocol deviations, and audit findings, are in development.
Tier Advancement Has Empowered the Research Workforce
Early data suggests that the Tier Advancement process has positively impacted retention among CRPs at Duke (Stroo et al., 2020). As demonstrated in Figure 8, there is an upward trend of CRPs remaining at Duke for more than two years. The workforce response has been incredibly positive about the experience. Clinical research professionals at Duke are pleased that they can now grow their careers through competency development and appreciate the creation of an objective professional pathway where one did not previously exist. Ultimately, this advancement process provides an opportunity for managers and employees to jointly develop a transparent plan for career advancement. Staff have the opportunity to carve a professional pathway for themselves and envision a long-term career at Duke.
Paving the Way for Career Advancement at Academic Medical Centers
Much of the competency alignment and career advancement work that has been done by well-known clinical research associations, such as ACRP and SOCRA, is focused on industry based clinical research professionals, which does not always translate directly to the CRPs within an academic medical center. In an effort to promote adaptation, rather than re-invention, all of the work developed on Tier Advancement and the mapping of clinical research staff into competency based positions (Brouwer et al., 2017) has been published to the Duke WE-R website (https://medschool.duke.edu/node/97565). Duke has created strong partnerships with the University of Alabama Birmingham as they have utilized our framework to move their clinical research staff into competency-based job classifications under their Clinical Research Career Ladder. Many other academic medical institutions (including Johns Hopkins University, Medical University of South Carolina, MD Anderson Cancer Center, Ohio State University, University of Michigan, Boston University) have called upon the WE-R team to assist in implementing portions of the competency framework. This initiative is just the beginning of creating a larger workforce change for clinical research professionals at academic medical centers.
Duke recognizes both the opportunities and challenges associated with launching an overhaul of clinical research professional job classifications with an accompanying annual commitment for Tier Advancement. This project is more than reworking job descriptions and professional advancement. WE-R is a commitment for clinical research managers and institution administration to work together with clinical research staff in supporting competency and job growth amongst the clinical research workforce at Duke. This partnership, trust, and engagement with clinical research staff is necessary to produce the highest quality research. The clinical research infrastructure owes it to the participants, investigators, sponsors and Duke to employ a committed, high-quality clinical research workforce. Clinical research changes health care—without it, there is only standard of care. There is a need to work together to support the development of the workforce responsible for advancing clinical research efficiently and safely. WE-R (and other competency-based framework adoption) is an important step in understanding how clinical research is advanced to benefit everyone.
Authors’ Note
The authors would like to acknowledge funding for this initiative from Duke’s CTSA grants UL1TR002553 and UL1TR001117, as well as contribution and leadership from the following individuals: Billy Newton, Betsy Hames, Mark Stacy, David Smithwick, Adrian Hernandez, Donna Crabtree, Bryan Feger, Research Practice Managers and Directors of Clinical Research Units, and the countless research professionals and faculty that contributed to this work.
Data were collected and managed using REDCap electronic data capture tools hosted at Duke University School of Medicine (Harris et al., 2009). REDCap (Research Electronic Data Capture) is a secure, web-based software platform designed to support data capture for research studies, providing: 1) an intuitive interface for validated data capture; 2) audit trails for tracking data manipulation and export procedures; 3) automated export procedures for seamless data downloads to common statistical packages; and 4) procedures for data integration and interoperability with external sources.
Christine Deeter
Project Planner I
Duke University School of Medicine, Duke Office of Clinical Research
2200 West Main St. Box 21
Durham, NC, 27705, United States
(919) 681-0922
christine.deeter@duke.edu
Deborah Hannah
Research Practice Manager
Duke University School of Medicine, Department of Dermatology
Camila Reyes
Analyst Programmer I
Duke University School of Medicine, Duke Office of Clinical Research
Michelle E. Mack, PhD
Associate Director for Research Operations - Outreach
Duke University School of Medicine, Duke Office of Clinical Research
Marissa Stroo
Director, Clinical Research Operations - Data
Duke University School of Medicine, Duke Office of Clinical Research
Stephanie A. Freel
Director, Clinical Research Education and Outreach
Duke University School of Medicine, Duke Office of Clinical Research
Rebecca J. Namenek Brouwer
Director, Research Initiatives
Duke University
Heather E. Gaudaur
Project Manager, Rewards and Recognition
Duke University and Duke University Health System
Andrea L. Doughty
Assistant Director, Compensation
Duke University
Duke University School of Medicine, Duke Office of Clinical Research
Correspondence concerning this article should be addressed to Christine Deeter and Denise Snyder, Duke University School of Medicine, Duke Office of Clinical Research, 2200 West Main St. Box 21, Durham, NC, 27705, United States.
References
Abshire, M., Dinglas, V. D., Cajita, M. I. A., Eakin, M. N., Needham, D. M., & Himmelfarb, C. D. (2017). Participant retention practices in longitudinal clinical research studies with high retention rates. BMC Medical Research Methodology, 17(1), 30. https://dx.doi.org/10.1186/s12874-017-0310-z
Association of Clinical Research Professionals. (2018, June 11). 5 clinical research trends emerge at ACRP 2018 [Blog post]. https://acrpnet.org/2018/06/11/5-clinical-research-trends-emerge-at-acrp-2018/
Baer, A. R., Zon, R., Devine, S., & Lyss, A. P. (2011). The clinical research team. Journal of Oncology Practice, 7(3), 188-192. https://doi.org/10.1200/JOP.2011.000276
Brandt, D. S., Bosch, M., Bayless, M., Sinkey, C. A., Bodeker, K., Sprenger, K., Johnson, K., & Gilmore, J. M. (2011). A CTSA-sponsored program for clinical research coordination: Networking, education, and mentoring. Clinical and Translational Science, 4(1), 42-47. https://doi.org/10.1111/j.1752-8062.2011.00259.x
Breza, E., Kaur, S., & Shamdasani, Y. (2018). The morale effects of pay inequality. The Quarterly Journal of Economics, 133(2), 611-663. https://doi.org/10.1093/qje/qjx041
Brouwer, R. N., Deeter, C., Hannah, D., Ainsworth, T., Mullen, C., Hames, B., Gaudaur, H., McKellar, T., & Snyder, D. C. (2017). Using competencies to transform clinical research job classifications. Journal of Research Administration, 48(2), 11-25.
Califf, R. M. (2009). Clinical research sites—the underappreciated component of the clinical research system. JAMA, 302(18), 2025-2027. https://doi.org/10.1001/jama. 2009.1655
Calvin-Naylor, N. A., Jones, C. T., Wartak, M. M., Blackwell, K., Davis, J. M., Divecha, R., . . . Luzuriaga, K. (2017). Education and training of clinical and translational study investigators and research coordinators: A competency-based approach. Journal of Clinical and Translational Science, 1(1), 16-25. https://doi.org/10.1017/cts.2016.2
Center Watch Staff. (2015, May 1). Job satisfaction mixed as workload increases. The CenterWatch Monthly, 22(5). https://www.centerwatch.com/articles/ 16686#:~:text=Yet%20one%2Dthird%20of%20respondents,and%20nearly%2020%25%20expressed%20dissatisfaction.&text=Nearly%20three%2Dquarters%20of%20respondents,past%20three%20to%20five%20years.
Getz, K. (2018, June 7). Has the study coordinator landscape reached a tipping point? https://forteresearch.com/news/study-coordinator-landscape/
Gwede, C. K., Johnson, D. J., Roberts, C., & Cantor, A. B. (2005). Burnout in clinical research coordinators in the United States. Oncology Nursing Forum 32(6). https://doi.org/10.1188/05.ONF.1123-1130
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009). Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377-381. https://doi.org/10.1016/j.jbi.2008.08.010
Owens Pickle, E. E., Borgerson, D., Espirito-Santo, A., Wigginton, S., Devine, S., & Stork, S. (2017). The Clinical Research Associate Retention Study: A report from the Children’s Oncology Group. Journal of Pediatric Oncology Nursing, 34(6), 414-421. https://doi.org/10.1177/1043454217723861
Sonstein, S. A., Namenek Brower, R. J., Gluck, W., Kolb, H. R., Aldinger, C., Bierer, B. E., & Jones, C. T. (2018). Leveling the joint task force core competencies for clinical research professionals. Therapeutic Innovation & Regulatory Science, 54(1), 1-20. https://doi.org/10.1007/s43441-019-00024-2
Sonstein, S. A., Seltzer, J., Li, R., Silva, H., Jones, C. T., & Daemen, E. (2014, June). Moving from compliance to competency: A harmonized core competency framework for the clinical research professional. Clinical Researcher, 28(3), 17-23. https://doi.org/10.14524/CR-14-00002R1.1
Speicher, L. A., Fromell, G., Avery, S., Brassil, D., Carlson, L., Stevens, E., & Toms, M. (2012). The critical need for academic health centers to assess the training, support, and career development requirements of clinical research coordinators: Recommendations from the Clinical and Translational Science Award Research Coordinator Taskforce. Clinical and Translational Science, 5(6), 470-475. https://doi.org/10.1111/j.1752-8062.2012.00423.x
Stroo, M., Asfaw, K., Deeter, C., Freel, S. A., Brouwer, R. J., Hames, B., & Snyder, D. C. (2020). Impact of implementing a competency-based job framework for clinical research professionals on employee turnover. Journal of Clinical and Translational Science, 1-5. https://doi.org/10.1017/cts.2020.22
#JournalofResearchAdministration#VolumeLINumber2#CareerDevelopment#competency#WorkforceDevelopment#clinicalResearch#Professionalism