Volume XLIX, Number 2
Authors:
- Alyson E. Becker, MPH Sanford Research, Population Health
- Jessica Heinzmann, BA Sanford Research, Population Health
- DenYelle Baete Kenyon, PhD Sanford Research, Population Health, University of South Dakota, Sanford School of Medicine, Department of Pediatrics
Introduction
NIH Research Funding
Funding and grants are becoming increasingly difficult to obtain (Daniels, 2015; National Institutes of Health, 2017d). NIH funding for studies and projects have evolved over the years to promote and encourage different types of researchers to apply. The NIH alone has 240 distinctive funding mechanisms through the organization (National Institutes of Health, 2016). However, in the research community, the recognized standard of an independent researcher is receiving an NIH R01 grant (Daniels, 2015; Levine, 2007; Tragesser, 2011), which is NIH’s earliest and oldest funding mechanism (National Institutes of Health, 2016a). This increasingly competitive grant (and its grant equivalents) only had a 20% success rate for those who applied in 2016 (National Institutes of Health, 2015c). Therefore, it is imperative to have a quality study and accompanying preliminary data to apply for an R01 grant, particularly for first-time R01 applicants.
As a part of their “Next Generation Researchers Initiative,” which was implemented in 2017 to encourage independent research careers, the National Institutes of Health (NIH) modified the definitions of the stages of career researchers (National Institutes of Health, 2017d). One area of interest to the NIH, and a main focus of the “Next Generation Researchers Initiative” (National Institutes of Health, 2017d) is the development of early stage investigators, which the NIH defines as,
A Program Director / Principal Investigator (PD/PI) who has completed their terminal research degree or end of post-graduate clinical training, whichever date is later, within the past 10 years and who has not previously competed successfully as PD/PI for a substantial NIH independent research award. (National Institutes of Health, 2017c).
Disparities of NIH Grant Awardees
Early stage investigators can be of any age, race, and gender, but it is increasingly difficult for any early stage investigators to secure significant funding, such as an R01-equivalent grant, from the NIH. In 2016, for applications where the contact Principal Investigator (PI) was a first-time investigator, the success rate was only at 16%; this is down from 23% in 1998 (National Institutes of Health, 2015b). Of NIH R01-equivalent grant applicants in 2016, only 32% were applying for the first time, which is down from 39% in 1998 (National Institutes of Health, 2015b).
The NIH has tried to lessen the disparity between first-time and established researchers through several methods. One attempt by the NIH suggested imposing a funding limit for those with labs that have the equivalent of three R01 grants (Kaiser, 2017b; National Institutes of Health, 2017b). That policy, however, came with backlash as some viewed it as limiting productive labs (Kaiser, 2017a). Another recent effort aimed to assist with the development of early-stage and early-established investigators is the “Next Generation Research Initiative,” launched by the NIH at the end of August in 2017, which aims to support an additional 400 researchers by restructuring classifications and adopting policies to promote diversity (National Institutes of Health, 2017d).
The majority of NIH R01 and equivalent grant recipients are white, above 40 years old, and male (Daniels, 2015; National Institutes of Health, 2015a, 2017c). Similar to early stage investigators, underrepresented researchers (e.g. racial minorities) experience struggles in obtaining funding, but there is less research about the distribution of race/ethnicity and NIH R01-equivalent grants. Hayden (2015) reported in Nature that every year from 1985 to 2013, underrepresented racial minorities received NIH funding at 78-90% the rate of other races. NIH award rates have been on the downward trend overall, but the disparity still exists. In 1985, the NIH award rate for R01 and equivalent grants was at 48.6% for Whites and 42.1% for non-Whites, while that decreased to 23.3% for Whites and 19.3% for non-Whites in 2013 (Oh et al., 2015).
Sadly, although Ginther et al. (2011) may be dated, it is a widely referenced source of NIH grant awardee race and ethnicity data, including specific race and ethnicity data from 2000–2006. It is disheartening to find that AI researchers are not well represented among the pool applying for R01-equivalent grants, and warrants a focus on identifying those potential applicants, awardees, and the overall research pipeline that develops AI investigators.
AI researchers represented 0.1% of employees in the science field in 2015 (National Science Foundation, 2017), so it is evident that the underrepresentation not only exists for NIH funding but throughout the industry (National Science Foundation, 2017). Minority researchers (racial minorities and women) face several barriers in building successful science careers, including receiving funding (Kameny et al., 2014). Four common barriers, as identified by Kameny et al. (2014), are institutional, cultural, skills and personal. Institutional barriers can be significant in stifling successful careers as they consist of lacking in research support, insufficient mentoring, and work politics (Kameny et al., 2014). Institutional barriers, combined with cultural barriers minority researchers experience, can place additional burdens on developing minority researchers (Kameny et al., 2014).
The NIH is working on addressing those barriers and increasing workforce diversity through several mechanisms. A national effort by the NIH is the Scientific Workforce Diversity Toolkit (n.d.) which provides guidance on how to increase workforce diversity through diversifying the talent pool, performing unbiased talent searches, outreach and networking, and mentoring relationships. Other efforts, not on a national level, include specific programs, such as The Native Investigator Development Program, which aims to assist AI/AN investigators in career development (Manson, Goins, & Buchwald, 2006).
As noted in previous literature (Manson et al., 2006), meaningful and reliable information on AI researchers is lacking. This was evident in that finding research literature outlining the lack of minority researchers in itself was not difficult; however, finding research that explicitly discussed AI researchers, particularly those who have received NIH grant funding, was next to impossible. Therefore, it is evident additional workforce development funding should be invested in building a cadre of AI researchers.
Purpose of Pilot Grants
One type of funding that is commonly used for the development of early stage investigators are pilot grants. PGPs are a unique funding mechanism that can help provide a research development opportunity to early stage investigators by providing funding to collect initial data for applying for a larger, future research project (National Institutes of Health, 2016b). The NIH provides funding for a Pilot Research Project (2016b), but many other organizations and universities have their own pilot grant programs (PGPs) that provide funding opportunities to research specific interests to that institution, leading PGPs to cover a myriad of subjects, from biomedical to social and behavioral research.
PGPs provide funding to diverse areas of research to assist investigators in testing out new and innovative methods while collecting preliminary data to use for further grant funding and research (Doody & Doody, 2015; van Teijlingen & Hundley, 2001). The process of writing, applying, and receiving a pilot grant leads to an increased, competitive experience in future applications (Moore, Carter, Nietert, & Stewart, 2011). The eventual goal is to guide the research trajectory of early stage investigators into empowering them to do non-pilot project studies and gain independence as a researcher. PGPs also provide capacity-building opportunities to further develop the researcher’s skills necessary for performing future studies, and therefore displaying the scientific rigor of the investigator (Moore et al., 2011).
As other funding mechanisms, such as corporate funding or organizational pilot grants, are becoming an attractive source of funding for early-stage and underrepresented investigators (Jahnke, 2015), the Collaborative Research Center for American Indian Health (CRCAIH) decided to dedicate funds to starting an organizational PGP. Motive for incorporating the PGP in CRCAIH included providing experience with grant writing and overall building confidence about the grant process for underrepresented investigators. The process outlining the CRCAIH PGP is described below.
Collaborative Research Center for American Indian Health
Compared to the rest of the country, South Dakota (SD) has a higher percentage of the population that identify as AI; approximately 9.0% of the population in SD are self-identified as AI, compared with 1.3% in the United States (United States Census Bureau, n.d.). This led Sanford Health, the largest employer and health care provider in the Dakotas, to try to address the issue from an organized and collaborative state and regional effort. In 2012 Sanford Research, a non-profit research organization within Sanford Health, applied for and received a five-year, $13.5 million grant from the NIMHD to start CRCAIH, (pronounced “KIRK-uh”), or the Collaborative Research Center for American Indian Health, which at the time was the largest grant ever received by Sanford Research (Elliott et al., 2016).
CRCAIH’s overall goal is “to build tribal research infrastructure and transdisciplinary research teams to improve American Indian health through examination of social and environmental influences on health” (CRCAIH, 2017a). The organizational structure of CRCAIH (see Figure 1) supported that goal in many different ways: the cores and divisions, the three large research projects, and the PGP.
CRCAIH funded several regional tribal partners to build their infrastructure for research in various ways, mainly focusing on building their research regulation capacity through tribal codes, establishment and growth of research review boards, and related policies and procedures.
CRCAIH is comprised of three cores (Culture, Science, & Bioethics; Regulatory Knowledge; and Methodology) and two divisions (Administrative and Community Engagement & Innovation). The creation of CRCAIH was not to be a single effort to expand the research knowledge of AI health disparities, but rather to be a common platform to provide communication and infrastructure to unify efforts through partner tribal nationals, research institutions, and healthcare organizations (Elliott et al., 2016).
The original aims of CRCAIH were to:
- Establish strong relationships needed for tribal research on AI health disparities
- Provide capacity-building assistance to help tribes create and manage research in the future
- Perform three studies on regional AI health issues
- Fund and maintain a PGP to research health disparities among AIs (CRCAIH, 2017a)
CRCAIH was successfully able to address each aim during the initial funding period, including the PGP, which lasted for the duration of initial five years of CRCAIH funding (CRCAIH, 2017d)
However, through the NIH/NIMHD Transdisciplinary Collaborative Center grant (National Institute on Minority Health and Health Disparities, n.d.) that funded CRCAIH, there are no pre-established procedures to follow for creating a NIH-funded pilot grant program. Therefore, CRCAIH supported research projects, cores, divisions, and developed the PGP collectively, from the beginning.
This paper aims to describe the CRCAIH PGP and its role in developing underrepresented investigators, including AI researchers and early-stage investigators, and adding literature to the knowledge gap of AI health research. The process of developing the CRCAIH PGP and sharing lessons for other organizations considering starting a PGP will also be discussed.
CRCAIH Pilot Grant Program Process
As a part of the Administrative division, the PGP was a significant undertaking of CRCAIH. The CRCAIH PGP had two specific aims: (1) provide a funding mechanism for the formation of transdisciplinary research teams within North Dakota, South Dakota, and Minnesota, to initiate research on significant health issues for AIs in the Northern Plains, and; (2) create a rigorous independent peer review process to provide the selection of quality pilot projects in line with CRCAIH goals and identified priority areas and to provide useful feedback to all submitting investigators to help improve future grant submissions (CRCAIH, 2017d).
It took less than a year to plan the PGP as the first Request for Applications (RFA) had a 2013 Spring release date, after funding started in September 2012. There were four separate rounds of funding pilot grant projects, each for a maximum of one year and $100,000 in direct funds per project. The CRCAIH PGP process followed a fairly standard order that took approximately one year to complete for each round of funding. Table 1 outlines the PGP process and the amount of time allotted for each step for applications from release RFA to the beginning of the pilot grant funding.
Release of RFA: CRCAIH released an RFA for each round of funding that detailed the specific sections and requirements in submitting an application for the PGP. The RFA was not a rigid document, but rather fluid to outline changing priorities and feedback. Significant changes in RFA include, from round 1 to round 2, expanding the application period from 8 to 14 weeks, due to feedback about potential applicants wanting additional time for grant preparation. Often, NIH (2017a) and other federal grants release the funding announcement only 6-8 weeks before the deadline, so CRCAIH Administrative division and the Pilot Grants Subcommittee felt this time period was acceptable.
Other changes in the RFA throughout the CRCAIH PGP highlighted the increased importance of community-based participatory research (CBPR) and Tribal Research Priorities. For example, in the RFA from round 1 to round 2, Letters of Support went from “Recommended” to “Required” for any collaborating or tribal partner included in the CRCAIH PGP application. The importance of changing CBPR from “Recommended” to “Required” stemmed from the commitment of CRCAIH to not only add knowledge to AI health disparities research, but to ensure that the researchers were building strong relationships with the tribes involved in their pilot studies, relating back to Aim 1 of CRCAIH.
Another significant change in the RFA occurred between 2014 and 2015. In order to show CRCAIH’s dedication to AI health and health disparities, and entire section called “Tribal Health Research Priorities” was added along with examples of what those types of projects might look like. Tribal priorities were also highlighted in the 2015 RFA (round 3) by the addition of the “Tribal Approvals” subheading in the Human Subjects section that outlined how appropriate Tribal Approvals would be required before any funding would be received.
Other updates may not have been as direct as adding the sections on tribal health, but nonetheless were important in evolving the RFA to provide as much relevant information as possible. The resources were updated every year to provide relevant information, and between round 1 and round 2, applicants were required to submit the narrative of their application in a Microsoft Word™ document to make sure it fit the page requirements.
Letter of Intent: The Letter of Intent (LOI) was not a required document when submitting for PGP funds, but strongly recommended in the RFA. LOIs are a common practice when applying for grant funds. Specifically, for the CRCAIH PGP, it was encouraged as a way to draw in investigators in order to follow up and encourage them to connect with the CRCAIH cores and divisions for assistance in designing their study and preparing their application.
Application Due: During the first round of funding for 2013, an application form was available from the CRCAIH website as a fillable Microsoft Word™ document. That application was due by 5:00 pm CST to the Sanford Research Grants Office. Besides the Technical Assistance webinar held by the grants office, applicants did not require much other guidance when submitting the application.
Internal Grant Office Triage: The internal grants office (analogous to a sponsored projects office) conducted the first step of the application review with a checklist for completeness of the grant, adherence to the instructions, and eligibility of the organization and Principal Investigator. Only three submitted applications were triaged over the years and not sent on for external review due to reasons such as lateness in submission and research strategy extending past the page limit.
External Review: An important aspect to developing the CRCAIH PGP was the decision to have a rigorous review process. This was created to be similar to NIH review process to prepare CRCAIH PGP applicants for an NIH grant application and review process after their experience with the CRCAIH PGP.
To maintain objectivity, the external reviewers were not affiliated with CRCAIH or the applicant institutions. They were recruited by the lead of the PGP, and included colleagues from conferences and previous university affiliations, as well as referrals from several CRCAIH staff from their previous universities. Reviewers came from organizations spanning three time zones and two countries (e.g., Alaska, British Columbia, Arizona, and Alabama). To help bring a transdisciplinary perspective, various disciplines in community-based and minority health were represented, with at least half focusing on American Indian health. The reviewers were split evenly between early stage and senior investigators. The group benefited from stability across the years, with ten reviewers covering the eight slots over time, and with five reviewers involved all four years. Reviewers were paid a $1000 honorarium as a “thank you” for their time and commitment to a thorough review.
After reviewers had committed to the review, they were sent their assigned applications and a conflict of interest statement, which they signed and returned after confirming they were not in conflict with their assigned applications. After the first year, the date of review was chosen and reviewers confirmed they could attend before applications were sent out for review.
The first year of the program had the largest number of applications, and only a primary and secondary reviewer were assigned for each application. For the following years, CRCAIH moved to having three (Primary/Secondary/Tertiary) reviews of each application. The benefit of taking an average of three scores per application versus two was so there is less chance of positively or negatively skewed reviews. Reviewers’ scoresheets were due a week before the teleconference review so the combined scores could be calculated and rank ordered. After receiving reviewer feedback, the second year onwards, a private online file sharing space was used to upload files, and the other reviews were posted. Reviewers were encouraged to read other reviews before the teleconference to understand the other assigned reviewers’ perspectives and why their scores may differ.
The review was via teleconference and led by the Program Director/Chair of Pilot Grants Subcommittee. Each year, the meeting took no longer than 2 hours, with at least the top half of the applications discussed, with a vote at the top and bottom of the meeting to discuss any of the bottom applications. Additional comments that arose during the review were added to the detailed comment sheets from the reviewers and sent to the applicants to aid in improving their project for implementation or grant resubmission.
Funding Decisions: Shortly after the External Review, a PGP Subcommittee meeting was held. Although the details of the applications and reviews were not released to the members, they received the project abstracts and relative ranking of the top scoring projects. CRCAIH Administration discussed the aspects highlighted by the reviewers, including concerns. This lively discussion resulted many times in confirmation of the top scoring applications being funded, however also brought about change in funding an additional project the first year at a 6-month delay because the benefits of the project were strong, but to give the Project Lead more time to prepare revisions and for budgetary reasons. It was in the first year post-review subcommittee meeting where a concern about a proposed project’s buy-in/commitment from the tribal community was questioned, and the idea in future years to make the letter of commitment mandatory was established.
The applications were scored according to Figure 3 and the applications with the highest scores were funded. There were no preferences given to investigators based on their career stage. Proposed budgets could range from $25,000 to $100,000 for direct costs with indirect costs allowed at the applicant institution’s approved negotiated rate. With $1.2 million available to fund the PGP, the total number of projects supported depended on the budgets of the awardees. Most project proposed budgets were closer to the maximum amount (average proposed direct costs = $77,268; average total proposed budget = $103,744), leading to a varying amount of awardees for each round, as there was only a limited amount of funding available for the CRCAIH PGP.
Request for Just in Time: Emails were sent by the grants office to the selected applicants and their organizations, asking for items such as revised budget, Institutional Review Board Approvals, human subjects training certificates, photos for publicizing, FWA information, and “other support” documentation. After the project funding decisions were made, budgets were closely examined for items that could be trimmed, and often a reduced budget amount was offered to the applicants. Recipients were given approximately 2 weeks to return the materials back to the grants office.
NIH Review: Each year, after the complete materials were received by the grants office for all the recipients, the complete packages were sent to CRCAIH NIH Project Officer and Fiscal Contact for approval. The length of this review varied, and could extend to several weeks, therefore for subsequent years of the pilot grants program the application deadline was pushed earlier to account for the final approvals and to give awardees more time to secure IRB approvals.
Funding Begins: The amount that CRCAIH offered for each pilot grant is significantly larger than traditional pilot grants due to the community involvement and the FTE involved with employing a community liaison. Approximately 68% of CRCAIH funds went outside of CRCAIH core and division services to support community partners and projects, which includes the funds dedicated to the PGP. Although awarded as a one-year project, CRCAIH permitted awardees to carry over unspent funds into a second year if requested.
CRCAIH Resources
Support from Cores and Divisions: In order for CRCAIH to achieve their aims and to assist in the development of researchers, the cores and divisions of CRCAIH were available to the applicants as resources during the application process. CRCAIH advertised three modes of communication: email, phone number, and website. Applicants were encouraged to contact the core or division relevant to their question on the RFAs. General applicant questions went to the program director and were forwarded to the appropriate core or division. The questions and responses were documented and organized by round of funding (see Table 3, right-hand column).
Website: The website was not only a communication tool, but also a resource. Along with webinar recordings, the CRCAIH website housed the CRCAIH Frequently Asked Questions, or FAQs page (CRCAIH, 2017b). The page was developed through documentation of what questions potential applicants had when contacting the cores and divisions. The CRCAIH FAQ page addressed questions from several topics including general questions, application questions, approvals, partners, principal investigators, funding/budgets, and indirect costs/facilities & administrative costs (2017b). Providing these kind of thorough resources to the applicants led to greater capacity building for the researcher and their community partner, and to stronger applications.
Webinars: After the RFA was released, CRCAIH held webinars that were directly related to the PGP. The first year of the PGP, a webinar was held/recorded with representatives from the cores and divisions to focus on what types of assistance they could provide. Each following year, a Pilot Grant Pre-Application Technical Assistance webinar was held with the Program Director and representation from the Internal Grants’ Office. Those webinars were:
- 2014 Core Division Resources
- 2014 Pilot Grants Program Pre-Applications Technical Assistance
- 2014 Tips on Writing a Pilot Grant
- 2015 Pilot Grants Program – Building relationships in Community-based Research
- 2015 CRCAIH Pilot Grants Program Pre-Application Technical Assistance
- 2016 Pilot Grant Pre-Application Technical Assistance. (CRCAIH, 2017f)
Targeted Outreach: During the subsequent rounds of the application time periods, after the first round of pilot grant awardees, CRCAIH made targeted outreach a priority to encourage particular people to apply. This included previous applicants who were unfunded, particularly those who were close to the funding line, those applicants from tribally-based organizations, and AI investigators. This outreach was sometimes an email of encouragement, but oftentimes an in-person or phone meeting with representatives from the cores and divisions to discuss weaknesses raised by reviewers and recommendations for addressing the concerns. This targeted outreach may account for the rising percentages of applications from AI PIs, which ranged from 32% AI PI/Co-PI applicants the first year to 100% AI PI/Co-PI applicants the fourth year of the CRCAIH PGP.
Observations: Applications & Awardees
The number of awardees varied per year due to the quality of the application and the amount requested. A total of 58 applications were received and 15 projects were funded through all four rounds of the CRCAIH PGP, which is an overall success rate of 26%. The success rate varied by year, ranging from 20-40%. Although the overall success rate for CRCAIH was slightly higher than the NIH success rate of 18.8% for R01-equivalent grants during the same timeframe (2013-2016), CRCAIH did have a notable difference on race/ethnicity of Principal Investigators (National Institutes of Health, 2015c). There was no data found on race/ethnicity of NIH applicants found for the years 2013-2016, but previous numbers (Ginther et al., 2011; Kaiser, 2011) indicate that the number of AI PIs who apply and receive NIH R01-equivalent grants is incredibly low. In 2006, a total of 41 AIs were PIs on R01-equivalent grant applications, or merely 0.05% (Ginther et al., 2011). In the CRCAIH PGP, 29 AIs were listed as PIs on applications out of 58, or 50%. Table 2 outlines the demographics of the applicants over each of the four rounds of CRCAIH PGP funding, with the awardees in parentheses. Of the awardee PIs and Co-PIs, nine were AI (47%) and of the PI organizations, 27% were Tribal/Tribal Academic (CRCAIH, 2017d).
Table 3 lists the funded CRCAIH pilot studies by year (round), title, social determinant of health studied, and number of contacts the PI, Co-PI, or supporting staff made to CRCAIH on behalf of the grant application. Although contacting CRCAIH resources was not required for applying and receiving funding, with the amount of time and effort that went into developing the cores and divisions and their various resources, some individuals did choose to utilize those services to improve their applications. Out of the 15 awardees, 11 (73%) contacted CRCAIH about their PGP project for a total of 27 contacts cataloged for all awardees.
Continued Interactions
During the entire PGP process, pilot grant awardees were encouraged to continue utilizing the cores and divisions’ assistance. Subcontracts were established and awardees submitted quarterly reports detailing their project’s progress. Awardees were included in panel presentations and encouraged to present posters at the Annual CRCAIH Summit (CRCAIH, 2017c, 2017f). Because the panel presentations were only a snapshot of their project and findings, CRCAIH held an Annual Pilot Grant Program Seminar Series where awardees from each round were brought to Sanford Research to give a full one-hour presentation. Presentations were livestreamed and recorded for later archiving on the CRCAIH website (2017c). This presentation took place approximately 24 months after funding was officially received by the awardee. This allowed for sufficient time in analyzing data from their participation in the CRCAIH PGP. Advertisements for these presentations went out through the CRCAIH bi-weekly email newsletter. During these visits, CRCAIH arranged meetings with additional investigators to encourage collaboration, and with the cores and divisions to reignite ideas for utilization of their resources. This resulted in several new interactions, particularly in assistance with new quantitative and qualitative analyses with the Methodology Core and follow-up from CRCAIH’s NIH Project Scientist to encourage applications for specific mechanisms.
Dissemination/Return on Investment
Dissemination is an important part of any type of research and CRCAIH encouraged dissemination from all parts of the organization, including those who received funding from the CRCAIH PGP. Through the Annual Summit and Pilot Grant Program Seminar Series, CRCAIH provided a venue for formal academic presentation of pilot study findings reaching a broad audience. As for peer-reviewed scholarly output, awardees currently have nine manuscripts published or in press resulting from their CRCAIH pilot grants. Additionally, there are four more manuscripts under review or revise and resubmit with several more in preparation. CRCAIH shares links to recent publications (2017e) with our listserv as well as archiving them on our website.
Despite much emphasis placed on the necessity of dissemination of research results through peer-reviewed publications, the importance of getting research results and project-generated resources back to the community should not be overlooked. Community-based participatory research (CBPR) approaches, like those undertaken by CRCAIH pilot grant PIs, seek to involve the community as equitable partners in all aspects of the research process. One of the key principles of CBPR partnerships is the dissemination of findings to all partners and involving them in the dissemination process (Israel, Schulz, Parker, & Becker, 2008). A majority of PIs indicated they provided informal presentations or reports of pilot study results to the community in which they were working. Through collaboration with the Research Ethics And Dissemination (READ) Core of Sanford Research, one awardee is creating infographics for use in social media and print campaigns to disseminate findings to the community, taking into account cultural context.
A follow-up survey was administered to the PI of the 13 projects in the first three funding cycles. Eleven PIs responded to the survey, allowing further exploration of the impact of the CRCAIH PGP. Since their participation in the CRCAIH PGP, 82% (n=9) have submitted additional grants, including federal, state, and foundation grants. Forty-nine grant applications were reported by PIs in the years following their CRCAIH pilot grant. Some were reported more than once, reflecting a proposal which was resubmitted to a different funder or in multiple cycles. Applications for federal funding accounted for 73% (n=36) of those reported, with NIH funding mechanisms (n=24), reported most often. Other federal funding agencies targeted include: DHHS Office of Adolescent Health, SAMHSA, HRSA, CDC, and the Department of Justice. Nine grant applications (18%) were submitted to national and regional foundations (e.g. American Cancer Society, Bush Foundation, and Robert Wood Johnson Foundation). Although not always related to the topic of their particular pilot study, over 30 of these grant submissions were in the area of American Indian health research. Five (45%) indicated that they have submitted additional grants which utilized their pilot grant findings. The applicants’ roles on these grants ranged from PI, to evaluation director, to consultant. Overall, 17 of the 49 reported applications were funded.
Recently, a CRCAIH PGP awardee received sizable SAMHSA funding to build on the PGP study that was conducted in that community. Three awardees, two of which were early stage investigators, also submitted NIH R01 applications. One early stage investigator’s R01 was recently funded to continue her pilot grant work in that community; the other resubmitted her application in the next funding cycle. Another awardee submitted for a NIH U19 grant (unfunded). Two pilot grant awardees are currently Project Leads under the Center for Health Outcomes and Population Research CoBRE, awarded to Sanford Research in 2017. It is not just the CRCAIH PGP PIs using the CRCAIH pilot study as a springboard for additional funding applications. Four (36%) of CRCAIH pilot grant PIs reported that their partner organizations or members of their research team have submitted additional grants as a result of their involvement with the CRCAIH PGP.
Although not all awardees have peer-reviewed publications from their CRCAIH pilot grant, it must be taken into consideration that the success of a pilot study utilizing a CBPR approach with AI communities cannot be measured solely on the basis of peer-reviewed scholarly output. Employing community members from their study sites, as approximately three-quarters did, fosters a deeper connection to the community and provides a wealth of knowledge otherwise unattainable. Half included undergraduate and graduate students as members of their research team, which provides potential future researchers valuable experience. Six (55%) of PIs indicated that their CRCAIH pilot grant led to additional collaborations with members of their research team, including tribal/community organizations or additional research projects at their study sites. The CRCAIH PGP contributions to research in tribal communities and the development of future investigators will be of lasting impact.
Evaluation
Evaluation was critical to continuous improvement of the PGP processes. CRCAIH conducted surveys of potential (everyone who contacted CRCAIH for assistance) and actual applicants. For example, this is where the suggestion to extend the amount of writing time in year 1 was mentioned by several people, and changed for future years. Likewise, CRCAIH also conducted surveys among the pilot grant reviewers in years 1 and 2 to determine if improvements should be made in the reviewing process. An example of those improvements detailed above were sharing reviewers’ critiques ahead of time.
2013-2014 Pilot Grant Cohort
Feedback from the 2014 Pilot Grant Completion survey on how assistance pilot grant awardees received was most helpful included: “just good to know that I had a support system there to help submit the grant, ask questions about gaining IRB approval, and analyzing the data;” “assistance with the IRB and reports;” “that everyone was very eager and willing to help me answer my questions;” and “I feel I had great support and had my questions answered quickly and in a timely manner.”
CRCAIH Pilot Grant Program Follow-up
Though not as structured as the evaluation of the 2013-2014 cohort, the Administrative division of CRCAIH has kept contact with the PGP awardees over the years. This has primarily been done through a survey using SurveyMonkey® on an annual basis. Overall, the CRCAIH PGP awardees have answered with positive responses of their experience in the CRCAIH PGP. Out of the 11 pilot grant awardee responses, many said it was crucial for submitting other grants, for example: “[The PGP] gave us the opportunity to collect pilot data necessary for R01 grant submission;” and “…the CRCAIH pilot was the perfect opportunity to gather pilot data. I think if I had tried to write this into a larger NIH grant, I would have gotten dinged because it wasn’t a methodology I had done yet. But now I can say I have done it and can cite these efforts via the manuscript we produced.”
Other respondents mentioned the PGP was helpful to relationship building, which is especially critical for early stage investigators. For example, “[The PGP] Increased visibility / credibility for our University-Community partnership; this is situating us as more competitive for further funding”, and “the pilot program gave us the opportunity to build collaborations with the community that has led to the formation of 3 new project ideas”.
Reflection and Recommended Solutions
Challenges/Lessons Learned
In establishing the CRCAIH PGP and running it for four years, many lessons were learned and corresponding improvements were made in the process along the way. For example, although the CRCAIH PGP found success with simplified application materials and scoring rubric (see Figure 3), one applicant and one reviewer over the years mentioned in the survey evaluation wanting CRCAIH to utilize the standard NIH application and scoring materials, respectively. After much internal discussion, the PGP Subcommittee decided to continue using the simplified materials because the pilot grant was often an entry point for obtaining funding, and CRCAIH wanted to create a process that was easy to navigate for research novices. However, it is important to weigh the potential benefits of utilizing the NIH forms and scoring system, because that would give both applicants and reviewers more exposure to NIH standards for their future work. In this way, it would be easier for applicants to turn their applications into submissions to the NIH.
Additionally, as shown in Table 2, the number of applications received over the four years decreased, starting at 25 in year 1 and reducing to five in year 4. While specific reasons for why this occurred are not known, a few reasons were hypothesized. One optimistic view is that once applicants were funded, the pool of available investigators was reduced. In addition, it could be argued in subsequent years, applicants had the benefit of seeing the types of previously funded grants which helped tighten the field of applications, and the information posted on the FAQ page may have helped investigators determine their project was not a close fit for CRCAIH’s purpose. In addition, previous applicants who were unfunded may have been discouraged from applying again, judging their chances of funding on resubmission not worth the time investment. It would be impossible to calculate the number of possible investigators who were interested in CRCAIH PGP funding over the course of the CRCAIH grant.
Future Directions
The CRCAIH PGP was an overall success that would continue funding projects if funding was available. Awardees who participated in the follow up provided specific suggestions and ideas for improvement of the CRCAIH PGP were it to be reinstated in the future, including “It would have been great if I could have applied for additional funding to buy me out of teaching a class so I would have had the time to submit this work for publication.” Although CRCAIH continually encouraged the utilization of the core and division resources to the awardees, very few took advantage of the services after their pilot grant was funded. One idea for future PGPs would be to make the use of the cores mandatory. Similarly, another awardee recommended pilot grant trainings by cohort before funding was slated to begin, “… This could help with implementing innovative angles / ideas along the way that we might not have thought of beforehand.”
If CRCAIH were to redesign a funding program in the future, it would also include a formal mentoring component. Mentoring can encourage success and is an essential part of increasing diversity in the scientific workforce (Kameny et al., 2014; National Institutes of Health, n.d.). One awardee suggested a similar idea, “Potentially providing a peer or senior mentor at some point throughout the program.” Due to limitations of time and resources, much of the input for this program went into capacity-building assistance for potential pilot grant applicants. However, to better serve those awardees, more focus could be given to mentoring them throughout the project startup period, data analysis, publication writing, and future grant writing.
Another idea similar to other mentoring programs (Manson et al., 2006) is to establish ongoing group and individual mentoring meetings to establish mentorship, identify and support applications for further funding, and continue to use of CRCAIH cores and divisions for capacity-building assistance beyond the one-year pilot grant program. After notification of the pilot grant award, this would entail developing a mentorship plan to identify their strengths and weaknesses in research skills and identify one or two areas (statistics, interviewing, analyzing focus group data) to improve professional development and develop a research agenda that expands beyond the pilot grant year.
The mentorship plan would be used as a guide to match a mentorship team with the awardee, work with the grants management office to identify funding announcements and sources throughout the year, such as Career Development K-awards, and help set a timeline that includes a grant application and pilot grant publications. Mentors would provide support through activities such as reviewing and commenting on research studies, publication drafts, and discussion on specific aims for grant applications.
The CRCAIH PGP shows the promise of investment in underrepresented investigators in AI health. There is a clear need for additional scientific workforce development funding, which should be invested in building a cadre of AI researchers.
Author’s Note
The authors have no conflicts of interest to disclose.
Acknowledgements
This research was supported by funding from the National Institute on Minority Health and Health Disparities (U54MD008164) from the National Institutes of Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health (NIH). We would also like to thank Emily White Hat, Tabatha Lemke, Amy Elliott, Jerry Yutrzenka, Ronda Hinsch, Derrick Tabor, and Dottie Castille for their assistance and input.
Alyson E. Becker, MPH
Senior Research Specialist
Sanford Research, Population Health
2301 E. 60TH St. N
Sioux Falls, SD 57104, United States
(605) 312-6211
Email: Alyson.Becker@SanfordHealth.org
Jessica Heinzmann, BA
Senior Research Specialist
Sanford Research, Population Health
DenYelle Baete Kenyon, PhD
Associate Scientist, Sanford Research, Population Health
Associate Professor, University of South Dakota, Sanford School of Medicine, Department of Pediatrics
Correspondence concerning this article should be addressed to Alyson E. Becker, Senior Research Specialist, Sanford Research, Population Health, 2301 E. 60TH St. N, Sioux Falls, SD, 57104, United States, Alyson.Becker@SanfordHealth.org
References:
CRCAIH. (2017a). About. Retrieved from https://crcaih.org/about-crcaih.html
CRCAIH. (2017b). CRCAIH Pilot Grant FAQs. Retrieved from
https://crcaih.org/resources-miscellaneous/crcaih-pilot-grant-faqs.html
CRCAIH. (2017c). Pilot Grant Seminar Series. Retrieved November 29, 2017, from
https://crcaih.org/seminar-series.html
CRCAIH. (2017d). Pilot Grants Program. Retrieved from
https://crcaih.org/pilot-grant-program.html
CRCAIH. (2017e). Publication list. Retrieved June 11, 2018, from
https://crcaih.org/about-crcaih/publications.html
CRCAIH. (2017f). Summit. Retrieved November 29, 2017, from
https://crcaih.org/summit.html
CRCAIH. (2017g). Training/Resources. Retrieved June 15, 2018, from
https://crcaih.org/training-and-resources/91-all-training-resources.html
Daniels, R. J. (2015). A generation at risk: Young investigators and the future of the biomedical workforce. Proceedings of the National Academy of Sciences, 112(2), 313–318. https://doi.org/10.1073/pnas.1418761112
Doody, O., & Doody, C. M. (2015). Conducting a pilot study: Case study of a novice researcher.British Journal of Nursing (Mark Allen Publishing), 24(21), 1074–1078. https://doi.org/10.12968/bjon.2015.24.21.1074
Elliott, A. J., White Hat, E. R., Angal, J., Grey Owl, V., Puumala, S. E., & Kenyon, D. (2016). Fostering social determinants of health transdisciplinary research: The Collaborative Research Center for American Indian Health. International Journal of Environmental Research and Public Health, 13(1), 24.
https://doi.org/10.3390/ijerph13010024
Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., & Raynard, K. (2011). Race, ethnicity, and NIH research awards. Science, 333(6045), 1015–1019. https://doi.org/10.1126/science.1196783
Hayden, E. C. (2015). Racial bias continues to haunt NIH grants. Nature, 527(7578), 286–287. https://doi.org/10.1038/527286a
Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (2008). Community-based participatory research for health: From process to outcomes. In M. Minkler & N. Wallerstein (Eds.), Critical issues in developing and following community-based participatory research principles (pp. 47–66). San Francisco, CA: Jossey-Bass.
Jahnke, A. (2015, April 6). Who picks up the tab for science? Retrieved June 6, 2018, from http://www.bu.edu/today/2015/funding-for-scientific-research/
Kaiser, J. (2011). NIH uncovers racial disparity in grant awards. Science, 333(6045), 925–926. https://doi.org/10.1126/science.333.6045.925
Kaiser, J. (2017a). NIH abandons grant cap, offers new help to younger scientists. Science, 356(6343), 1108. https://doi.org/10.1126/science.aan6947
Kaiser, J. (2017b). NIH to cap grants for well-funded investigators. Science, 356(6338), 574. https://doi.org/10.1126/science.356.6338.574
Kameny, R. R., DeRosier, M. E., Taylor, L. C., McMillen, J. S., Knowles, M. M., & Pifer, K. (2014). Barriers to career success for minority researchers in the behavioral sciences. Journal of Career Development, 41(1), 43–61.
https://doi.org/10.1177/089484531247 2254
Levine, I. S. (2007, March 2). Making the leap to independence. Retrieved June 6, 2018, from http://www.sciencemag.org/careers/2007/03/making-leap-independence
Manson, S. M., Goins, R. T., & Buchwald, D. S. (2006). The Native Investigator Development Program: Increasing the presence of American Indian and Alaska Native scientists in aging-related research. Journal of Applied Gerontology, 25(1), 105S-130S. https://doi.org/10.1177/0733464805282727
Moore, C. G., Carter, R. E., Nietert, P. J., & Stewart, P. W. (2011). Recommendations for planning pilot studies in clinical and translational research. Clinical and Translational Science, 4(5), 332–337. https://doi.org/10.1111/j.1752-8062.2011.00347.x
National Institute on Minority Health and Health Disparities. (n.d.). Transdisciplinary collaborative centers for health disparities research program (TCC). Retrieved May 25, 2017, from
https://www.nimhd.nih.gov/programs/extramural/transdisciplinary-collab-centers.html
National Institutes of Health. (2015a, June 15). Data by gender | R01-Equivalent grants: Awards, by gender. Retrieved November 28, 2017, from
https://report.nih.gov/NIHDatabook/ Charts/Default.aspx?showm=Y&chartId=172&catId=15
National Institutes of Health. (2015b, June 15). New NIH investigators R01-equivalent grants, new (Type 1): Success rates, by career stage of investigator. Retrieved November 14, 2017, from https://report.nih.gov/NIHDatabook/Charts/Default.aspx?showm=Y&chartId =165&catId=22
National Institutes of Health. (2015c, June 15). Research grants R01-equivalent grants: Applications, awards, and success rates. Retrieved November 14, 2017, from https://report.nih.gov/NIHDatabook/Charts/Default.aspx?showm=Y&chartId=29&catId=2
National Institutes of Health. (2016a, May 23). NIH research project grant program. Retrieved June 5, 2018, from https://grants.nih.gov/grants/funding/r01.htm
National Institutes of Health. (2016b, September 1). Activity codes search results. Retrieved November 14, 2017, from https://grants.nih.gov/grants/funding/ac_search_results.htm
National Institutes of Health. (2017a, March 10). Grants process overview. Retrieved November 28, 2017, from https://grants.nih.gov/grants/grants_process.htm
National Institutes of Health. (2017b, May 2). New NIH approach to grant funding aimed at optimizing stewardship of taxpayer dollars. Retrieved June 6, 2018, from https://www.nih.gov/about-nih/who-we-are/nih-director/statements/new-nih-approach-grant-funding-aimed-optimizing-stewardship-taxpayer-dollars
National Institutes of Health. (2017c, August 29). Early stage and early established investigator policies. Retrieved October 10, 2017, from https://grants.nih.gov/policy/early-investigators/index.htm#esi
National Institutes of Health. (2017d, August 31). Next generation researchers initiative. Retrieved November 14, 2017, from https://grants.nih.gov/ngri.htm
National Institutes of Health. (2018, January 19). NIH Data Book. Retrieved February 6, 2018, from https://report.nih.gov/nihdatabook/
National Institutes of Health. (n.d.). NIH scientific workforce diversity toolkit. Retrieved from https://diversity.nih.gov/sites/coswd/files/images/SWD_Toolkit_Interactive-updated_508.pdf
National Science Foundation. (2017, January). Women, minorities, and persons with disabilities in science and engineering: Data tables. Retrieved November 28, 2017, from https://www.nsf.gov/statistics/2017/nsf17310/data.cfm
Oh, S. S., Galanter, J., Thakur, N., Pino-Yanes, M., Barcelo, N. E., White, M. J., … Burchard, E. G. (2015). Diversity in clinical and biomedical research: A promise yet to be fulfilled. PLoS Medicine, 12(12), e1001918. https://doi.org/10.1371/journal.pmed.1001918
Tragesser, S. (2011, July). How to become a truly independent investigator. Retrieved June 5, 2018, from http://www.apadivisions.org/division-28/publications/newsletters/ psychopharmacology/2011/07/independent-investigator.aspx
United States Census Bureau. (n.d.). QuickFacts: UNITED STATES; South Dakota. Retrieved November 28, 2017, from https://www.census.gov/quickfacts/fact/table/ US,SD/PST040216#viewtop
van Teijlingen, E. R., & Hundley, V. (2001). The importance of pilot studies. Social Research Update, (35), 1–4. Retrieved fromhttp://sru.soc.surrey.ac.uk/SRU35.html
Keywords:
Pilot grant program, American Indian, racial disparities.
Have any questions? Contact the editor »
#VolumeXLIXNumber2#PilotGrant#AmericanIndian#RacialDisparities