Blog Viewer

Volume LVII, Number 2 |

Rita S. Graef
College of Agricultural Sciences, the Pennsylvania State University

Daniel D. Foster, PhD
College of Agricultural Sciences, the Pennsylvania State University

Kathleen M. Hill, PhD
College of Education, the Pennsylvania State University

Tracy S. Hoover, PhD
College of Agricultural Sciences, the Pennsylvania State University

Abstract

As principal investigators (PIs) develop funding proposals, the framework presented here offers a model for considering how scientists engage audiences in research itself, in activities directly related to specific research project, and in activities complementary to research project. This study advances previous work by (1) considering research proposals funded by multiple federal agencies, (2) characterizing research impact activities with independent variables for audience and engagement, and (3) analyzing the concurrence of these characteristics. Informed by the Theory of Change and Program Logic Model, the framework describes activities and resources included in proposals using three outwardly expanding circles of audiences and five levels of engagement. Content analysis of abstracts by researchers in a college of agricultural sciences at one land grant institution depicts how far beyond academia and how deeply stakeholders are engaged in the research and science that lead to societal benefit. We find that while researchers in agricultural science-related research describe activities and societal benefit in their research proposals, the characteristics of audience and engagement varies by funder. We provide PIs a tool to assess their description of program inputs and outputs that are expected to deliver desired outcomes and impact of scientific research, regardless of the sponsor.

Keywords: benefit to society, broader impact, proposal review criteria, agricultural science research, audience, engagement, program logic model, theory of change, NIH, NSF, USDA

Introduction

Most federal funding agencies have incorporated societal benefits as essential review criteria for research proposals, signaling the importance of engaging the public in scientific endeavors. Growing emphasis on broader impacts in scientific research has shaped federal funding policy as well as proposal review criteria. While much of the work to describe broader impact activities has focused studies on National Science Foundation (NSF)-funded proposals, we expand the scope of analysis to include proposals funded by multiple federal agencies, including NSF, the U.S. Department of Agriculture (USDA), and the National Institutes of Health (NIH). Falk-Krzesinski and Tobin's (2015) study of broader impacts across 10 U.S. federal agencies highlights the increasing importance of broader impacts alongside scientific merit in funding decisions. Successful proposals must demonstrate not only scientific rigor but also clearly articulate societal relevance. The NSF continues to apply the statutorily required Intellectual Merit and Broader Impacts review criteria, along with any solicitation-specific criteria (NSF, 2025). As funding becomes increasingly constrained by policy, the societal impact of research will play an even greater role in proposal evaluations and award decisions. To support both researchers and reviewers, adopting an explicit framework for defining and characterizing impact can help ensure activities are described clearly and evaluated consistently.

While NSF has included broader impacts as a core element of its review process for decades, since 1987, for over a century and a half the mission of land-grant colleges has emphasized societal benefits and enshrined in foundational federal legislation such as the Morrill Act (1862), the Hatch Act (1887), and the Smith Lever Act (1914). Over time, agricultural colleges have nurtured a tradition of applied scientific research, engaging directly with producers and farmers to address practical challenges (Boyer, 1990). The historical integration of research impact activities in agricultural science aligns with the goals set forth by the Morrill, Hatch, and Smith Lever Acts, emphasizing education, experimentation, and the dissemination of knowledge for public good. This study examines research-related activities shaped by the long history of cooperative research with farmers and producers, as well as the translation of research into practical applications through cooperative extension research and education efforts. An agricultural college’s historical roots in applied research and community engagement provide a unique context for analyzing how principal investigators (PIs) describe the potential impacts of their proposed research projects. This study utilizes content analysis to characterize how scientists propose research that meets funders’ review criteria. Comparing abstracts from proposals to multiple funding agencies, this study characterizes activities using independent variables for audience type and level of engagement and further analyzes and maps the concurrence of these variables. 

Theoretical Framework

As broader impacts become more central to proposal evaluations, understanding how PIs articulate their research's societal relevance is vital. Literature on this topic has primarily examined how researchers frame their broader impacts in response to NSF’s criteria, most often focusing on proposal abstracts (Roberts, 2009; Kamenetzky, 2014; London, 2014). These studies have analyzed language in abstracts to identify the types of activities proposed, including education, participation, infrastructure, and dissemination. For example, Nadkarni and Stasch (2013) studied abstracts to assess the presence of impact activities, audience types, and communication strategies, while Nagy (2013) surveyed PIs to identify the categories of broader impact activities: "education, broadening participation, infrastructure, dissemination, and other." Skrip (2015) offers PIs a framework of five questions (who, why, what, how, and with whom) to design broader impact activities that meet the varied and complex social needs of different audiences. Acknowledging that "abstracts generally contain 3-5 lines on proposed broader impacts," Nadkarni & Stasch (2013) looked at proposal abstracts for the presence of broader impact activities, the types and size of intended audiences, the kind of communication used to reach that audience, and the proximity of the audience to academia.

This study recalibrates the assessment of the content of research proposals to more accurately describe activities. Theory of Change provides a logic model for thinking about public engagement with science, by defining the inputs, activities, and intended outcomes for the short-, medium-, and long-term, and an overall vision for how this engagement will impact both science and society (AAAS, 2016). Schoedinger et al (2019) describe Theory of Change as helpful in planning, implementing, and evaluating programs. Specifically, Program Logic Model provides a framework that identifies and organizes variables in the analysis and describes how program inputs relate to outputs and ultimately impact. Figure 1 illustrates the relationship between planned program activities that can create desired outcomes that move toward impacts, increased pursuit of STEM careers or improved pollinator habitat. Logic models serve to illustrate how a program works to draw out the relationship between resources, activities, and outcomes. When developing a program plan starting with the desired impact, the outcomes are designed into the program as measurable goals and objectives that can be measured and evaluated. Changes in awareness and knowledge can be measured, as can behavior change. The output in our model is a derived by characterizing audiences and how they are engaged. The inputs are resources required for these activities, including subject matter experts, partners skilled in the design and delivery of the activities, facilities and funding.

Figure 1. Theory of Change and Program Logic Model describe steps from inputs to impact

Previous studies have coded and analyzed the language that investigators use in their research proposals and have focused on understanding how researchers propose to meet NSF's review criteria by looking at the funding proposals abstracts. In identifying categories of broader impact from literature, these studies have described broader impact activities using a taxonomy to classify and organize based categories of activities. This study differs by further characterizing activities with independent variables for audience type and level of engagement. 

Characterizing Audience 

In his "Framework for Evaluating Impacts of Informal Science Education," Friedman (2008) considers the relationship between impact and audience-based variables from informal science education proposals, final reports, and summative evaluations. Friedman differentiates between public and professional audiences, suggesting how an audience engages with concepts in STEM and informal science education may determine what kinds of activities are planned and measures of success. The American Association for the Advancement of Science (AAAS, 2016) white paper "Theory of Change for Public Engagement, a Logic Model for Public Engagement with Science" is a primary reference across informal science education providers. Describing the steps from inputs to impact, the AAAS identifies participants as "Scientists, Publics, and Practitioners." In his November 1976 editorial for the journal Science, Editor Philip Abelson describes public audiences for scientific communication as holding different points of view and value systems from the scientists wanting to share knowledge. In a multiphase mixed methods research design that included coding the narratives in PI conference report abstracts, London (2014) summarizes claims that PIs make when articulating the impact of their research in three dimensions of research impact related to audiences and their relative distance away from academia and the work of scientific research. When communicating with various audiences, scientists must consider their audience's context, values, and goals and appreciate differences between audiences and their understanding.

As our framework characterizes audience, (Figure 2a), the scientific community is the nearest to the scientist. By publishing papers in peer-reviewed journals, researchers can share findings with an audience of peers who can understand the jargon and are fluent with principles underlying their research. Speaking at conferences or training graduate students extends the research within the closest circle of peers who can utilize this new knowledge in their own research and communicate using the same language.

Professionals who apply research findings to their field of work are in this study identified as practitioners. Practitioners may come from various fields, such as farming, forestry, manufacturing, or education. Practitioners use scientific research findings to improve their work processes, inform their methods, and apply best practices to their discipline. Although there is a shared language, there may be a need for translation or interpretation to successfully integrate research findings into another field. Practitioners may collaborate with researchers by providing valuable feedback and data, such as farmers planting trial varieties of seeds or forest managers sharing practices between landowners and researchers.

Scientists may also engage directly with the general public, which includes individuals, families, students from preschool to undergraduate, as well as elected representatives and policymakers. The public may participate in science outreach and engagement activities or contribute to research through citizen science or public discourse and policy-making.

Figure 2. Characterizing Audience and Engagement 

a.	b.

Characterizing Engagement

Engagement between research and society is at the core of broader impact activities. In the first external study to quantitatively examine the proposed broader impact activities of NSF-funded projects, Roberts (2009) organizes broader impact under two headings: (1) criteria for science and (2) criteria for society. Kamenetzky (2013) coded 360 publicly available abstracts of successfully funded NSF grants from three NSF directorates. Furthermore, Wiley (2014) examined the broader impact activities evident in 87 NSF-funded and unfunded research proposals, organizing findings in clusters by engagement and science communication types. Wiley divides activities by types of science communication, separating activities into two levels of engagement by the target audiences' participation in knowledge making. While previous studies described activities as either for science or for society, or organized them into different categories, we created a framework with mutually exclusive characteristics.

Searching for a hierarchy to help organize engagement, we found precedent in community planning and civic engagement. Sherry Arnstein's article for the Journal of the American Planning Association, "A Ladder of Citizen Participation" (1969), describes a typology of citizen participation arranged as a ladder, with each rung corresponding to the extent of citizens' power in determining the community plan or social program. Lower rungs represent informal communication, like published papers or websites. While Public Understanding of Science (PUS) traditionally focuses on filling 'knowledge gaps,' Skrip (2013) recommends placing a higher value on Public Engagement in Science and Technology (PEST), where scientists engage audience members to improve their lives. "Upstream Science" (Krzywoszynsky et al., 2018; Rogers-Hayden & Pidgeon, 2007; Wilsdon & Willis, 2014) whereby dialogue and deliberation about the value to society of science moves "upstream" engages the public and policymakers to prioritize the goals of scientific pursuit before controversy arises about potentially divisive issues. Organized as a pyramid, our scale of engagement illustrates "Informal" one-way communication with the potential to reach more stakeholders at the base and "Deliberation" requiring greater effort at the top reaching fewer, more deeply engaged participants (Figure 2b).

Characterizing Resources

In proposal abstracts, PIs may mention essential people (PIs, co-PIs, graduate students), partnerships (with community organizations or museums), or equipment for a project. PIs may choose to include mention of resources or capacity based on the highest-ranked costs for equipment and supplies enumerated for the project, with more detail included in the budget narrative of the full proposal. Project abstracts might overlook incremental items like program activities or evaluation costs while highlighting high-value resources such as equipment purchases, personnel, new facilities, and project partners. 

Qualitative Research Design

This study used content analysis to examine federally funded proposal abstracts from one college of agricultural sciences in a land grant university. The purpose was to investigate how Principal Investigators (PIs) describe the broader impact activities in their research for various funders. The study focused on the characteristics of research impact activities that can be measured in proposal abstracts, where PIs summarize their proposed project. The researchers developed a coding scheme and codebook that included an a priori code set based on the theory and framework defined for the study. Multiple rounds of review and testing refined the codebook and completed the coding scheme. We created a list of terms and related definitions based on a literature review. The coding scheme includes variables for broader impact, audience, engagement, resources, and associated definitions. Multiple coders tested the draft coding scheme (codebook, instructions, definitions, and coding form) to check for intercoder consistency and refine definitions for clarity. This interrater reliability check validated the coding scheme and enabled multiple coders to tackle large quantities of data. After the coder(s) coded the texts, the researcher checked for reliability, accuracy, precision, and internal and external validity (Neuendorf, 2017).

Dataset

This study examines broader impact activities described in 105 research proposal abstracts. Multiple agencies, including the NIH, USDA, and NSF, support research in the agricultural sciences. Several other sources of data were considered, before settling on data available from the Federal RePORTER database. Locally available data from a college’s grants office contained limited information; federal agency websites provide downloadable award detail; all of which must be merged into a working dataset. At the time of this study the Federal RePORTER aggregated all federally awarded research funding, including variables such as institution, principal investigators (PIs), awarding Federal Department and Agency, project title, and complete text of the proposal abstract. From this comprehensive source for federally funded research project abstracts, we downloaded federally funded research awarded to the university (N=2218) for fiscal years 2014, 2015, 2016, 2017, 2018, and 2019. Projects awarded in 2020 were not yet available. Using a publicly available directory of college faculty, we filtered the project proposal abstracts to include only those authored by PIs with a college affiliation. We noticed that 2014 data differed from 2015 and forward, as it included funding for appropriations that were not categorized as research. Because of this discrepancy, we excluded data from 2014 from the study, focusing on FY2015 through FY2019. Table 1 details the study frame representing projects by 69 PIs over five years (2015-2019). We coded projects running over multiple years in their first year. We omitted from further analysis the single instances of proposals from the Department of Education (ED) and the National Aeronautics and Space Administration (NASA). The study dataset includes 105 unique abstracts from NIH, NSF, and USDA for FY2015, FY2016, FY2017, FY2018, and FY2019.

Table 1. Data Selection and Sampling

NOTE: In 2022 the Federal RePORTER database was archived at the request of National Institutes of Health, Office of Extramural Research (NIH/OER) using Archive-It (Wayback, 2025). Today, USAspending is the official open data source for federal spending information, including information about federal awards, encompassing data from 2008 to present. 

Method

The study used content analysis to categorize the types of broader impact activities mentioned in research proposal abstracts. Content analysis is a research method that extracts reliable and valid inferences from text. This technique involves applying codes to the content and refining and reapplying them iteratively to ensure consistent and accurate outcomes. This study classified broader impact activities into "exhaustive and mutually exclusive measures" (Neuendorf, 2017) by creating separate indicators for the level of engagement, targeted audience, and resources. For instance, the phrase "extension workshop for producers" was separated into three indicators of variables for resources (extension), engagement (workshop), and audience (producers).

A Paper Tool. As we planned this study, and prior to using software to complete coding and content analysis, our method and codebook were first tested to work on paper. For each document (such as a proposal abstract) the coder would highlight mention of any broader impact activity one sentence at a time; then in corresponding boxes on the paper-based tool (Figure 3) a mark is placed at the intersection of audience type and level of engagement. While coding conducted using a digital tool, such as MAXQDA™, affords detailed analysis and mapping capabilities, the paper-based tool can serve as a prompt for clearer proposals, encouraging writers to explicitly describe activities proposed. It can be used to a review a writing sample for clarity when describing program inputs and outputs expected to deliver desired outcomes and impact. Researchers and research administrators can quickly quantify and visualize the intersection of audience and engagement in a broader impacts plan, highlight similarities and differences between proposals, or broadly illustrate a researcher’s impact identity. 

Figure 3. Paper Tool for Coding Broader Impact Activities by the Intersection of Audience and Engagement 

The coding process was performed step-by-step. The codebook with definitions and instructions guides the coding effort and describes the workflow. Due to the manageable number of abstracts in the dataset (n~100), we completed a census, coding all abstracts available in the selected data set. First, we scanned abstracts to identify broader impact (BI) type, screening every sentence in an abstract for a broader impact reference. Then, the coded segments were further analyzed based on the Audience Type (AT), Engagement Type (ET), and Resources or Capacity (RC) mentioned in each segment. 

Typology for Classifying Activities

Operational definitions (Table 2) describe how we, as researchers, measure variables in the data and apply codes to characteristics found in abstracts. Definitions in the codebook offer clarity and guide the researcher. This method and typology can serve as a check for researchers and grant support specialists to score abstracts before submission for inclusion of broader impact.

Table 2. Definitions for coding Broader Impact Activities

Codebook. A codebook defines variable measures, including definitions, examples, and guidelines for coding. For this project, all coding was completed manually using MaxQDA software. This software selected an unbiased randomized sample evenly dispersed across funders to test the codebook. This randomized representative sample of abstracts was coded and tested for intercoder agreement with peer review and debriefing to ensure the findings' reliability, trustworthiness, and credibility. Entire sentences were considered when coding to unitize segments to be considered and ensure consistency between coders. More than one characteristic code might be applied to a segment.

Software. Descriptive statistical analysis and charts and graphs were completed using Microsoft Excel software. In addition to mapping code co-occurrence, MAXQDA™ software facilitated random sampling, interrater reliability check, and validating coding scheme, which allows the practical advantage of using multiple coders to tackle large quantities of data.

Units and Sampling. In this study, we defined a "unit of sampling" as one abstract and a "unit of data collection" as a verbal clause. To ensure consistency between coders, we used a whole sentence as the coded element and allowed multiple codes to be applied to an entire sentence. The sample studied consisted of 105 successful abstracts from the college of agriculture between 2015 and 2019, each containing around 300-400 words. 

Discussion

This study analyzed the abstracts of proposed research projects within a single college at a large university, providing a focused view of principle investigators (PIs) at the nature of their proposed work. Findings from agri-science-related proposals indicate that broader impact activities vary notably by funder, particularly in terms of target audiences and modes of engagment. Future research should explore how these patterns compare across colleges within the same institution (e.g., engineering, sciences, agriculture) to better understand the institutional infrastructure that supports research translation. Additionally, comparative studies across college of agriculture within the Land Grant university system could deepen our understanding of how agricultural research contributes to societal benefit, particularly through extension and outreach activities.

All abstracts in the dataset of successful proposals contain a statement about the benefit to society or broader impact. 96% to 100% of all funders included a segment about the research itself (coded as BI1). If all proposals include broader impact statements, how does a reviewer differentiate between them to award the most impactful?

All projects contain at least a statement about the broader impact and state something about the research itself (Figure 4a, code BI1). A high percentage of the NIH and NSF abstracts (71% and 79%) included activities directly related to the research (Figure 4a, coded as BI2). The most significant difference between funders is found when analyzing activities complementary to the project, a finding that aligns with previously published research lending confidence to this coding scheme (Nadkarni & Stasch, 2013). The greatest range between funders in coding for activities complementary to the project (Figure 4a, coded as BI3) is NSF at 71%, the USDA at 57%, and only a small percentage (17%) for NIH. Figure 4a presents the types of broader impact activities that appear across funders. 83% of NIH abstracts had no activities complementary to the project (coded as BI3).

Figure 4.  Visualizing findings across funder, by the frequency of codes that identify the types of broader impacts, audience types, engagement types, and resources or capacity mentioned in the abstract.  

As funders increasingly require researchers to be specific about the benefits of their projects, including multiple characteristics of broader impact activities can more fully describe their potential. Audience types (Figure 4b) and engagement types (Figure 4c) are most frequently identified in NSF abstracts. The kinds of resources identified in abstracts varied by funder; "people and partnerships" predominate in USDA projects, and "facilities and equipment" are more commonly found in NIH and NSF projects. In USDA abstracts, people (RC1) at 35% and partnerships (RC3) at 57% were identified as the highest types of resources. NSF and NIH coded highest for facilities (RC2) at 25% and 24%, respectively (Figure 4d).

Comparing Code Maps Across Funders

PIs describe the broader impact of their project in a few brief sentences. Deconstructing broader impact activities into two characteristics of audience and engagement, with variables for each, allows for coding and analysis of audience and engagement individually or where these codes overlap. Examining the co-occurrence (defined as the intersection of codes within a single phrase or sentence) in the abstract can better describe the relative types of each and reveal how PIs plan to communicate with a particular audience within a specific context.

A Code Co-occurrence Model Map (Figure 5) shows the intersection of coded activity by funder. Line width reflects co-occurrence frequency  –  lines between codes with many relations appear thicker. Examining the visual representation of the co-occurrence of codes in these abstracts reveals that NIH projects skew toward the top right with more frequent connections between types of broader impact (BI) and resources (RC). Connection to audience types (AT) predominates in NSF projects, with heavier lines skewed towards the lower left. In addition to frequencies, co-occurrence in the same segment of codes for the type of broader impact and audience type were analyzed, where 30% of USDA abstracts indicated that practitioners were engaged in the research itself (coded as BI1 and AT2), 39% of USDA abstracts described partnerships in the same segments coded as the research itself (BI1 and AT2). The highest co-occurrence of the research itself and facilities (BI1 + RC2 ) is evident in NIH abstracts.

Figure 5.  Code Co-occurrence by Funder

Code co-occurrence model maps show the intersection of coded activity by funder, with weighted lines indicating higher overlap frequency. NSF abstracts (orange oval) reveal a higher incidence of public audiences (AT3) engaged in activities complementary to the research (BI3), while USDA-funded abstracts (yellow oval) center around practitioner audiences involved in all levels of research. NIH abstracts shifted towards the upper right quadrant of the circle, emphasizing the Research itself and resources to deliver broader impacts. 

While only 57% of USDA abstracts contain activities directly related to the research (coded as BI2), considering co-occurring types of engagement and audiences illustrate into how broadly those reach beyond the scientific community. When organizing levels of science communication in research activities, Wiley (2014) differentiates between Public Understanding of Science (PUS) and Public Engagement in Sciences and Technology (PEST). If we agree with Skrip (2015) and place a higher value on PEST than PUS, activities that engage practitioners (AT2) in the research activities (BI2) might hold more weight in the review criteria. 
USDA-funded research may include extension-type activities, while omitting explicit mention of “Extension” in the proposal  abstract. Partnerships and collaboration with subject matter experts or delivery experts may make a stronger case for proposal review. Internally, the college of agriculture is creating seed grant opportunities for research and extension faculty to collaborate in pursuing its research agenda. Aligned with the goals and current activities of the college of agricultural sciences, fostering collaboration between PIs and Extension expertise should continue and be acknowledged in proposals and related abstracts where appropriate. Coding abstracts for extension-like activities as well as for keywords like “Extension” may help Research Development track trends and success of such internal award programs.

Conclusions

Results show that though almost every abstract contains a statement about the impact of the research, a predictable percentage of activities "beyond the research itself" can be found in some funders rather than others. This research describes the characteristics of activities in terms of three outwardly expanding rings of stakeholder audiences and five levels of stakeholder engagement. By studying patterns of overlapping types of impact, audience, engagement, and resources, we see that PIs indicate in their abstracts how far beyond academia and how deeply stakeholders are involved in the research and science that lead to societal benefit.

The authors recommend additional, larger studies. The dataset used in this study includes abstracts for funded project proposals completed by PIs affiliated with one college of agricultural sciences. It is neither reflective of the entire university nor of other institutions’ colleges. Watts et al. (2013) included proposals and abstracts in their research data set and found that "full proposals were much more likely to include broader impact activities than were public abstracts." They found that 96.4% of the proposals (2000 through 2010) included at least one broader impact activity, but only 66.4% of the abstracts did. Future research may examine the relationship between full proposals and abstracts and the relative frequency of mention of broader impact in each. Further recommendations include research that widens the scope of content analysis to include full proposals, budget, unfunded proposals, and reviewer comments on successful and unsuccessful proposals obtained with permission of the PI and institution. Surveying PIs about the resources they have and the resources they need to accomplish the broader impacts of their projects can provide insight to help the institution strategically plan investments in infrastructure, expertise, and staffing, and developing partnerships and capacity to deliver successful broader impact and benefit. 

Author's Note

This article is based on the research conducted by Rita Graef for her graduate thesis. The authors declare no conflicts of interest. This project is not funded. The Office for Research Protections determined that the proposed activity, STUDY00016768, does not meet the definition of human subject research as defined in 45 CFR 46.102(d) and/or (f). Institutional Review Board (IRB) review and approval are not required.

Rita S. Graef
Department of Agricultural Economics, Sociology, and Education
College of Agricultural Sciences
Pennsylvania State University

Daniel D. Foster, PhD
Department of Agricultural Economics, Sociology, and Education
College of Agricultural Sciences
Pennsylvania State University

Kathleen M. Hill, PhD
Center for Science and the Schools College of Education
Pennsylvania State University

Tracy S. Hoover, PhD
Department of Agricultural Economics, Sociology, and Education
College of Agricultural Sciences
Pennsylvania State University

Corresponding Author

Correspondence concerning this article should be addressed to Rita Graef, College of Agricultural Sciences, Penn State University, 431 Curtin Road, Office 201, University Park PA 16802, telephone: 814-863-1385, email: rsg7@psu.edu. 

Acknowledgements

Appreciation to the Journal of Research Administration (JRA) Author Fellowship Program and Society of Research Administrators International (SRAI) Mentor Jeanne Kisacky for reading and commenting on drafts.

References 

Abelson, P. H. (1976). Communicating with the Public. Science, 194(4265), 565. https://www.science.org/doi/10.1126/science.194.4265.565

American Association for the Advancement of Science AAAS. (2016). Theory of change for public engagement with science. 1–12. https://www.aaas.org/sites/default/files/content_files/2016-09-15_PES_Theory-of-Change-for-Public-Engagement-with-Science_Final.pdf

Casey Foundation, A. E. (2004). Theory of Change A Practical Tool for Action, Results and Learning. www.organizationalresearch.com

Arnstein, S. R. (1969). A Ladder of Citizen Participation, Journal of the American Institute of Planners, 35:4, 216-224, DOI: 10.1080/01944366908977225

Boyer, E. (1996). The scholarship of engagement. Journal of Public Service and Outreach, 1(1), 11-20.

Bonney, R., Phillips, T., Enck, J., & Ballard, H. (2016). Can citizen science enhance public understanding of science? Public Understanding of Science, 25(1). https://journals.sagepub.com/doi/10.1177/0963662515607406

Falk-Krzesinski, H. J., & Tobin, S. C. (2015). How Do I Review Thee? Let Me Count the Ways: A Comparison of Research Grant Proposal Review Criteria Across US Federal Funding Agencies. The Journal of Research Administration, 46(2), 79–94. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4892374/

Friedman, A. Ed. . (2008). Framework for evaluating impacts of informal Science; Education projects report from a national science foundation workshop. In Framework for Evaluating Impacts of Informal Science Education Projects. https://www.informalscience.org/framework-evaluating-impacts-informal-science-education-projects

Krzywoszynska, A., Matt, W., Buckley, A., Chiles, P., Gregson, N., Holmes, H., & Mawyin, J. (2018). Opening Up the Participation Laboratory: The Cocreation of Publics and Futures in Upstream Participation. Science Technology and Human Values, 43(5), 785–809. https://journals.sagepub.com/doi/abs/10.1177/0162243917752865

London, J. (2014). The Impact of National Science Foundation Investments in Undergraduate Engineering Education: A Comparative Mixed Methods Study. https://docs.lib.purdue.edu/open_access_dissertations/327

Nadkarni, N. M., & Stasch, A. E. (2013). How broad are our broader impacts? An analysis of the National Science Foundation's Ecosystem Studies Program and the Broader Impacts requirement. Frontiers in Ecology and the Environment, 11(1), 13–19. https://esajournals.onlinelibrary.wiley.com/doi/abs/10.1890/110106

Nagy, D. (2016). Determinants of broader impacts activities: A survey of NSF-funded investigators. Journal of Research Administration, 47(1), 68–93. Retrieved from https://ezaccess.libraries.psu.edu/login?qurl=https%3A%2F%2Fwww.proquest.com%2Fscholarly-journals%2Fdeterminants-broader-impacts-activities-survey%2Fdocview%2F1886564165%2Fse-2%3Faccountid%3D13158

NSF - National Science Foundation (2025). Implementation of Recent Executive Orders (updated June 9, 2025). Retrieved July 29, 2025, from https://www.nsf.gov/executive-orders#information-for-panelists-and-reviewers-ead

Neuendorf, K. A. (2017). The Content Analysis Guidebook. In The Content Analysis Guidebook. SAGE Publications, Inc. https://doi.org/10.4135/9781071802878

Risien, J., & Storksdieck, M. (2018). Unveiling impact identities: A path for connecting science and society. Integrative and Comparative Biology, 58(1), 58–66. https://academic.oup.com/icb/article/58/1/58/4985726

Roberts, M. R. (2009). Realizing societal benefit from academic research: Analysis of the national science foundation's broader impacts criterion. Social Epistemology, 23(3–4). https://www.tandfonline.com/doi/abs/10.1080/02691720903364035

Rogers-Hayden, T., & Pidgeon, N. (2007). Moving engagement "upstream"? Nanotechnologies and the Royal Society and Royal Academy of Engineering's inquiry. Public Understanding of Science, 16(3), 345–364. https://journals.sagepub.com/doi/abs/10.1177/0963662506076141

Skrip, M. M. (2015). Crafting and evaluating Broader Impact activities: A theory-based guide for scientists. Frontiers in Ecology and the Environment. https://esajournals.onlinelibrary.wiley.com/doi/abs/10.1890/140209

Solomon, S., & Abelson, J. (2012). Why and when should we use public deliberation? In Hastings Center Report (Vol. 42, Issue 2, pp. 17–20). NIH Public Access. https://doi.org/10.1002/hast.27

Schoedinger, S. E., McDougall, C., and Bey, G. (2019). Education for Community Resilience - A Theory of Change for How K-12 and Informal Education are needed to achieve Community Resilience to Climate Change and other Environmental Hazards. AGUFM, 2019, ED33E-1019. https://ui.adsabs.harvard.edu/abs/2019AGUFMED33E1019S/abstract

Taylor-Powell, E., & Henert, E. (2008). Developing a logic model: Teaching and training guide Distinguished Evaluation Specialist DEVELOPING A LOGIC MODEL: TEACHING AND TRAINING GUIDE Content guide. http://www.uwex.edu/ces/pdande

Watts, S. M., George, M. D., & Levey, D. J. (2015). Achieving broader impacts in the National Science Foundation, division of environmental biology. In BioScience (Vol. 65, Issue 4, pp. 397–407). Oxford University Press. https://doi.org/10.1093/biosci/biv006

Wayback Archive-It, 2025. https://wayback.archive-it.org/18816/20220223125200/https:/federalreporter.nih.gov/FileDownload

Wilsdon, J., & Willis, R. (2004). See-through Science: Why public engagement needs to move upstream. In Demos (Vol. 2004, Issue 11/7/2004). https://demos.co.uk/wp-content/uploads/files/Seethroughsciencefinal.pdf

Wiley, S. L. (2014). Doing broader impacts? the National Science Foundation (NSF) broader impacts criterion and communication-based activities. In Graduate Theses and Dissertations. https://doi.org/10.31274/etd-180810-2638