Clear all

12 results found

reorder grid_view

Learning from Digital Democracy Initiative Grantees

January 20, 2022

Democracy Fund's Digital Democracy Initiative (DDI) and its grantees are radically reimagining what it looks like to make platforms accountable to the American public and renew public interest media.To support this work, the team's evaluation and learning partner, ORS Impact, conducted learning conversations with DDI grantees in September and October 2021 to understand:How grantees have responded to the past yearWhat it would take to better center racial equity in DDI's strategy and in grantees' workWhere grantees see opportunities in the current momentThe report summarizes findings about these three topics within and across learning conversations and raises considerations for funders about how to better center racial equity in their grant making, how to better support their grantees, and opportunities ripe for investment. The report encourages funders to reflect on these considerations and how they might be applicable to their strategy.

MacArthur Foundation Nuclear Challenges Big Bet: 2020 Evaluation & Learning Synthesis

January 29, 2021

Since November 2017, ORS Impact has served as evaluation and learning partner to the Nuclear Challenges team. ORS' evaluation focused on our Nuclear Challenges strategy and its specific, time-bound goals. The purpose of evaluation was to assess whether the strategy as articulated by the program team has contributed to progress towards a stabilized nuclear regime and a negotiated W-UM agreement within the timeframe set by the Big Bet. There were myriad evaluation inputs for the program team over the course of several years. This particular evaluation report served as one of several inputs for the program team and the Board in our strategy review process.The report lays out substantive findings regarding outcomes and the external landscape that, together, present a picture of progress to date for the Nuclear Challenges strategy. The report sets out to answer three overarching questions:Does progress to date demonstrate momentum and provide a line of sight to the Big Bet goals of a more stable nuclear regime by 2025 and a negotiated W-UM agreement by 2030?Is the Nuclear Challenges theory of change valid and adequate to reach the intended Big Bet goals?Does the landscape suggest continued windows of opportunity for progress toward the goals in the Nuclear Challenges strategy and theory of change within the limited timeframe of the Big Bet?To answer these questions, ORS employed a mixed method design to measure the status of and progress within module components and related to the strategy's end goal. The synthesized results presented in this report reflect cross-module data, analysis, findings, and evidence which speak specifically to the Nuclear Challenges theory of change. This report presents the first synthesis of evaluation data about the Nuclear Challenges strategy.Findings and evidence presented in this report are based on the specific time period in which data were collected, from May 2019 to June 2020. Given the dynamic nature of the field, there have since been further developments and continued evolution in the external landscape that have implications for the nuclear landscape and field.

MacArthur Foundation Nuclear Challenges Big Bet Report Annex 1: Summary of Methods

January 29, 2021

Since November 2017, ORS Impact has served as evaluation and learning partner to the Nuclear Challenges team. ORS' evaluation focused on our Nuclear Challenges strategy and its specific, time-bound goals. The purpose of evaluation was to assess whether the strategy as articulated by the program team has contributed to progress towards a stabilized nuclear regime and a negotiated W-UM agreement within the timeframe set by the Big Bet. There were myriad evaluation inputs for the program team over the course of several years. This particular evaluation report served as one of several inputs for the program team and the Board in our strategy review process.The report lays out substantive findings regarding outcomes and the external landscape that, together, present a picture of progress to date for the Nuclear Challenges strategy. The report sets out to answer three overarching questions:Does progress to date demonstrate momentum and provide a line of sight to the Big Bet goals of a more stable nuclear regime by 2025 and a negotiated W-UM agreement by 2030?Is the Nuclear Challenges theory of change valid and adequate to reach the intended Big Bet goals?Does the landscape suggest continued windows of opportunity for progress toward the goals in the Nuclear Challenges strategy and theory of change within the limited timeframe of the Big Bet?To answer these questions, ORS employed a mixed method design to measure the status of and progress within module components and related to the strategy's end goal. The synthesized results presented in this report reflect cross-module data, analysis, findings, and evidence which speak specifically to the Nuclear Challenges theory of change. This report presents the first synthesis of evaluation data about the Nuclear Challenges strategy.Findings and evidence presented in this report are based on the specific time period in which data were collected, from May 2019 to June 2020. Given the dynamic nature of the field, there have since been further developments and continued evolution in the external landscape that have implications for the nuclear landscape and field.Synthesis report: https://search.issuelab.org/resource/macarthur-foundation-nuclear-challenges-big-bet-2020-evaluation-learning-synthesis

MacArthur Foundation Nuclear Challenges Big Bet Report Annex 2: Data Collection Tools

January 29, 2021

Since November 2017, ORS Impact has served as evaluation and learning partner to the Nuclear Challenges team. ORS' evaluation focused on our Nuclear Challenges strategy and its specific, time-bound goals. The purpose of evaluation was to assess whether the strategy as articulated by the program team has contributed to progress towards a stabilized nuclear regime and a negotiated W-UM agreement within the timeframe set by the Big Bet. There were myriad evaluation inputs for the program team over the course of several years. This particular evaluation report served as one of several inputs for the program team and the Board in our strategy review process.The report lays out substantive findings regarding outcomes and the external landscape that, together, present a picture of progress to date for the Nuclear Challenges strategy. The report sets out to answer three overarching questions:Does progress to date demonstrate momentum and provide a line of sight to the Big Bet goals of a more stable nuclear regime by 2025 and a negotiated W-UM agreement by 2030?Is the Nuclear Challenges theory of change valid and adequate to reach the intended Big Bet goals?Does the landscape suggest continued windows of opportunity for progress toward the goals in the Nuclear Challenges strategy and theory of change within the limited timeframe of the Big Bet?To answer these questions, ORS employed a mixed method design to measure the status of and progress within module components and related to the strategy's end goal. The synthesized results presented in this report reflect cross-module data, analysis, findings, and evidence which speak specifically to the Nuclear Challenges theory of change. This report presents the first synthesis of evaluation data about the Nuclear Challenges strategy.Findings and evidence presented in this report are based on the specific time period in which data were collected, from May 2019 to June 2020. Given the dynamic nature of the field, there have since been further developments and continued evolution in the external landscape that have implications for the nuclear landscape and field.Synthesis report: https://search.issuelab.org/resource/macarthur-foundation-nuclear-challenges-big-bet-2020-evaluation-learning-synthesis

MacArthur Foundation Nuclear Challenges Big Bet Report Annex 3: Secondary Sources

January 29, 2021

Since November 2017, ORS Impact has served as evaluation and learning partner to the Nuclear Challenges team. ORS' evaluation focused on our Nuclear Challenges strategy and its specific, time-bound goals. The purpose of evaluation was to assess whether the strategy as articulated by the program team has contributed to progress towards a stabilized nuclear regime and a negotiated W-UM agreement within the timeframe set by the Big Bet. There were myriad evaluation inputs for the program team over the course of several years. This particular evaluation report served as one of several inputs for the program team and the Board in our strategy review process.The report lays out substantive findings regarding outcomes and the external landscape that, together, present a picture of progress to date for the Nuclear Challenges strategy. The report sets out to answer three overarching questions:Does progress to date demonstrate momentum and provide a line of sight to the Big Bet goals of a more stable nuclear regime by 2025 and a negotiated W-UM agreement by 2030?Is the Nuclear Challenges theory of change valid and adequate to reach the intended Big Bet goals?Does the landscape suggest continued windows of opportunity for progress toward the goals in the Nuclear Challenges strategy and theory of change within the limited timeframe of the Big Bet?To answer these questions, ORS employed a mixed method design to measure the status of and progress within module components and related to the strategy's end goal. The synthesized results presented in this report reflect cross-module data, analysis, findings, and evidence which speak specifically to the Nuclear Challenges theory of change. This report presents the first synthesis of evaluation data about the Nuclear Challenges strategy.Findings and evidence presented in this report are based on the specific time period in which data were collected, from May 2019 to June 2020. Given the dynamic nature of the field, there have since been further developments and continued evolution in the external landscape that have implications for the nuclear landscape and field.Synthesis report: https://search.issuelab.org/resource/macarthur-foundation-nuclear-challenges-big-bet-2020-evaluation-learning-synthesis

Culture of Congress Evaluation Report

February 18, 2020

The William and Flora Hewlett Foundation's Madison Initiative (MI) seeks to strengthen U.S. democracy and its institutions in a time of political polarization. The goal is to help create the conditions in Congress in which its Members can deliberate, negotiate, and compromise in ways that work for most Americans. Launched in 2014, this nonpartisan initiative supports nonprofit organizations across the ideological spectrum—academic researchers, advocacy groups, think tanks, and civic leadership organizations—that seek to understand and improve the political system so that elected representatives are better equipped to solve society's greatest problems and in turn, earn public trust and support. The Hewlett Foundation's board authorized MI to make $15-20 million in grants per year from 2014 to 2021, for a total commitment of $150 million.

Sustainability of Feedback Practice: 2016 Listen4Good Cohort

July 30, 2019

Past evaluations of Shared Insight's Listen4Good grantees have reported that through L4G they increased their technical ability to perform high-quality feedback loops, gained insights that informed data-driven changes to programming and internal operations, and improved both their programs' effectiveness and their overall ability to serve clients. Evaluations have also found that L4G helped foster a culture of openness and listening in participating organizations and advanced their equity, diversity, and inclusion efforts.Until now, however, these studies have examined L4G participants only during their grant periods. In this report, ORS Impact takes on a central learning question for the sustainability of Shared Insight's feedback work: To what extent do grantees continue collecting high-quality perceptual feedback from clients after the L4G grant ends?Here, ORS looks at 46 nonprofits that were the first to receive L4G grants in 2016, conducting interviews with 35 of them about a year after their grant ended.

MEL Practice at the David and Lucile Packard Foundation: Evaluation in Support of Moving from Good to Great

January 31, 2019

In early 2017, ORS Impact evaluated and re-examined the David and Lucile Packard Foundation monitoring, evaluation, and learning (MEL) principles and practice. The purpose of this evaluation was to discover what works well, identify areas for improvement, and stimulate reflection and experimentation. While this report uncovered many examples of strong MEL practice across the Foundation it also highlighted opportunities for improvement. Research findings fed into Foundation decisions to update both internal and external MEL processes and requirements, including refinement of the Foundation's Guiding Principles for MEL.A key audience of this report include readers wrestling with how to best support MEL in philanthropic settings so that it can support greater learning and impact, such as MEL staff working inside foundations and external evaluators working with foundations.

When Collective Impact Has an Impact: A Cross-site Study of 25 Collective Impact Initiatives

October 26, 2018

A rigorous study to understand when and how collective impact contributes to systems and population change. Looking across 25 initiatives working on different focus areas, the study looks at the contribution and outcomes of collective impact, the design and implementation of the collective impact approach, with a specific deep-dive into equity.

Beyond the Win: Pathways for Policy Implementation

January 5, 2016

When it comes to policy, a lot of attention is given to "the win." Whether it is something new and big like the Affordable Care Act, a piece of legislation in a large federal omnibus bill, or inclusion of critical language in a state policy, seeing the fruits of advocacy efforts put into law makes advocates and champions feel that their hard work, often many years in the making, has paid off.However, in reality, "the win" is just the beginning -- a necessary first step in a much longer and equally as fraught process of policy implementation. Once a policy is created, there are numerous factors that shape and determine how that policy is implemented -- and ultimately, the impact it will have -- regardless of how well the policy is formulated. Some of these factors include rulemaking, funding, capacity of local implementing agencies, and fights to repeal or modify wins, among many others.And, just as in the case of "the win," advocacy plays an important role in shaping implementation whether in advocating across these factors or participating in ongoing monitoring over time. Interestingly, while the role of advocacy in agenda setting, policy formulation, and policy adoption has been widely explored in theory and practice, the role of advocacy in the policy implementation process has received less attention in the literature.To learn more about the role of advocacy at the policy implementation stage, ORS Impact spoke with organizations that engage in, or provide funding for, advocacy efforts at the state and/or federal level. We focused on the following questions:When had advocates played a positive role in policy implementation?When had implementation not gone as well as expected, and what did advocates take away from that?Our conversations yielded important learnings about the unique characteristics of, and range of approaches to, advocacy efforts during the implementation phase. The two following scenarios illustrate some of the different types and levels of advocacy intervention, as well as the results they produce, to demonstrate the ways advocacy can play out when shifting from policymaking to implementation.

The William and Flora Hewlett Foundation's Nuclear Security Initiative: Findings from a Summative Evaluation

March 1, 2015

The Nuclear Security Initiative (NSI) began as an exploratory effort in 2008 and, as with other Foundation initiatives, was intended to be a time-limited effort (though the timeframe for the Foundation's exit was not specified at the outset). The NSI was extended in 2011 and the last grants were made in 2014. Over seven years, the NSI pursued a number of strategies designed to reduce the risk of a nuclear disaster.Although security issues have never been a central element in the Hewlett Foundation's main programs, the Foundation does have a history of funding projects in the peace and security space when these issues touched on the Foundation's main focus areas. Re-entry into the nuclear security space was opportunistic; at the time of the NSI's inception, windows appeared to be opening, signaling that nearterm gains on pressing policy issues were possible. In 2007, four eminent statesmen (Kissinger, Shultz, Nunn and Perry) authored a provocative Wall Street Journal op-ed calling for "a nuclear-free world" and outlining the policy steps required to achieve it. This was the first such articulation by prominent foreign policy and national security leaders from across the political spectrum. Shortly thereafter came the election of Barack Obama who, as a Senator, had taken interest in advancing nuclear security and nonproliferation and who, when newly elected, began making these issues a first-tier national security concern of his Administration. Finally, there was increasing movement by some growing powers, such as Brazil and Turkey, to explore development of nuclear energy domestically, thereby increasing the risk of global nuclear proliferation.The evaluation that is the subject of this report revealed that the NSI set in motion new things in the field—including an increased and more intentional focus on advocacy and communications, increased coordination among funders, and increased attention to building the expertise and capacity of states outside of the P5 (United States, Russia, China, United Kingdom, France) and other established nuclear powers, as opposed to focusing exclusively on US-Russia and US-China relations. The Hewlett Foundation's re-entry into the nuclear security space was seen as bringing "excitement and energy" and "innovation."

Evaluation of the California Education Policy Fund 2011-2014

October 1, 2014

ORS Impact was asked by Hewlett to conduct a retrospective evaluation of the Foundation's investment in the CEPF from 2011-2014.4 The evaluation addressed the following questions:1. What has been achieved as a result of funding the first two CEPF cohorts for ecosystem support from 2011-2013?2. To what degree has there been a successful shift from the ecosystem approach to a Deeper Learning focus among CEPF grantees who received funding as part of Cohort 3 in 2013?3. To make an informed decision regarding the structure of the CEPF going forward:- What is the perception among key informants about the balance between the need for new or changed policies versus implementation of existing policies and/or expansion of pilot activities around the strategic work plan goal areas?- What are the perceptions about the "ripeness" or timeline with which policy advances can be expected in these areas from key informants' points of view?The first two questions were specific to CEPF grantees; the third question asked informants about the broader education reform field in California, including but not limited to CEPF grantees.