How Unmet Desire Surveys Can Advance Learning Agendas and Strengthen Evidence-Based Policymaking
Summary
The 2018 Foundations for Evidence-Based Policymaking Act (Evidence Act) promotes a culture of evidence within federal agencies. A central part of that culture entails new collaboration between decision-makers and those with diverse forms of expertise inside and outside of the federal government. Federal chief evaluation officers lead these efforts, yet they face challenges getting buy-in from agency staff and in obtaining sufficient resources. One tool to overcome these challenges is an “unmet desire survey,” which prompts agency staff to reflect on how the success of their programs relates to what is happening in other agencies and outside government, as well as consider what information about these other programs and organizations would help their work be more effective. The unmet desire survey is an important data-gathering mechanism and also encourages evaluation officers to engage in matchmaking between agency staff and people who have the information they desire. Using existing authorities and resources, agencies can pilot unmet desire surveys as a concrete mechanism for advancing federal learning agendas in a way that builds buy-in by directly meeting the needs of agency staff.
Challenge and Opportunity
A core mission of the Evidence Act is to foster a culture of evidence-based decision-making within federal agencies. Since the problems agencies tackle are multidimensional, with the success of one government program often depending on the performance of others, new collaborative relationships between decision-makers in the federal government and those in other agencies and in organizations outside the federal government are essential to realizing the Evidence Act’s vision. Indeed, Office of Management and Budget (OMB) implementation guidance stresses that learning agendas are “an opportunity to align efforts and promote interagency collaboration in areas of joint focus or shared populations or goals” (OMB M-19-23), and that a culture of evidence “cannot happen solely at the top or in isolated analytical offices, but rather must be embedded throughout each agency…and adopted by the hardworking civil servants who serve on behalf of the American people” (OMB M-21-27).
Chief evaluation officers at federal agencies are the main point people for fostering cultures of evidence. Yet they and their evaluation staff face many challenges, including getting buy-in from agency staff, understanding the needs of program and operational offices that go beyond the organizational boundaries of those offices, and limited resources. Indeed, OMB guidance acknowledges that many agency staff may view learning agendas as just another compliance exercise.
This memo proposes a flexible tool that evaluation officers can use to generate buy-in among agency staff and leadership while also promoting collaboration as emphasized in OMB guidance and in the Evidence Act. The tool, which has already proven valuable in local government and in the nonprofit sector, is called an “unmet desire survey.” The survey measures unmet desires for collaboration by prompting staff to consider the following:
- How does the success of your program relate to what is happening elsewhere? Is there information about other programs within the government and/or outside organizations that would help improve the effectiveness of your work? What kinds of people from these other agencies/organizations would be helpful to connect with?
- Are you looking for informal collaboration (oriented toward knowledge exchange) or formal collaboration (oriented toward projects with shared ownership, decision-making authority, and accountability)?
- What hesitations (perhaps due to prior experiences, lack of explicit permission, stereotypes, and so on) do you have about interacting with other stakeholders?
- What hesitations do you think they might have about interacting with you?
- Why do you think these collaborations don’t already exist?
Unmet desire surveys elicit critical insights about needs for connection and are highly flexible. For instance, in the first question posed above, evaluation officers can choose to ask staff about new information that would be helpful for any program or only about information relevant to programs that are top priorities for their agency. In other words, unmet desire surveys need not add one more thing to the plate; rather, they can be used to accelerate collaboration directly tied to current learning priorities.
Unmet desire surveys also legitimize informal collaborative relationships. Too often, calls for new collaboration in the policy sphere immediately segue into overly structured meetings that fail to uncover promising areas for joint learning and problem-solving. Meetings across government agencies are often scripted presentations about each organization’s activities, providing little insight on ways they could partner to achieve better results. Policy discussions with outside research experts tend to focus on formal evaluations and long-term research projects that don’t surface opportunities to accelerate learning in the near term. In contrast, unmet desire surveys explicitly legitimize the idea that diverse thinkers may want to connect only for informal knowledge exchange rather than formal events or partnerships. Indeed, even single conversations can greatly impact decision-makers, and, of course, so can more intensive relationships.
While online platforms for spurring new collaborative relationships have been previously proposed, they have not achieved uptake at scale among federal policymakers. One reason for this is that the problem that needs to be solved is both factual and relational. In other words, the issue isn’t simply that strangers do not know each other—it’s also that strangers do not always know how to talk to one another. People care about how others relate to them and whether they can successfully relate to others. Uncertainty about relationality routinely stops people from interacting with others they do not know. This is why unmet desire surveys also include questions that directly measure hesitations about interacting with people from other agencies and organizations.
After the surveys are administered, evaluation staff can use survey data to engage in matchmaking: brokering connections among people with similar goals but diverse expertise and helping overcome uncertainty about relationality so that new cross-agency and cross-sector collaborative relationships can take root. In sum, by deliberately inquiring about connections with others who have diverse forms of relevant expertise—and then making those connections anew—evaluation staff can generate greater enthusiasm and ownership among people who may not consider evaluation and evidence-building as part of their core responsibilities.
Plan of Action
Using existing authorities and resources, federal evaluation officers can take three steps to position unmet desire surveys as a standard component of the government’s evidence toolbox.
Step 1. Design and implement pilot unmet desire surveys.
Chief evaluation officers are well positioned to pilot unmet desire surveys within their agencies. While individual evaluation officers can work independently to design unmet desire surveys, it may be more fruitful to work together, via the Evaluation Officer Council, to design a baseline survey template. Chief evaluation officers could then work with their teams to adapt the baseline template to their agencies, including identifying which agency staff to prioritize as well as the best way to phrase particular questions (e.g., regarding the types of connections that employees want in order to improve the effectiveness of their work or the types of hesitancies to ask about). Given that the question content is highly flexible, unmet desire surveys can directly accelerate learning agendas and build buy-in at the same time. Thus, they can yield tangible, concrete benefits with very little upfront cost.
Step 2. Meet unmet desires by matchmaking.
After the pilot surveys are administered, chief evaluation officers should act on their results by matchmaking. There are several ways to do this without new appropriations. One is for evaluation teams within agencies to engage in informal, low-lift matchmaking—wherein those who implement the survey also act as initial matchmakers—as an early proof of concept. A second option is to bring on short-term matchmakers through flexible hiring mechanisms (e.g., through the Intergovernmental Personnel Act). Documenting successes and lessons learned then set the stage for using agency-specific discretionary funds to hire one or more in-house matchmakers as longer-term or staff appointments.
Step 3. Collect information on successes and lessons learned from the pilot.
Unmet desire surveys can be tricky to field because they entail asking employees about topics they may not be used to thinking about. It often takes some trial and error to figure out the best ways to ask about employees’ substantive goals and their hesitations about interacting with people they do not know. Piloting unmet desire surveys and follow-on matchmaking can not only demonstrate value (e.g., the impact of new collaborative relationships fostered through these combined efforts) to justify further investment but also suggest how evaluation leads might best structure future unmet desire surveys and subsequent matchmaking.
Conclusion
An unmet desire survey is an adaptable tool that can reveal fruitful pathways for connection and collaboration. Indeed, unmet desire surveys leverage the science of collaboration by ensuring that efforts to broker connections among strangers consider both substantive goals and uncertainty about relationality. Chief evaluation officers can pilot unmet desire surveys using existing authorities and resources, and then use the information gathered to identify opportunities for productive matchmaking. Ultimately, positioning the survey as a standard component of the government’s evidence toolbox has great potential to support agency staff in advancing federal learning agendas and building a robust culture of evidence across the U.S. government.
The best place to start—especially when resources are limited—is with potential evidence champions. These are people who already have an idea of what information would help them improve the impact of the programs they run and which people would be helpful to collaborate with. These potential evidence champions may not self-identify as such; rather, they may see themselves as falling into other categories, such as customer-experience experts, bureaucracy hackers, process innovators, or policy entrepreneurs. Regardless of terminology, the unmet desire survey provides people who are already motivated to collaborate and connect with a clear opportunity to articulate their needs. Evaluation staff can then respond by matchmaking to stimulate new and productive relationships for those people.
The administrator should be someone with whom agency staff feel comfortable discussing their needs (e.g., a member of an agency evaluation team) and who is able to effectively facilitate matchmaking—perhaps because of their network or their reputation within the agency. The latter criterion helps ensure that staff expect useful follow-up, which in turn motivates completion of the survey and participation in follow-on activities; it also generates enthusiasm for engaging in new collaborative relationships (as well as creating broader buy-in for the learning agenda). In some cases, it may make the most sense to have multiple people from an evaluation team surveying different agency staff or co-sponsoring the survey with agency innovation offices. Explicit support from agency leadership for the survey and follow-on activities is also crucial for achieving staff buy-in.
The bulleted list in the body of the memo illustrates the types of questions that an unmet desire survey might ask. Yet survey content is meant to be tailored and agency-specific. For instance, the first suggested question about information that would help increase program effectiveness can be left entirely open-ended or be focused on programs related to learning-agenda priorities. Similarly, the second suggested question may invite responses related to either informal or formal collaboration, or instead may only ask about knowledge exchange (a relatively lower commitment that may be more palatable to agency leadership). The third and fourth questions should refer to specific types of hesitancy that survey administrators believe are most likely (e.g., ask about a few hesitancies that seem most likely to arise, such as lack of explicit permission, concerns about saying something inappropriate, or concerns about lack of trustworthy information). The final question about why these collaborations don’t exist can similarly be left broad or include a few examples to help spark ideas.
Again, the answer will be agency-specific. In many organizations, matchmaking happens informally. Formalizing this duty as a part of one or more people’s official responsibilities sends a signal about how much this work is valued. Exactly who those people are will depend on the agency’s structure, as well as on whether there are already people in a given agency who see matchmaking as part of their job.
While unmet desire surveys can be done anytime and on a continuous basis, it is best to field them when there is identified staff capacity for follow-on matchmaking and employee willingness to build collaborative relationships.
The incoming administration must act to address bias in medical technology at the development, testing and regulation, and market-deployment and evaluation phases.
The incoming administration should work towards encouraging state health departments to develop clear and well-communicated data storage standards for newborn screening samples.
Proposed bills advance research ecosystems, economic development, and education access and move now to the U.S. House of Representatives for a vote
NIST’s guidance on “Managing Misuse Risk for Dual-Use Foundation Models” represents a significant step forward in establishing robust practices for mitigating catastrophic risks associated with advanced AI systems.