One of our core-funded programmes between 2015 and 2020, ‘Integrating research for policy and practice’ (CFS Programme 7), sought to improve our understanding of the impacts of Forest Research’s evidence and advice beyond the research community, especially within policy and practice, how this has been achieved, and how it can be enhanced, particularly through improved dialogue and collaboration.
Research impact evaluation framework
A framework was developed by Forest Research, with consultant Dr Laura Meagher, to improve how we understand, plan, evaluate and communicate the impacts of research, for example the contributions we make to environmental policy and practice. The framework built upon recent research into knowledge utilisation, plus our own experience of impact evaluation, and was piloted using a series of research impact case studies selected to represent the diversity of research projects led by Forest Research.
The framework asks us to consider three core evaluation questions:
Core evaluation questions
- Impacts: Who or what changed, in what ways, and how do we know?
- Causes of impact: Why/how did changes occur? Which factors or processes caused impact?
- Lessons and actions: What lessons can be learned? Which actions should follow to generate impact?
Impacts
Ia. What changed? (Progress towards goals)
- Instrumental: changes to plans, decisions, behaviours, practices, actions, policies
- Conceptual: changes to knowledge, awareness, attitudes, emotions
- Capacity-building: changes to skills and expertise
- Enduring connectivity: changes to the number and quality of relationships and trust
- Culture/attitudes towards knowledge exchange, and research impact itself
Ib. Who changed? (Influencers and influenced)
- Policy-makers (including regulatory bodies; local, national and international)
- Practitioners (public, private, NGO)
- Communities (of place or interest, general public)
- Researchers (within and beyond the project and institution)
- Other
Ic. How do we know? (Evidence and feedback)
- Which indicators and methods should be used, and questions asked, to demonstrate impacts, and progress towards generation of impacts?
Causes of impact
II. Why/how did changes occur?
- Problem-framing: Level of importance; tractability of the problem; active negotiation of research questions; appropriateness of research design.
- Research management: Research culture; integration between disciplines and teams; promotion of research services; planning; strategy.
- Inputs: Funding; staff capacity and turnover; legacy of previous work; access to equipment and resources.
- Outputs: Quality and usefulness of content; appropriate format.
- Dissemination: Targeted and efficient delivery of outputs to users and other audiences.
- Engagement: Level and quality of interaction with users and other stakeholders; co-production of knowledge; collaboration during design, dissemination and uptake of outputs.
- Users: Influence of knowledge intermediaries, e.g. ‘champions’ and user groups; incentives and reinforcement to encourage uptake.
- Context and contingencies: Societal, political, economic, biophysical, climate and geographical factors, risks and uncertainties.
Lessons and actions
III. Lessons learned for impact identification and generation?
- What worked? What could (or should) have been done differently?
- What could (or should) be done in the future?
The framework can be used in multiple ways:
- at any scale, from small projects led by a single researcher through to programmes implemented by any research organisation, where a number of cases representing the diversity of research activities can be selected for analysis.
- at any stage in the project or programme cycle: at the start to inform planning; mid-way to take stock and plan next steps; and at the end of a project to gain feedback, learn lessons, communicate impacts to external audiences and consider options for the future. (To use the framework at the start of a project, the question ‘what changed?’ might be reframed as ‘what changes are we aiming for, or what would success look like?’)
- by an individual researcher, or a research team or an end-user, but the quality of the findings, and lessons learned, would be enhanced if a wider range of stakeholders were involved.
When piloting the framework, we found that:
- the breadth of impact types stimulated researchers to think beyond ‘instrumental’ impact and identify other changes generated by their work.
- the generic list of ‘reasons for impact’ encouraged researchers to think about interrelationships that had led, or could lead, toward impacts, including less tangible reasons that might otherwise go unmentioned.
- asking for ‘lessons learned’ opened the way for researchers to step back and reflect critically on what had happened and what could be improved in the future.
Use of the framework can potentially transform the overall direction and detailed implementation of a research project, by providing a means to make the process of reflection more explicit, structured, thorough, balanced and credible, in a format that supports internal learning and external communication.
The framework also has the potential to inform changes to culture and procedures at an organisational level. It has been incorporated into Forest Research’s knowledge exchange and impact planning processes.
The framework has been adopted by other researchers and organisations beyond forestry in other applied fields and/or institutes. For example, it has since been used to assess the impact of agricultural research programmes by a government research agency in New Zealand.
Further reading
Journal article describing the framework: A framework to evaluate the impacts of research on policy and practice: A forestry pilot study – ScienceDirect
LSE Impact blog article: How to tell an impact story? The building blocks you need | Impact of Social Sciences (lse.ac.uk)
Research objectives
The objectives were to demonstrate how we can assess and improve the quality of:
- integration between scientists across FR and its research partners, by encouraging collaboration and interdisciplinarity where appropriate,
- knowledge exchange between the research community and external stakeholders through targeted engagement at appropriate stages in the research process, and
- research impact, i.e. changes in attitudes and behaviour that help realise the full range of benefits provided by trees, woods and forests into the future.
Findings and Recommendations
The programme addressed these objectives through four inter-linked activities:
- Investigations to improve practical and theoretical understanding of how to evaluate and enhance research integration and knowledge exchange, and of the diverse contexts within which scientific evidence can potentially support decision-making.
- Profiling, monitoring and evaluation of the status of research integration and knowledge exchange across FR and its research partners, the Forestry Commission and the wider land-use sector. This helped us understand the factors influencing research impact.
- A series of workshops and seminars to facilitate dialogue, collaboration and learning among Forest Research scientists, and between researchers and targeted groups of policymakers, practitioners and knowledge intermediaries across the land-use sectors.
- Case studies to understand and enhance the effectiveness of Forest Research’s response to specific research challenges identified in the current Science and Innovation Strategy.
Working with a consultant, Dr Laura Meagher, we developed and tested an evaluation framework that: a) incorporates lessons from experiences in academia, b) makes sense for an applied agency that already has a considerable impact on policy and practice, but could benefit from further outreach and communicating successes, and c) supports the culture of collaboration and learning which we value and wish to encourage in Forest Research.
We refined and tested the framework through a dozen case studies: the first phase was a self-evaluation by our own researchers (the report was submitted to the External Review Group in November 2017); the second phase involved eliciting feedback through interviews with stakeholders, plus subsequent analysis and dialogue, to generate a rounded picture of impacts and their causes to understand and communicate successes and identify how these can be enhanced.
We also ran several workshops to explore issues around knowledge exchange, interdisciplinarity and research impact. These had three goals: a) to develop a suitable format for a bespoke training session that could be rolled out if required, b) to further refine and test the framework and supporting materials, c) to explore options for embedding these into the business processes of Forest Research and the wider sector.
By 2020 we had developed clear proposals to implement this agenda, and in 2021 a Research Impact Coordinator was recruited to lead this work and develop and implement a Knowledge Exchange and Impact Strategy.
For further information, contact Dr David Edwards
Our Involvement
The programme is led by the Social and Economic Research Group (SERG) at Forest Research, who are collaborating with researchers across the agency and other forestry-sector stakeholders.
