Demonstrate the significant contribution of knowledge and library services to achieving the vision of Knowledge for Healthcare.
Evaluation framework - what it is
An Evaluation Framework is designed to provide a coherent approach to evaluating impact. It focuses on the difference a service wants to, or needs to make, then works systematically to identify how you will know you are getting there and what you need to do to make this likely. The overall emphasis in any impact evaluation framework is on achieving outcomes, where an outcome is defined as:
Quote
Changes in behaviour, relationships, activities or actions of people, groups and organisations with whom a programme works directly.
Earl, Outcome Mapping: Building Learning and Reflection into Development Programs.
The Evaluation framework has been developed to measure our achievement through phase 2 of Knowledge for Healthcare.
It provides a coherent approach to measuring the impact of Knowledge for Healthcare.
The framework will enable the evaluation of library and knowledge services delivery providing data to demonstrate impact, assure quality of services, measure performance and enable service improvement.
The framework therefore provides an outcomes-based, rather than process based approach, to evaluation and performance. The evidence collected will also support advocacy and raising the profile of library and knowledge services.
The Evaluation Framework is a good tool to support both service improvement and advocacy for your own service. Many of the indicators do relate to your own service and therefore will show how your service is contributing to the overall vision of Knowledge for Healthcare and against the six outcomes of the framework. The evidence you will be collecting for national submission will therefore also be valuable in the evaluation of your own service.
Developing the framework
The framework development has been facilitated by Sharon Markless using an evidence based process for evaluating the impact of library services which has been tested and adopted across the world. The methodology is based on an outcomes mapping approach which is used by business worldwide.
The methodology involved three key initial stages
- Agreeing the focus by identifying impact areas (where do you want to make a difference?) and then setting a specific objective in each area.
- Identifying the impact indicators for each objective that taken together will show whether the desired impact is occurring
- Planning evidence collection by identifying the most appropriate impact evidence for indicators.
The second part of the process involved identifying key actions that need to be taken to achieve the desired impacts and how to monitor these.
The group involved in the development of the framework included HEE’s strategic library and knowledge services leads, Chairs of Knowledge for Healthcare working groups and library and knowledge services managers.
The composition of the group was critical to ensure that the elements of the framework reflected both strategic and operational requirements to demonstrate transformational change
Those involved:
- Sue Lacey Bryant, Senior Advisor, Knowledge for Health, HEE
- Clare Edwards, Education Development Programme Lead, HEE West Midlands
- Louise Goswami, Head of Library and Knowledge Services Development, HEE Kent Surrey Sussex
- Helen Bingham, Head of Knowledge Services and Technology Enhanced Learning, HEE South
- David Stewart, Director of Health Libraries, HEE North
- Richard Osborn, Strategic Library Services Development Manager, HEE London
- Dominic Gilroy, Library and Knowledge Services Development Manager, HEE Yorkshire and the Humber
- Doug Knock, Library and Knowledge Services Manager, Kings College London NHS Foundation Trust
- Alan Fricker, Head of NHS Partnership and Liaison, Kings College London
The work to oversee the implementation of the Evaluation Framework is being led by the Quality and Impact Working Group.
Evaluation Framework and the Quality and Improvement Outcomes Framework
The Evaluation Framework is not the same as a Quality Assessment Framework.
The Evaluation Framework is concentrating on evaluating progression against the Knowledge for Healthcare vision while the Quality Framework is used to audit the quality of the service and provides a framework to focus, target and improve the quality of library and knowledge services.
A key priority of the Quality and Impact Working Group is to review and develop new quality standards for library and knowledge services across the NHS. Most of these new standards will align to the Evaluation Framework and the evidence collected will be used to demonstrate progress against both.
The new Evaluation Framework and Quality Standards will integrate in to HEE’s quality framework to assure the quality of library and knowledge services.
More national data/information collections
The framework will provide a clear purpose for the collection of KLS evidence, activity and statistical collections. The current collections will be reviewed to identify the data and information requirements and the frequency for collection. A single evaluation assessment process will be developed to support this process. It is not expected that the amount of data collected will increase however the focus will change.
Difference between the Evaluation Framework and the Value and Impact Toolkit
The Value and Impact Toolkit helps users to understand what impact is and how best to measure and demonstrate this for different stakeholders. It offers tools to assist in this process, including templates which can be used to record and demonstrate this impact to the wider organisation and beyond.
Evidence of impact case studies and narratives will be key evidence sources to demonstrate the outcome indicators within the Evaluation Framework. The Value and Impact Toolkit is therefore key in generating the evidence for the Evaluation Framework.
The Framework is based on the premise that it is usually very difficult to provide clear evidence of impact in complex systems such as healthcare.
The approach taken is to identify a series of indicators, which when taken together as a whole, would suggest that progress is being made in a particular area.
The Evaluation framework defines 6 impact objectives, each offering a clear statement of difference that will be made as Knowledge for Healthcare is successfully implemented.
The 6 Evaluation Framework objectives
Effective organisations
Impact objective 1
Organisations are more effective in mobilising evidence and internally generated knowledge.
Knowing a change has occured
- Organisations make active use of Knowledge and Library Services (KLS) expertise and value added services in mobilising evidence and organisational knowledge.
- Decisions are routinely informed by KLS provided evidence and best practice information.
- Tools, techniques and processes are used to capture and share organisational knowledge effectively.
- KLS staff are competent to articulate and deliver knowledge management effectively as part of everyday practice.
Demonstrating this
- Work with the executive team and across the organisation to identify needs and priorities.
- Implement an Action Plan to make better use of knowledge as an asset with the organisation or across the system.
- KLS provide evidence and best practice information to decision makers.
- Promote research enquiries; literature searches; synthesis and summaries; horizon scanning and briefings; alerting services to decision makers.
- Promote information products and knowledge sharing tools and techniques to develop awareness to decision makers.
- Individual librarians and knowledge specialists complete appropriate training and continuing professional development.
Patients, carers and the public
Impact objective 2
Patients, carers and the public are empowered to use information to make health and well-being choices.
Knowing a change has occured
- Healthcare staff draw on healthcare KLS expertise to inform the choices of patients and carers.
- Public library and third sector information staff draw on healthcare KLS expertise to empower patients, carers and the public.
- Healthcare KLS staff are competent in their role and activities related to patient, health and well-being information.
Demonstrating this
- Raise awareness of available training on high quality information sources on health and wellbeing to healthcare staff.
- Signpost high quality information sources on health and wellbeing to healthcare staff, public library and third sector.
- Establish local networks of health information providers to promote training opportunities and enable sharing of guidance, experience and resources around health and wellbeing information.
- Individual librarians and knowledge specialists complete appropriate training and continuing professional development.
Improved consistency
Impact Objective 3
Improved consistency and increased productivity and efficiency of Healthcare Library and Knowledge Services.
Knowing a change has occured
- Proportion of resources are pooled centrally to deliver healthcare KLS.
- Streamlined knowledge and library services and systems.
Demonstrating this
- Pool regional and local budgets for e-resources.
- Develop consistent e-resource portfolio through increased national procurement.
- Undertake collaborative procurement, with partners.
- Centralise and streamline functions at national level to avoid duplication.
- Establish and/or contribute to collaborative systems and mechanisms to deliver local library and knowledge services.
- Develop a business case to procure a national library management system.
- Develop training and marketing materials nationally for local use.
- Establish a collaborative nationwide interlibrary loan and document supply service.
- Collaborative systems and mechanisms deliver current awareness and alerting services.
Enhanced quality
Impact objective 4
Enhanced quality of healthcare knowledge and library services.
Knowing a change has occured
- All healthcare workforce members can freely access knowledge and library services.
- KLS meet professional practice, education, development and research information needs.
- The healthcare workforce, educators and learners routinely use KLS provided evidence, best practice information, services and resources.
- The healthcare workforce, educators and learners, have a positive experience of accessing and using KLS provided evidence, resources and services.
Demonstrating this
- Monitor number of organisations with an SLA for KLS.
- LKS collaborate to ensure that services, including procurement, meet needs.
- Optimise funding locally to ensure the KLS meets local needs.
- Targeting and tailoring of services to healthcare staff groups.
- KLS staff work with teams across their organisation(s) to identify user needs and priorities.
- KLS provide evidence and best practice information to healthcare workforce, educators and learners.
- KLS routinely participate in planning and implementation of education and CPD within the organisation.
- KLS routinely participate in planning and implementation of research and innovation.
- Provide appropriate information literacy training to meet needs.
- KLS undertake targeted promotion of resources and services to the healthcare workforce, educators and learners.
- A process for continuous service evaluation and performance monitoring, to drive service improvement, is in place.
- KLS workforce complete appropriate training and continuing professional development.
Partnership working
Impact Objective 5
Partnership working is the norm in delivering knowledge to healthcare.
Knowing a change has occured
- People from partner organisations feel committed to working collaboratively with healthcare KLS to achieve shared goals.
- Documents and/or systems mechanisms enabling partnership working are in place and used.
- Level of investment in library and knowledge services through working with stakeholders and partners.
- Sharing ideas generated by partnership working.
Demonstrating this
- Promote the value and benefits of partnership working.
- Seek feedback from partners annually.
- Regular meetings and other forms of networking take place with key partners.
- Joint actions are agreed and implemented.
- Service level agreements (SLAs), memorandum of understanding (MOUs) or some other forms of agreement are in place.
Increased capability
Impact Objective 6
Increased capability, confidence and capacity of LKS workforce.
Knowing a change has occured
- Capacity and structure of KLS workforce meets evolving needs of the organisation.
- Paraprofessional staff organise and deliver day to day customer facing services and library operations.
- Professional staff have embedded roles in healthcare teams.
- The confidence and capability of KLS staff.
Demonstrating this
- Undertake and analyse a biannual workforce audit.
- Clear management and staff structures put in place to optimise delivery of the service.
- Paraprofessional staff are supported to extend their role through training and current operational procedures.
- Professionally qualified KLS staff deliver the specialist KLS service.
- Extend reach and broaden the range of KLS delivery.
- Bi-annual Learning Development Needs Analysis is carried out and analysed with input from individuals, managers and KLS leads.
- Staff are trained to fulfil the requirements of their evolving roles.
This evidence-based framework is modelled on the Impact Planning Assessment used internationally with the Global Libraries Programme of the Bill and Melinda Gates Foundation.
Development and design of the framework has been facilitated with the support of Sharon Markless of Kings College London who has written Evaluating the Impact of Your Library with David Streatfield.