Quality Outcomes Framework 2019-22
Process, documentation and FAQs
Background
A review in 2018 established the need for a new approach focusing on service improvement aligned to Health Education England’s Knowledge for Healthcare Strategy and Quality Framework.
What are the six outcomes assessed by this framework?
These outcomes will be evaluated on a scale from level 0 to 4:
- All NHS organisations enable their workforce to freely access proactive knowledge and library services (KLS) that meet organisational priorities within the framework of Knowledge for Healthcare.
- All NHS decision making is underpinned by high quality evidence and knowledge mobilised by skilled library and knowledge specialists.
- Library and knowledge specialists identify the knowledge and evidence needs of the workforce in order to deliver effective and proactive services.
- All NHS organisations receive knowledge and library services provided by teams with the right skill mix to deliver on organisational and Knowledge for Healthcare priorities.
- Knowledge and library specialists improve the quality of knowledge and library services using evidence from research, innovation and good practice.
- Knowledge and library specialists demonstrate that their services make a positive impact on healthcare.
These new outcomes were outlined in HEE's 2019 Quality and Improvement Outcomes Framework for NHS Funded Knowledge and Library Services.
Contact the Knowledge for Healthcare team on [email protected] for this document in an accessible format.
What is the purpose of the outcomes?
The outcomes have a dual role, designed and developed to:
- drive progress in KLS improvement leading to increased satisfaction for users of the services
- provide a tool for NHS organisations to ensure that KLS specialists deliver a quality, high performing service; one that is continually developing and improving to meet the changing evidence and knowledge needs of organisations and individuals
Features of the outcomes
The outcomes are underpinned by four important features:
- a self-evaluation process focusing on KLS quality improvements delivered to the organisation or organisations served
- a whole team approach to planning and gathering information and evidence to support quality improvements made to the library service
- external validation to ensure consistency in self-evaluations and for sharing good practices between services
- a written report and service improvement action plan to demonstrate the service’s performance and indicate areas for further improvement
Guides and templates to measure Quality and Improvement Outcomes
Guidance documents
Quality and Improvement Outcomes Framework: Information.
Quality and Improvement Outcomes Framework.
Quality and Improvement Outcomes Framework: Overview - a shorter version of the framework.
Quality and Improvement Outcomes Framework: Handbook.
Appendix: Additional Guidance for evidence 2020-21 and COVID-19.
Quality and Improvement Outcomes Framework: Glossary.
Baseline Self-evaluation and Validation Process Flowchart.
Contact the Knowledge for Healthcare team on [email protected] for any of the documents in an accessible format.
Templates for use during self-evaluation
The following templates are required to be completed for the Outcomes Framework self-evaluation submission:
Self-evaluation and evidence report template .
Self-evaluated and validated levels report template.
Improvement planning worksheet (optional ).
How to embed evidence in an MS Word document.
Contact the Knowledge for Healthcare team on [email protected] for any of the documents in an accessible format.
Resources to help KLS specialists understand the new tools and prepare them for carrying out a self-evaluation using the Outcomes Framework.
Includes webinars, slides and scripts for each outcome. There are also some guest presentations from subject matter specialists.
Introduction to the Outcomes Framework
Download the slides and transcript.
Outcome 1
Strategic development of KLS with the aim that staff proactively deliver the priorities of both the organisation and Knowledge for Healthcare.
Listen to guest speaker Donald MacKay.
Download the slides and transcript.
Outcome 2
Facilitating the mobilisation of evidence and internally generated knowledge.
Guest speaker Ruth Carlyle.
Guest speakers Louise Goswami and Alison Day.
Download the slides and transcript
Outcome 3
Proactive stakeholder and user focused Knowledge and Library Services.
Guest speaker Natalie Parsley.
Guest speaker Morag Evans.
Download the slides and transcript.
Outcome 4
Are there sufficient KLS staff with the appropriate experience and skill mix to deliver and evolving services?
Guest speaker David Stewart.
Guest speaker Suzanne Wilson.
Download the slides and transcript.
Outcome 5
Showing that evidence based practice is integral to the planning, design, and delivery of activities.
Download the slides and transcript.
Outcome 6
Collecting evidence to demonstrate the impact of KLS.
Guest speaker Stephen Ayre.
Guest speaker Susan Smith.
Download the slides and transcript.
Reflective writing for the Outcomes Framework
Guidance webinar to help you complete the self assessment.
Contact the Knowledge for Healthcare team on [email protected] for any of the slides in an accessible format.
FAQs about the quality and improvement outcomes framework
Jump to section
Timeline and submission deadlines
Organisations
Which organisations need to do a self-evaluation for submission?
Each NHS organisation with an Education Contract (former Learning and Development Agreement) with Health Education England will need to carry out and submit a self-evaluation.
What guidance is there for Higher Education Institutions (HEIs)?
There is no separate guidance for HEIs. The generic guidance is detailed enough to assist in the self-evaluation process.
Any questions and concerns should be directed to the local LKS Lead.
Timeline and submission deadlines
What is the timeline for the baseline self-evaluation and validation process?
Submission deadline deferred until 24th September 2021.
Given Covid-19 pressures on NHS organisations, the current deadline for baseline submission from all NHS organisations with and LDA/Education Contract will now be September 2021.
Due to staffing issues/organisational changes, we would like to request an extension to the Outcomes Framework self-evaluation process
The baseline self-evaluation provides a snapshot of how the organisation and LKS meets the Outcomes Framework at the time of submission. Consequently we are unable to extend the deadline for submission.
Self-evaluation process
I want to do a preliminary self-evaluation. Is there any documentation that I can use?
Yes this is fine. Use the self-evaluation template from the documentation page and associated documents.
What will be the process for submitting the self-evaluation?
We are currently setting up a Libguides website for submission of the self-evaluation.
The website will be available from June 2021 for submission. Further detail and guidance on submission will be sent to all organisations once the website is set up.
Can you involve teams outside the LKS in the self-evaluation process?
While the Trust is responsible for the submission of the Outcomes Framework evaluation, the LKS manager will most likely complete it.
You may work with others across the organisation as well as other members of the team. It is not just for the manager to complete.
I think that some workshops for people to work together to look at this would be a good idea.
We welcome suggestions from colleagues about the workshops/topics might be of use, once the introductory webinars have been completed.
Discuss with your local HEE LKS Lead first. There may be opportunities to incorporate this into forthcoming meetings and events.
Can HEE review my submission in advance for one or more outcomes and let me know how my organisation is performing?
While we are happy to offer advice on specific issues, HEE does not have the capacity to undertake a comprehensive review of organisations’ self-valuations in advance of the baseline in 2021.
Outcome Levels
How do the levels work?
We spent a long time thinking about and devising the levels and indicators.
Using our knowledge, we considered how library and knowledge specialists operate at various levels of LKS development.
The step-progression from 1 and 2 to 3 and 4 is logical. You should find that you what is needed for levels 1 and 2 needs to be in place before you can address what is needed for 3 and 4.
How do the levels in each outcome relate to each other?
For each outcome the framework offers a spectrum of five levels. These range from:
Level 0, where a service is not developed, up to Level 4 which suggests a highly developed service in relation to the outcome.
Levels are cumulative. Each level builds on the previous one to enable service improvement.
How do we use the levels in each outcome for self-valuation and service improvement?
Within each level there are a series of indicators which suggest whether the level has been met.
If all indicators within a level have been met, and can be evidenced, this suggests the organisation and library and knowledge specialists have achieved that level and may be working at the next level.
How do we apply Low, Medium and High to each level?
If only some of the indicators within a level appear to have been met then the low, medium and high sub levels may provide a further way of tracking progress.
Low level suggests that you are at the initial stages for the particular level and can evidence this.
High suggests that you can demonstrate you are working fully within a level but as yet you are not working at the next level. This will be identified for service improvement and development.
Does the narrative need to describe and evidence why all levels have been reached or only the highest?
Yes the narrative needs to evidence both the level at which you have assessed your service and those below it.
In some cases it is not possible to evidence the higher level without evidencing the levels below.
However, where items are mentioned in lower levels and don’t automatically appear in the higher it is expected that the narrative and evidence captures these too.
I want to set myself a SMART target relating to achieving a level for the Outcomes Framework in the baseline year. How do I know what a realistic target might be?
The outcomes are not just about the LKS or individuals working within it. Rather it looks at how the organisation as a whole supports and makes use of its LKS.
Therefore, placing the responsibility for achieving certain levels in the Framework on the service – or individuals within it would be inappropriate.
The organisation will not evaluate as having a single level but rather a level per outcome. It may not be the case, or even likely, that the levels will be the same for every outcome.
To set a realistic, meaningful target for any of the outcomes in terms of a level for next year you would need to know where you are now. Otherwise the target could mean no progress at all or even going backwards.
This would require a self-evaluation on all the outcomes now and having this verified by HEE’s LKS leads. HEE will not be undertaking advance verifications of self-evaluations.
If there is a desire to set an Outcomes Framework associated SMART target we would advise looking at the indicators associated with the outcomes and linking the aims and objectives to these.
With outcome one for example the objective might be involving specific senior stakeholders within the organisation(s) served in the development of any LKS strategy or associated plans by X date.
Will validations be subject to the same traffic light system, i.e. red, amber and green services that the LQAF was?
There won’t be any overall “score” from the new process.
There will be a level for each outcome – so for example level 3 for outcome 1, level 2 for outcome 2.
HEE and the LKS Leads would have concerns about any services and Trusts who don’t engage with the process or provide a self-evaluation and plans for development.
There may also be interventions from HEE where specific problems or issues have been identified.
How will we report to our senior stakeholders if there is no overall score?
The validated self-valuation for your organisation will give a level for each outcome.
This includes a visual radar chart/spider graph demonstrating current levels and over time where improvements have been achieved.
This can be shared with stakeholders. The report will highlight areas of strength and good practice against all the outcomes, as well as areas for improvement. This will provide an accurate and more meaningful way of reporting.
If there is no overall score how will we benchmark our services?
The Quality and Improvement Outcomes Framework is a service improvement and development tool focusing on an individual organisation. Therefore benchmarking is not appropriate.
We are working with HEE Communications and HEE Quality teams on the communications to Trusts explain the new framework.
Quality leads will understand the new framework, levels and how the new process works.
Results will be reported to the national quality team and integrated with the wider quality education processes with which Trusts are familiar.
Evidence
How can I reflect the work that we have undertaken during the pandemic in my self-evaluation?
This guidance will help with this.
Contact the Knowledge for Healthcare team on [email protected] for this document in an accessible format.
Are you able to share good examples from LKS services that undertook the pilot last year?
We did for the pilot. However, some pilots followed the example too closely with the result that they didn’t fully demonstrate their own service and organisation.
It is difficult to pull together evidence examples as each organisation is different and will be working at different levels for each outcome.
For each outcome a list of suggested evidence has been provided. Don’t see these as a prescriptive list; there may be evidence more relevant to your organisation and service.
What is the maximum age of the evidence? For example in outcome 4 if you did a role redesign in 2017 would that count?
The evidence shows what you did and the outcomes for your service in the period being reviewed.
In this example it would only be relevant if the outcome of the role redesign was achieved during the period covered by your self-evaluation.
If you have a piece of evidence, for example an annual report, that covers all the organisations you cover how will that work if you are supposed to have a submission for each organisation?
The same piece of evidence can be used for each submission. You should ensure that, if it is generic, it covers the relevant organisations in sufficient detail to evidence the narrative at the levels you are demonstrating.
Can you use the same piece of evidence for more than one outcome?
Yes and we would encourage this approach. Think quality not quantity and cross reference against it for the different outcomes.
Ask the “So what?” question of the evidence. Why is it relevant and what does it tell us? This will help you identify “evidence” that is not appropriate so it can be omitted.
What happens when you are incorporated into a wider team for strategy, plan etc.? For example, I have 2 points in the corporate strategy.
If you are restricted in this way within wider corporate documents you will need to consider how you can further outline the plans and strategies of your service.
This may include using tools such as the Plan on the Page.
Will the Strategy and annual report be assessed separately?
No they will not. These are key documents giving a good overview of your LKS.
The strategy shows how the service is intended to develop and improve and the annual report shows what has been done over the last 12 months.
Elements of the documents could act as evidence for several of the outcomes. Cross-reference in relevant outcomes if needed.
Validation
There was a consistent national process put in place, based on the learning from undertaking the pilot of 12 services in 2018:
- The validation review was based on the guidance and principles provided, as set out in the validation section of the Framework Handbook, and the Evidence and Outcome Levels section of FAQs
- The aim was to establish a consistent national baseline, reviewing evidence from 1st April 2020 to 31st March 2021, with the option of including evidence from 1st April 2019 – 31st March 2020 where Covid had delayed activity in specific areas – see guidance
- The validation was based solely on the narrative and supporting evidence provided in the return.
- Each self-evaluation was reviewed by two validators to ensure consistency.
The validation for each self-evaluation was undertaken in two stages:
1. An individual review of the submission documentation
2. A meeting at which the validators discussed and reviewed their findings for each of the outcomes, agreed validated levels and compiled both the detailed and executive report
- Where queries arose, the validators escalated these to the national consistency panel.
- Throughout the validation consistency processes were put in place. These included sampling reports and validation decisions to ensure a consistent approach.
- All Executive Reports were reviewed to ensure consistency.
Resubmission
What type of evidence could I re-submit?
A range of different evidence documents can be included in your resubmission.
Evidence included needs to demonstrate the activity and outcome occurred in the period t1st April 2020 to 31st March 2021 – please see the guidance for evidence for 2020/21 and Covid 1
To consider additional evidence for re-submission you may wish to consider some of the feedback in the detailed report and the guidance and suggestions in the Outcomes Framework document.
There are some examples below of the different types of evidence that organisations included in their baseline submission:
Outcome 1
Some examples of evidence submitted by organisations for outcome 1:
A Board member promotes the role and value of the library and knowledge service
- Governance structure to demonstrate Board member engagement with knowledge and library service (This evidence would need to be in combination with evidence to demonstrate promotion)
- Opportunities and invitations to present to Board members and meetings, with the outcomes of the conversation e.g. minutes included from these meetings
- Social media cards or impact case vignettes from Board members used in promotion – both knowledge and library service promotion as well as other Trust promotion Regular tweets from a Board member to the organisation about the knowledge and library service
An approved strategy addresses Knowledge for Healthcare priorities, aligned to the goals and priorities of the organisation
- Strategy with the signed Board/Organisation committee cover sheet
- Copies of the approved minutes from the Board/Organisational committee that approved the strategy
- Formal communication, from the organisation or knowledge and library service, about the approval
A separately identified library and knowledge service budget allows for provision of a range of services and resources for users
A separately identified library and knowledge service budget allows for provision of a range of services and resources for users
- A copy of a budget statement – to show ownership by the knowledge and library service and sufficient funding
- Inclusion of the budget/expenditure overview in the annual report
- Extract from part 2 statistics - income and expenditure
With all the above they need to show ownership by the knowledge and library service and sufficient funding available for the resources
Outcome 2
Some examples of evidence submitted by organisations for outcome 2:
Evidence search services provided by the library and knowledge specialist support
- clinical decision making and
- non-clinical decision making
Evidence that demonstrates the searches undertaken and use of the search service:
- Extracts from spreadsheets that contain the details of clinical searches undertaken
- Extracts from spreadsheets that contain the details of non-clinical searches undertaken
- A range of impact case studies and vignettes, of both clinical and non-clinical decision making, gathered from literature searched carried out
- Inclusion in annual reports of searches undertaken and/or impact vignettes of both clinical and non-clinical decision making
Outcome 4
Some examples of evidence submitted by organisations for outcome 4:
A qualified library and knowledge specialist actively leads the service.
- Certificates of qualifications
- CILIP Professional Registration at Chartered level or above
Library and Knowledge specialist skills and capacity are considered in service planning
- Emails/Discussions with line managers/senior colleagues identifying additional capacity requirements
- Capacity and skills requirements included in strategy plans
- Reviews of team member activities
- Business cases for restructure
- Review of capacity against the recommended Staff Ratio
How will re-submitted evidence be reviewed and validated?
To ensure consistency across the process the same validation principles and guidance will be used. Please see the answer to the question 'How was my self-evaluation submission validated?'
Background
A review in 2018 established the need for a new approach focusing on service improvement aligned to Health Education England’s Knowledge for Healthcare Strategy and Quality Framework.
What are the six outcomes assessed by this framework?
These outcomes will be evaluated on a scale from level 0 to 4:
- All NHS organisations enable their workforce to freely access proactive knowledge and library services (KLS) that meet organisational priorities within the framework of Knowledge for Healthcare.
- All NHS decision making is underpinned by high quality evidence and knowledge mobilised by skilled library and knowledge specialists.
- Library and knowledge specialists identify the knowledge and evidence needs of the workforce in order to deliver effective and proactive services.
- All NHS organisations receive knowledge and library services provided by teams with the right skill mix to deliver on organisational and Knowledge for Healthcare priorities.
- Knowledge and library specialists improve the quality of knowledge and library services using evidence from research, innovation and good practice.
- Knowledge and library specialists demonstrate that their services make a positive impact on healthcare.
These new outcomes were outlined in HEE's 2019 Quality and Improvement Outcomes Framework for NHS Funded Knowledge and Library Services.
Contact the Knowledge for Healthcare team on [email protected] for this document in an accessible format.
What is the purpose of the outcomes?
The outcomes have a dual role, designed and developed to:
- drive progress in KLS improvement leading to increased satisfaction for users of the services
- provide a tool for NHS organisations to ensure that KLS specialists deliver a quality, high performing service; one that is continually developing and improving to meet the changing evidence and knowledge needs of organisations and individuals
Features of the outcomes
The outcomes are underpinned by four important features:
- a self-evaluation process focusing on KLS quality improvements delivered to the organisation or organisations served
- a whole team approach to planning and gathering information and evidence to support quality improvements made to the library service
- external validation to ensure consistency in self-evaluations and for sharing good practices between services
- a written report and service improvement action plan to demonstrate the service’s performance and indicate areas for further improvement
Guides and templates to measure Quality and Improvement Outcomes
Guidance documents
Quality and Improvement Outcomes Framework: Information.
Quality and Improvement Outcomes Framework.
Quality and Improvement Outcomes Framework: Overview - a shorter version of the framework.
Quality and Improvement Outcomes Framework: Handbook.
Appendix: Additional Guidance for evidence 2020-21 and COVID-19.
Quality and Improvement Outcomes Framework: Glossary.
Baseline Self-evaluation and Validation Process Flowchart.
Contact the Knowledge for Healthcare team on [email protected] for any of the documents in an accessible format.
Templates for use during self-evaluation
The following templates are required to be completed for the Outcomes Framework self-evaluation submission:
Self-evaluation and evidence report template .
Self-evaluated and validated levels report template.
Improvement planning worksheet (optional ).
How to embed evidence in an MS Word document.
Contact the Knowledge for Healthcare team on [email protected] for any of the documents in an accessible format.
Resources to help KLS specialists understand the new tools and prepare them for carrying out a self-evaluation using the Outcomes Framework.
Includes webinars, slides and scripts for each outcome. There are also some guest presentations from subject matter specialists.
Introduction to the Outcomes Framework
Download the slides and transcript.
Outcome 1
Strategic development of KLS with the aim that staff proactively deliver the priorities of both the organisation and Knowledge for Healthcare.
Listen to guest speaker Donald MacKay.
Download the slides and transcript.
Outcome 2
Facilitating the mobilisation of evidence and internally generated knowledge.
Guest speaker Ruth Carlyle.
Guest speakers Louise Goswami and Alison Day.
Download the slides and transcript
Outcome 3
Proactive stakeholder and user focused Knowledge and Library Services.
Guest speaker Natalie Parsley.
Guest speaker Morag Evans.
Download the slides and transcript.
Outcome 4
Are there sufficient KLS staff with the appropriate experience and skill mix to deliver and evolving services?
Guest speaker David Stewart.
Guest speaker Suzanne Wilson.
Download the slides and transcript.
Outcome 5
Showing that evidence based practice is integral to the planning, design, and delivery of activities.
Download the slides and transcript.
Outcome 6
Collecting evidence to demonstrate the impact of KLS.
Guest speaker Stephen Ayre.
Guest speaker Susan Smith.
Download the slides and transcript.
Reflective writing for the Outcomes Framework
Guidance webinar to help you complete the self assessment.
Contact the Knowledge for Healthcare team on [email protected] for any of the slides in an accessible format.
FAQs about the quality and improvement outcomes framework
Jump to section
Timeline and submission deadlines
Organisations
Which organisations need to do a self-evaluation for submission?
Each NHS organisation with an Education Contract (former Learning and Development Agreement) with Health Education England will need to carry out and submit a self-evaluation.
What guidance is there for Higher Education Institutions (HEIs)?
There is no separate guidance for HEIs. The generic guidance is detailed enough to assist in the self-evaluation process.
Any questions and concerns should be directed to the local LKS Lead.
Timeline and submission deadlines
What is the timeline for the baseline self-evaluation and validation process?
Submission deadline deferred until 24th September 2021.
Given Covid-19 pressures on NHS organisations, the current deadline for baseline submission from all NHS organisations with and LDA/Education Contract will now be September 2021.
Due to staffing issues/organisational changes, we would like to request an extension to the Outcomes Framework self-evaluation process
The baseline self-evaluation provides a snapshot of how the organisation and LKS meets the Outcomes Framework at the time of submission. Consequently we are unable to extend the deadline for submission.
Self-evaluation process
I want to do a preliminary self-evaluation. Is there any documentation that I can use?
Yes this is fine. Use the self-evaluation template from the documentation page and associated documents.
What will be the process for submitting the self-evaluation?
We are currently setting up a Libguides website for submission of the self-evaluation.
The website will be available from June 2021 for submission. Further detail and guidance on submission will be sent to all organisations once the website is set up.
Can you involve teams outside the LKS in the self-evaluation process?
While the Trust is responsible for the submission of the Outcomes Framework evaluation, the LKS manager will most likely complete it.
You may work with others across the organisation as well as other members of the team. It is not just for the manager to complete.
I think that some workshops for people to work together to look at this would be a good idea.
We welcome suggestions from colleagues about the workshops/topics might be of use, once the introductory webinars have been completed.
Discuss with your local HEE LKS Lead first. There may be opportunities to incorporate this into forthcoming meetings and events.
Can HEE review my submission in advance for one or more outcomes and let me know how my organisation is performing?
While we are happy to offer advice on specific issues, HEE does not have the capacity to undertake a comprehensive review of organisations’ self-valuations in advance of the baseline in 2021.
Outcome Levels
How do the levels work?
We spent a long time thinking about and devising the levels and indicators.
Using our knowledge, we considered how library and knowledge specialists operate at various levels of LKS development.
The step-progression from 1 and 2 to 3 and 4 is logical. You should find that you what is needed for levels 1 and 2 needs to be in place before you can address what is needed for 3 and 4.
How do the levels in each outcome relate to each other?
For each outcome the framework offers a spectrum of five levels. These range from:
Level 0, where a service is not developed, up to Level 4 which suggests a highly developed service in relation to the outcome.
Levels are cumulative. Each level builds on the previous one to enable service improvement.
How do we use the levels in each outcome for self-valuation and service improvement?
Within each level there are a series of indicators which suggest whether the level has been met.
If all indicators within a level have been met, and can be evidenced, this suggests the organisation and library and knowledge specialists have achieved that level and may be working at the next level.
How do we apply Low, Medium and High to each level?
If only some of the indicators within a level appear to have been met then the low, medium and high sub levels may provide a further way of tracking progress.
Low level suggests that you are at the initial stages for the particular level and can evidence this.
High suggests that you can demonstrate you are working fully within a level but as yet you are not working at the next level. This will be identified for service improvement and development.
Does the narrative need to describe and evidence why all levels have been reached or only the highest?
Yes the narrative needs to evidence both the level at which you have assessed your service and those below it.
In some cases it is not possible to evidence the higher level without evidencing the levels below.
However, where items are mentioned in lower levels and don’t automatically appear in the higher it is expected that the narrative and evidence captures these too.
I want to set myself a SMART target relating to achieving a level for the Outcomes Framework in the baseline year. How do I know what a realistic target might be?
The outcomes are not just about the LKS or individuals working within it. Rather it looks at how the organisation as a whole supports and makes use of its LKS.
Therefore, placing the responsibility for achieving certain levels in the Framework on the service – or individuals within it would be inappropriate.
The organisation will not evaluate as having a single level but rather a level per outcome. It may not be the case, or even likely, that the levels will be the same for every outcome.
To set a realistic, meaningful target for any of the outcomes in terms of a level for next year you would need to know where you are now. Otherwise the target could mean no progress at all or even going backwards.
This would require a self-evaluation on all the outcomes now and having this verified by HEE’s LKS leads. HEE will not be undertaking advance verifications of self-evaluations.
If there is a desire to set an Outcomes Framework associated SMART target we would advise looking at the indicators associated with the outcomes and linking the aims and objectives to these.
With outcome one for example the objective might be involving specific senior stakeholders within the organisation(s) served in the development of any LKS strategy or associated plans by X date.
Will validations be subject to the same traffic light system, i.e. red, amber and green services that the LQAF was?
There won’t be any overall “score” from the new process.
There will be a level for each outcome – so for example level 3 for outcome 1, level 2 for outcome 2.
HEE and the LKS Leads would have concerns about any services and Trusts who don’t engage with the process or provide a self-evaluation and plans for development.
There may also be interventions from HEE where specific problems or issues have been identified.
How will we report to our senior stakeholders if there is no overall score?
The validated self-valuation for your organisation will give a level for each outcome.
This includes a visual radar chart/spider graph demonstrating current levels and over time where improvements have been achieved.
This can be shared with stakeholders. The report will highlight areas of strength and good practice against all the outcomes, as well as areas for improvement. This will provide an accurate and more meaningful way of reporting.
If there is no overall score how will we benchmark our services?
The Quality and Improvement Outcomes Framework is a service improvement and development tool focusing on an individual organisation. Therefore benchmarking is not appropriate.
We are working with HEE Communications and HEE Quality teams on the communications to Trusts explain the new framework.
Quality leads will understand the new framework, levels and how the new process works.
Results will be reported to the national quality team and integrated with the wider quality education processes with which Trusts are familiar.
Evidence
How can I reflect the work that we have undertaken during the pandemic in my self-evaluation?
This guidance will help with this.
Contact the Knowledge for Healthcare team on [email protected] for this document in an accessible format.
Are you able to share good examples from LKS services that undertook the pilot last year?
We did for the pilot. However, some pilots followed the example too closely with the result that they didn’t fully demonstrate their own service and organisation.
It is difficult to pull together evidence examples as each organisation is different and will be working at different levels for each outcome.
For each outcome a list of suggested evidence has been provided. Don’t see these as a prescriptive list; there may be evidence more relevant to your organisation and service.
What is the maximum age of the evidence? For example in outcome 4 if you did a role redesign in 2017 would that count?
The evidence shows what you did and the outcomes for your service in the period being reviewed.
In this example it would only be relevant if the outcome of the role redesign was achieved during the period covered by your self-evaluation.
If you have a piece of evidence, for example an annual report, that covers all the organisations you cover how will that work if you are supposed to have a submission for each organisation?
The same piece of evidence can be used for each submission. You should ensure that, if it is generic, it covers the relevant organisations in sufficient detail to evidence the narrative at the levels you are demonstrating.
Can you use the same piece of evidence for more than one outcome?
Yes and we would encourage this approach. Think quality not quantity and cross reference against it for the different outcomes.
Ask the “So what?” question of the evidence. Why is it relevant and what does it tell us? This will help you identify “evidence” that is not appropriate so it can be omitted.
What happens when you are incorporated into a wider team for strategy, plan etc.? For example, I have 2 points in the corporate strategy.
If you are restricted in this way within wider corporate documents you will need to consider how you can further outline the plans and strategies of your service.
This may include using tools such as the Plan on the Page.
Will the Strategy and annual report be assessed separately?
No they will not. These are key documents giving a good overview of your LKS.
The strategy shows how the service is intended to develop and improve and the annual report shows what has been done over the last 12 months.
Elements of the documents could act as evidence for several of the outcomes. Cross-reference in relevant outcomes if needed.
Validation
There was a consistent national process put in place, based on the learning from undertaking the pilot of 12 services in 2018:
- The validation review was based on the guidance and principles provided, as set out in the validation section of the Framework Handbook, and the Evidence and Outcome Levels section of FAQs
- The aim was to establish a consistent national baseline, reviewing evidence from 1st April 2020 to 31st March 2021, with the option of including evidence from 1st April 2019 – 31st March 2020 where Covid had delayed activity in specific areas – see guidance
- The validation was based solely on the narrative and supporting evidence provided in the return.
- Each self-evaluation was reviewed by two validators to ensure consistency.
The validation for each self-evaluation was undertaken in two stages:
1. An individual review of the submission documentation
2. A meeting at which the validators discussed and reviewed their findings for each of the outcomes, agreed validated levels and compiled both the detailed and executive report
- Where queries arose, the validators escalated these to the national consistency panel.
- Throughout the validation consistency processes were put in place. These included sampling reports and validation decisions to ensure a consistent approach.
- All Executive Reports were reviewed to ensure consistency.
Resubmission
What type of evidence could I re-submit?
A range of different evidence documents can be included in your resubmission.
Evidence included needs to demonstrate the activity and outcome occurred in the period t1st April 2020 to 31st March 2021 – please see the guidance for evidence for 2020/21 and Covid 1
To consider additional evidence for re-submission you may wish to consider some of the feedback in the detailed report and the guidance and suggestions in the Outcomes Framework document.
There are some examples below of the different types of evidence that organisations included in their baseline submission:
Outcome 1
Some examples of evidence submitted by organisations for outcome 1:
A Board member promotes the role and value of the library and knowledge service
- Governance structure to demonstrate Board member engagement with knowledge and library service (This evidence would need to be in combination with evidence to demonstrate promotion)
- Opportunities and invitations to present to Board members and meetings, with the outcomes of the conversation e.g. minutes included from these meetings
- Social media cards or impact case vignettes from Board members used in promotion – both knowledge and library service promotion as well as other Trust promotion Regular tweets from a Board member to the organisation about the knowledge and library service
An approved strategy addresses Knowledge for Healthcare priorities, aligned to the goals and priorities of the organisation
- Strategy with the signed Board/Organisation committee cover sheet
- Copies of the approved minutes from the Board/Organisational committee that approved the strategy
- Formal communication, from the organisation or knowledge and library service, about the approval
A separately identified library and knowledge service budget allows for provision of a range of services and resources for users
A separately identified library and knowledge service budget allows for provision of a range of services and resources for users
- A copy of a budget statement – to show ownership by the knowledge and library service and sufficient funding
- Inclusion of the budget/expenditure overview in the annual report
- Extract from part 2 statistics - income and expenditure
With all the above they need to show ownership by the knowledge and library service and sufficient funding available for the resources
Outcome 2
Some examples of evidence submitted by organisations for outcome 2:
Evidence search services provided by the library and knowledge specialist support
- clinical decision making and
- non-clinical decision making
Evidence that demonstrates the searches undertaken and use of the search service:
- Extracts from spreadsheets that contain the details of clinical searches undertaken
- Extracts from spreadsheets that contain the details of non-clinical searches undertaken
- A range of impact case studies and vignettes, of both clinical and non-clinical decision making, gathered from literature searched carried out
- Inclusion in annual reports of searches undertaken and/or impact vignettes of both clinical and non-clinical decision making
Outcome 4
Some examples of evidence submitted by organisations for outcome 4:
A qualified library and knowledge specialist actively leads the service.
- Certificates of qualifications
- CILIP Professional Registration at Chartered level or above
Library and Knowledge specialist skills and capacity are considered in service planning
- Emails/Discussions with line managers/senior colleagues identifying additional capacity requirements
- Capacity and skills requirements included in strategy plans
- Reviews of team member activities
- Business cases for restructure
- Review of capacity against the recommended Staff Ratio
How will re-submitted evidence be reviewed and validated?
To ensure consistency across the process the same validation principles and guidance will be used. Please see the answer to the question 'How was my self-evaluation submission validated?'
Page last reviewed: 27 August 2021