Metrics

Metric aspect

E-resources; e-journals; e-books; decision support tools.

King's College London, Alan Fricker.

Metric definition

Number of active users on OpenAthens in a month.

Why is it important?

• can show users are aware and engaged with services
• shows meaningful activity to funders
• simple to collect in an area of high investment so demonstrate value for money for electronic resources
• active users shows breadth of engagement
• usage data counts in OpenAthens can be quite variable and feel less reliable than “has someone used something in a time period or not”

Process for compiling the metric

1. Generate report for required period in OpenAthens admin­.
2. In the data explorer select accounts, authentications and then set granularity to monthly.
3. Select start and end date of the month you want.
4. You can then see the Successful unique authentications.

What does it mean?

Compare performance during different time periods:

• month versus equivalent month in previous years

• pre and post promotion

Potential to benchmark against equivalent organisations.

Limitations

• changing staffing within the Trust
• IP enabled resources not captured
• not able to choose a period longer than a month
• figures not consistent with equivalent metric pre September 2018 no longer supported by OpenAthens

Desired outcomes

Increase number of active users versus equivalent periods.

Improvement plans

Promotional work.  Potential to use activity date by resource or user type to target this.

Reporting

Include in quarterly stats reports to library user boards. Include in annual reports.

Metric aspect

Quality assurance: measuring service quality.

King's College London, Alan Fricker

Metric definition

GMC Survey scores against access to educational resources and sub questions on library services, online Journals and space for private study. Use overall score, specialty outliers, positive versus negative satisfaction ratings.

Why is it important?

• key score for medical education in Trust
• high quality national data with good granularity from a core user group (can look at Trust, site and specialty).
• very high participation rate
• consistent year on year application
• not library delivered which reduces bias

Process for compiling the metric

1. Data from GMC Survey site - http://www.gmc-uk.org/education/surveys.asp. 
2. Overall score for Trust for access to educational resources from summary page.
3. Download scores for individual sub questions (click through the overall access to the educational resources score).
   • site by site data available but some question marks over accuracy of coding to sites
   • specialty data for outliers should be examined
4. Sentiment analysis by calculating (very good + good) – (very poor + poor) = sentiment score.

What does it mean?

• compares performance on different measures year on year
• compares shifts within specialties that have been targeted following red flags in previous years
• compares sites for local issues
• benchmarks against equivalent organisations
• be aware of wider issues within Trust / specialties that may have negative halo

Desired outcomes

• have useful conversation with medical education
• zero red flags for specialties
• improve absolute performance
• improve performance against benchmark Trusts

Improvement plans

Subject to areas highlighted and research on benchmark services.

Reporting

Results included in the annual report. Annual GMS Survey Report prepared for each Trust and discussed at library user boards. Annual benchmarking report prepared for library leadership team and wider library services.

Metric aspect

E-resources: e-journals; e-books; decision support tools.

North West Ambulance Service LKS, Matt Holland.

Metric definition

Ratio of OpenAthens accesses to resources per headcount.

The number of OpenAthens accesses to available resources in a given period divided by the Trust headcount. 

For example, for January 2017 using NWAS knowledge and library services (KLS) data, the calculation is 247 (OpenAthens accesses) / 5856 (NWAS headcount) = 0.04

Why is it important?

Open Athens accesses are impacted by all library activities to drive up the use of eresources. Where these are embedded in the library strategy and the marketing and are effectively implemented the ratio should be positively affected. The metric provides a simple number to evidence success (or lack of it).

Strategies and plans

Achieving a higher resource utilisation, evidenced by the ratio, could be a strategic objective in the KLS strategy and marketing plan.

Audience

The audiences are those external to the library including, other NHS health care library managers, Trust managers and other external stakeholders.

Process for compiling the metric

Data sources

The total number of Open Athens accesses to resources can be sourced from the OpenAthens administrative statistics module.  From the OpenAthens administrators dashboard for statistics, generate a report for required period.

Select the options, 'Total usage for all accounts at this level' and 'Usage total for all resources.'  Choose a start and end date.  

Typically, you can record the number for each month for a monthly presentation and for the previous twelve months for an annual percentage.

The number of employees is sourced from NHS Digital's 'Workforce Statistics.'  All things being equal the headcount will not change significantly over time.

You have the option for an annual ratio of taking the last headcount for the year you are calculating the ratio for or averaging the headcount over the previous 12 months. Be sure to make clear how the number you used was derived.

Limitations

Problems inherent in compiling OpenAthens accesses apply to this metric. Different profiles of use by your Trust staff will deliver the same ratio; a few super users can generate the same number of accesses as a larger number of infrequent users. 

The number of resources you provide access to and how you provide access will also influence the ratio.  While it is possible to compare with other Trusts, caution should be exercised.

Frequency

This metric can be repeated monthly or annually.

What does it mean?

This metric gives a single measure of how active Trust staff are in using eresources.  It provides a useful indication of how successful promotion of eresources is in affecting usage by library users. It is a useful complement to the 'Percentage of NHS Open Athens Accounts to Headcount'.  See also A note on Open Athens Statistics for a further explanation.

Desired outcomes

Improvement would be raising the ratio month on month or year on year.  If it is available the ratio can be compared to a regional average ration. An average is available for the North West region.

Improvement plans

Improvement in the ratio is the effect of the whole library effort (training, promotion, current awareness services etc. ) to promote us of eresources to users.  Effective promotion of eresources should cause an increase in the ratio over time.

Reporting

Reported on the monthly dashboard.

Reported on the annual report.

Metric aspect

User registration: library members and OpenAthens registrations.

North West Ambulance Service LKS, Matt Holland.

Metric definition

Percentage of NHS OpenAthens accounts to headcount.

Description

This metric uses NHS Open Athens as a proxy for user engagement to estimate the percentage of trust staff (FTE) who are potentially engaged with the Library and its eResources.

Calculate by (Number of Accounts / Headcount) * 100. For example, (2000 / 5000) * 100 = 40%

Why is it important?

Strategies and plans

Achieving a higher penetration of NHS OpenAthens accounts can be a strategic objective in the knowledge and library services (KLS) strategy and marketing plan.

Audience

The audiences are those external to the library including, other NHS health care library managers, Trust managers and other external stakeholders.

Process for compiling the Metric

Data Sources

The number of NHS Open Athens Accounts can be sourced from the front page of the Open Athens Administrative module. Typically, you can record the number on the first of the month for a monthly presentation and an average of the previous 12 months for an annual percentage.

For the annual percentage figure, you might also take the last recorded number for the period reported, e.g. 1 April each year. Be sure to make clear how the number you used was derived.

The number of employees is sourced from NHS Digital's 'Workforce Statistics.' 

Limitations

Trusts have different strategies for registering staff:

• staff may elect to have NHS OpenAthens accounts or be given them on contacting the library
• staff may be given an account on joining the Trust.  

Either strategy will effect the statistics and their comparability with other Trusts.  Another limitation is the imperfect match between accounts and current staff.  For example, unexpired accounts still on the system for staff who have left your organisation.

If you can identify a specific group of staff who have OpenAthens accounts, such as doctors, and a comparable headcount you can run this metric for these discrete groups.

Frequency

This metric can be repeated monthly or annually.

Repeatability

Any NHS Trust can create this metric.

What does it mean?

This is a simple, easy to understand metric.  Broadly the more NHS OpenAthens registrations, the more Trust staff have engaged with the library, at least registering for an account. 

For services without a physical library such as NWAS KLS, this is a useful measure of activity where other measures, for example physical visits to the library, are not available.   See also A note on Open Athens Statistics.

Desired outcomes

This metric works best against a benchmark.   For example, setting a benchmark to say 25% of Trust members will have registered for an NHS OpenAthens account. 

Achieving this benchmark or consistently exceeding it provides a measure against which to assess improvement.  NWAS KLS sets a benchmark of 25%.

Improvement plans

Raising the percentage of OpenAthens registrations is an objective in the NWAS KLS strategy. The means to achieve it are set out in the NWAS KLS marketing plan and the metric is reported in the monthly dashboard. 

Messages about OpenAthens are included in all the libraries communications, article in newsletters, meeting with students and with those running internal courses. 

The whole effort is aimed at a year on year increase in OpenAthens registrations and/or to exceed the benchmark of 25% of staff registered.

Reporting

Reported on the library monthly dashboard.

Reported on the library annual report.

The data is presented in the annual report, plotting the monthly percentage for the previous 12 months.

Metric aspect

Current Awareness: reach, click throughs

Brighton and Sussex NHS Trust, Ben Skinner.

Metric definition

Proportion of members registered for KnowledgeShare targeted evidence updates who receive at least one email in three months.

Why is it important?

Qualitative feedback shows that receiving regular KnowledgeShare evidence updates informs patient care and service decisions, research and education.

The number of emails received varies for each recipient in relation to their own professional interests and the coverage of content in KnowledgeShare.

KnowledgeShare is an expenditure, which includes a small amount of staff time for categorising members’ interests – this must be shown to be a good use of resources.

Process for compiling the metric

KnowledgeShare provides data quarterly showing all members who have signed up for targeted updates and how many emails each member has received in that time (called “FinalReportMembersCurrentAwareness”).

Using the Excel filter function, filter out those members who have been added to the system fewer than three months ago.  Excel displays how many remain.

Next, filter out those members who have received zero emails. Excel displays how many remain.

Calculate the proportion by dividing the second number by the first; this will be the proportion of members registered for more than three months that have received at least one email.

Changes to this proportion can be plotted quarterly.

NB. One can also calculate the average number of emails that members have received in this time period.  However, this is made less useful by the fact that members can choose the frequency with which they receive emails.

What does it mean?

The metric indicates whether KnowledgeShare is successfully delivering targeted evidence updates to your members.

For those members who do not receive emails it may be because:

• their professional interests have been set up too narrowly on the system and should be adjusted
• there is not sufficient content delivered through KnowledgeShare in their areas of interest (or both)

Desired outcomes

Ideally 100% of members would receive an email once per quarter but there are likely to be those whose interests are extremely niche, with little published evidence, who will receive relatively little content.

Suggested bandings would be:

• green = 95%-100%
• amber = 85%-94%
• red = 0%-84%

Improvement plans

Check individual members who are not receiving many updates.  Have they been categorised correctly? Can their categories be expanded (discuss with them where necessary)?

Consider whether the amount of content in a particular subject area needs to be increased. What sources would improve this coverage?

Discuss with suppliers of KnowledgeShare whether your team could offer to add content in this area, or whether there is capacity from another team.

Reporting

This could be reported in the annual report.

Metric aspect

Current Awareness: reach, click throughs.

Metric definition

Proportion of registered knowlerdge and library services (KLS) members who are registered for KnowledgeShare targeted evidence updates.

Why is it important?

Qualitative feedback shows that receiving regular KnowledgeShare evidence updates informs patient care and service decisions, research and education.

KnowledgeShare is an expenditure, which must be shown to be a good use of resources.

Process for compiling the metric

Figures for number of active (non-expired) registered LKS members can be obtained from the library management system. 

KnowledgeShare provides data quarterly showing all members who have signed up for targeted updates (called “FinalReportMembersCurrentAwareness”).

Registered KnowledgeShare members as a proportion of total LKS membership can therefore be determined.

Changes to this proportion can be plotted quarterly.

What does it mean?

The metric indicates whether KnowledgeShare is providing an extensive targeted evidence update function for your service. 

The system is designed to allow the creation of tailored updates for large numbers of members with little effort and so one should be aiming for a significant proportion of total users to be signed up.

Desired outcomes

Evidence updates will not be needed by some users – e.g. they may be of less use to undergraduates, estates and facilities staff, etc. Sign-up should be user-driven. However, increased promotion is likely to increase the number of recipients.

An increasing proportion would be expected over the first five years of implementation, with the increase eventually slowing down and stabilising at around 30%-40%.

Alternatively it would be possible to focus on particular staff groups, e.g. doctors-in-training, for whom one might expect a higher proportion.

Improvement plans

Increase promotion of the targeted evidence updates. Ensure that enquiry desk staff are able to promote the benefits when members join the library.

Reporting

This could be reported in the KLS annual report.