Generative AI in practice: EBSCO's AI Insights
About the testing of EBSCO AI Insights, a generative AI tool for summarising full-text.
Generative AI is increasingly being applied to content offered by publishers and aggregators.
AI Insights is a new feature which will be introduced by EBSCO to the new user interface for its Discovery Service and EBSCOhost. It generates summaries highlighting the key points of full-text articles. It is billed as ‘helping researchers quickly assess relevance of the article to their research and making better use of their research time.’
AI Insights is only available for articles for which EBSCO has obtained the appropriate licensing from journal publishers.
NHSE was asked to test and assess the feature with expert searchers working in KLS teams.
AI Insights was set up on a test instance of the NHS Knowledge and Library Hub and EBSCOhost in the new user interface for both CINAHL Ultimate and the Psychology and Behavioral Sciences Collection.
Expert searchers assessed it for:
-
The timeliness of response (how quickly the AI generated a summary).
-
Whether the tone used in the summary matched the tone of the article.
-
Whether the terminology used in the summary matched the terms used in the article.
-
The accuracy of the AI Insight.
-
How well the main themes of the AI insight covered those of the article.
-
How the AI Insight compared to the abstract of the article.
Testers were asked to assess the AI Insight against these criteria for:
-
The same article
and their choice of each of these types of article:
-
Systematic review
-
Randomised controlled trial
-
Guideline
-
Qualitative research
-
e-Book
Testers were also asked to re-assess two types of article a couple of weeks after their initial assessment.
Finally, our 17 testers gave their overall impression of the tool and what would make it even better. Testing took place during December 2024 and January 2025.
Timeliness
Timeliness varied, but was mostly as expected or quicker than expected. Sometimes an AI Insight loaded in steps rather than all at once.
Tone
Again, this varied. Sometimes it captured the academic and professional tone used in the article, at other times it adopted a declarative tone or one more aimed at the layperson.
Terminology
Sometimes the terminology used in the Insight was lifted directly from the article. Several testers commented that the Insight’s use of less jargon and simplified language was more accessible. They found it easier to make sense of the complexity found within the article.
Accuracy and main themes
Insights are broadly accurate in terms of the key themes they highlight. Testers felt that they relied on the conclusion and discussion rather than drawing on all elements of the paper. Key findings, study methodology, limitations of the study, and negative results in the paper were sometimes missing from the Insight.
While Insights are designed to give the key highlights, if some of the detail and nuance is missing, it may mislead the reader. Expert searchers would have appreciated more detail and more information about the study methodology and any limitations.
Comparison with the abstract
AI Insights were variable; sometimes similar to the abstract; sometimes they missed information thought important by testers. Insights seem to be aimed at the layperson rather than at clinicians or expert searchers. Testers also reported that occasionally the Insight did a better job of summarising the paper than the abstract.
The Insight did not necessarily add anything to the information which could have been gleaned from the abstract.
Overall impression
On the whole AI Insights generated summaries which were accurate in terms of the key themes included, easy to read and useful for identifying articles pertinent to the search question. The lack of detail or reporting of nuance and information felt to be useful by expert searchers was felt to be a barrier to use.
In addition, the fact that the Generate AI Insights option was not available for all results with full-text was felt to break the flow of skimming for relevant articles and make the process longer. There was concern that bias may be introduced if researchers bypass papers without Insights available.
Overall, testers report that AI Insights did not add anything to their experience over and above the abstract. They felt it might be a useful tool for identifying useful articles but not for generating summaries.
There was not enough detail; Insights dealt well with broad themes but lacked the detail and nuance, expert searchers expect.
Testers felt that it might be useful for general searchers and for those who might find the abstract inaccessible for example those with too much jargon or those dealing with complex topics.
Expert searchers wanted more control over what was included in the summary as well as the presentation of the Insight.
Even better
More control over the parts of the article to summarise and the way in which the information is presented were felt to be important. More consistency would give people confidence in the summary and give expert searchers more reason to use the feature. The ability to summarise across multiple articles would be useful as would the inclusion of a generated by AI disclaimer. Finally, for the feature to be more useful, it’s applicability to all the results whether the full-text is available or not would be appreciated.
And finally
The Resource Discovery’s Change Advisory Board looked into whether to switch on AI Insights in NHS Knowledge and Library Hub instances and EBSCOhost or not. Given the issues the testers raised to do with missing detail, inconsistency, the potential for bias and lack of customisation of the feature, it was decided not to switch it on until such time as these issues have been addressed.