Prompt engineering: adventures with ChatGPT, Bing, and Bard

About the use of new skills by KLS staff.

A relatively new skill, prompt engineering is simply getting language models like ChatGPT, Bing or Bard to perform tasks in a way you’d find useful.

You can get them to provide a summary or a tweet about BMJ Best Practice, but perhaps you’d prefer to tweak the content to suit a certain profession. You can prompt it to provide content about BMJ Practice for colleagues working in oncology, for example.

Prompting these tools can require a certain degree of creativity, and a little ‘trial and improvement’.

ChatGPT or Bard are both particularly useful for trying out new methods of prompting, as you can experiment easily without the query restrictions Bing has.

How can language models be prompted?

By making a ‘search thread’ in ChatGPT, I have prompted it to perform tasks to my specifications. This saves time; I can simply dip into this thread to ask it a query, and it will perform the task quickly without me needing to explain each time how I want it to present information.

I have a thread for translating those frustrating all-cap article titles to lower-case letters, and another for organising article information from RefWorks into tables.

ChatGPT requires human input to perform a task. I can ask it to create a simple Google search strategy, and it will do so. I can also ask it to create more complex search blocks for advanced search databases, which it will likely get wrong!

But there are creative solutions around this, and by wording my instructions in a particular way and having a quick natural language conversation with it, I can get better results.

Like any search strategy, the good ones often take a little more time to set up and requires both creativity and technical know-how. And Knowledge and Library professionals have these skills in abundance already!

I can ask it not to include MeSH (medical subject headings) to avoid it hallucinating incorrect subject headings. I can ask it to expand on its list of synonyms for keywords or break down a search into key components to design search blocks, before piecing everything back together in a database.

I don’t use search strategies entirely created by ChatGPT (it’s not perfect), but sometimes it does throw up some words or techniques that I can apply to my work.

I can also ask it to include URLs to lists of journal articles or professional bodies and embed the URLs in the titles. Just be sure to double-check these links, and if in doubt, hover your cursor/mouse over the links to check them before clicking.

The list isn’t exhaustive, there’s a lot of different tasks it can perform. It’s fair to say that my practice has changed significantly since I first started using these tools!

Beware of hallucinations and red herrings

Language models are designed to respond positively and helpfully to human queries, and sometimes they will simply make things up and present them as fact. There are lots of different reasons why they might do this, and it’s important to double-check information.

As an example, I asked a language model to provide me with a list of articles concerning university healthcare student attrition in the UK. It provided me a list of perfect-looking article references and URLs, but they were completely made up and the URLs took me to irrelevant pages on PubMed.

Sometimes these tools will also imply they can perform certain tasks, tasks which are beyond the scope of their abilities.

ChatGPT-3.5 for example, may agree to ‘summarise’ the content of articles if you provide it with a URL. Now, if the URL has words pertaining to your desired topic in the URL itself (as they sometimes do) it will simply use that information to generate a ‘summary’ based on the words in the URL, and not the actual content of the website.

ChatGPT-4 doesn’t generally do this, however, and is a much-improved tool.

Different prompts for different language models

Identifying the tools which are up for the job is also something to consider.

Bing can be prompted to quickly search for webpages and embed the URLs into the titles it presents to us. This can be a useful way to present long lists of current awareness documents.

I wouldn’t ask ChatGPT to search the internet for BBC News articles, because it does not have that capability.

In other words, knowing which tools work well for certain tasks continues to be a skill in itself!

ChatGPT, Bing and other tools have moved into the mainstream. Investment in similar technologies is set to boom over the coming years as interest grows in their capabilities, and organisations learn how to best utilise these tools to their advantage.

As tech companies hastily catch up with their own products, it’s possible that our users will start asking us about these tools and how they can be used effectively. Knowing what tools are useful to signpost to would be advantageous.

To summarise

 I’ve found the best way to learn about these tools is to simply play with them. Prompt engineering has its uses for search, marketing materials, and organising information. But there’s lots of different ways to use these tools, and it’s fascinating to see how others are using them!