Researching AI - Building an Education Chatbot

Hi - I'm Elliott, Taught by Human's newest Research Scientist focusing on how AI can be used in adult education. I have a PhD in Robotics and Autonomous Systems from University of Bristol focusing Swarm Robotics (large groups of robots that can co-operate locally to achieve complex tasks). I'm excited to share what I've been researching about using Large Language Models (LLMs) as a personalised educational chatbot.

Background - Why This is Hard

I’ve been exploring how LLMs can be used to help people learn about new topics using prompt engineering. Specifically, we wanted to test whether we can program an LLM to help people learn about new topics in a tailored manner.

Some terminology before we begin to help you understand this blog post:

• ChatGPT is OpenAI's chatbot. It is an interface for users to integrate with OpenAI's Large Language Models in a specific way (We have a resource on ChatGPT if you want to learn more)

• Large Language Model (LLM) are a form of AI which have learnt how to predict words from very large (much larger than previous) datasets of words and phrases (We have a resource on LLMs if you want to learn more)

• Model is a term used often in AI, it is the computer code or mathematics which the AI uses to find patterns and make predictions

Typically when asking ChatGPT about a certain topic, we will receive a long and generic answer to this question (we’ll examine this further in the next section). Instead, we want the model behind these responses to learn about the user, such as their background knowledge and goals, to provide more useful responses. There are some of the things we need to think about when trying to help people learn about new topics:

• Background knowledge - Typically it is difficult and time consuming to create a curriculum that is accessible to people with different background knowledge.

• Where to start - Some people may not know where to start when learning a new topic, or even what they should be looking for.

• Time - Dealing with varying time constraints. Some people may not want to take a long and in depth course, instead looking for high level knowledge.

I wanted to understand whether LLMs in general can help us solve some of these problems.

Introduction to the System Prompt with OpenAI Playground

Using system prompts we can program LLMs to behave in a certain manner when interacting with a user. For example, be friendly or sarcastic, reply like Shakespeare, only reply using rap lyrics. This involves us giving instructions in natural language to the AI model, telling it how to behave. This doesn’t require any traditional programming and we simply give the model instructions as if we were speaking to a person. If you would like to follow on and experiment with system prompting yourself you can do so through OpenAI playground.

The playground allows more control over how the model behaves in comparison to just using ChatGPT directly. Instead of one chat window, like in the ChatGPT interface, we have two dialog boxes shown rather than just one. The system box (on the left) is where we will be giving commands to the model to change it’s behaviour. On the right is the dialog entry part where we can act as a user and talk with the model.

Unlike the ChatGPT interface, we can actually go through our conversation and delete either our own entries to the model or responses of the model. This can be useful as the AI model will look back through our conversation with it when making new responses. Sometimes we might want to roll back to a certain point in the conversation when making changes to our prompts in the system dialog box. This can be useful as we might want to re-ask the same question and make sure the model forgets the response it gave before to the same question.

On the right we also have additional options such as choosing between different LLMs we want to test with and the maximum word limit when giving responses. In this case we won’t be changing any of these settings and will just be focussing on adding prompts into the system dialog box to control the models behaviour. In the OpenAI playground we are not actually using ChatGPT which is a refined version of their LLMs that has been tuned to behave as a chatbot in addition to the ChatGPT interface. In the playground we will be interacting directly with LLMs that have had no prior tuning.

One of the simplest ways we can change the behaviour of an LLM is asking it to respond with a certain tone or personality.

E - You will behave like a friendly helpful chatbot for the company "Taught By Humans". Your purpose is to help users learn about new topics and provide information.

Using the prompt above, we can first tell the model to respond with a certain tone, in this case, giving friendly responses. We can also tell the model specific information, telling it about our company and its intended purpose. Now when we add this additional prompt below we can instruct the model to tell the user about its purpose when it first interacts with it.

E - Introduce yourself when prompted and explain your purpose.

When we then act as a new user and simply say hello, we receive this response.

LLM- Hi there! I'm an AI chatbot from Taught By Humans. My purpose is to help you learn about new topics and provide information. How can I assist you today?

Using these few simple prompts we are able to tailor the model to behave in a way where we could imagine this being used as a friendly chatbot for our website.

Constructing a Basic Learning Guide

The main challenge I wanted to tackle with this experiment is getting the model to general a tailored plan which will help the user learn about a certain topic. In reality we don’t want the model to generate material that the user will learn from, instead, we just aim to test how well the model can produce a plan given some information about the user.

Normally if we asked ChatGPT about a certain topic it will give a generic response, relaying whatever information it has about a certain area. For example, if we prompt ChatGPT about Microsoft Excel (with none of the prior prompts) we receive this response.

E - Can you tell me about microsoft excel?

ChatGPT- Certainly! Microsoft Excel is a widely used spreadsheet application developed by Microsoft. It is a part of the Microsoft Office suite of productivity software and is available for both Windows and macOS operating systems. Excel is designed primarily for tasks involving numerical data, calculations, and data analysis, but it can also be used for a wide range of other tasks such as creating charts and graphs, organizing and managing data, and automating repetitive tasks.

Here are some key features and aspects of Microsoft Excel:

1. Spreadsheets: Excel uses a grid-based layout called a spreadsheet. Each spreadsheet is made up of rows and columns, and the intersection of a row and column is called a cell. Cells can contain text, numbers, formulas, and functions.

2. Formulas and Functions: Excel is known for its powerful formula and function capabilities. Formulas are used to perform calculations within cells, and functions are pre-built formulas that can be used to perform a wide range of tasks, such as arithmetic operations, statistical analysis, and more.

3…

Microsoft Excel is used in a variety of fields, including finance, accounting, engineering, data analysis, project management, and more. It's an essential tool for businesses and individuals alike, thanks to its versatility and the ability to handle a wide range of tasks related to data and calculations.

Whilst this provides us with a lot of information, listing up to 10 individual features of excel, it’s not particularly helpful for someone who has never used excel before. To improve this, we want our model to learn some information about the user before then recommending specific topics they should explore. Rather than just having the model giving a single response, we can use system prompts to tell the model that when asked about a topic, it should first ask a series of follow up questions.

To do this we first need to explicitly tell that model to ask questions when asked about any topic.

E - When the user asks for information about a certain topic, ask a series of follow up questions.

Following this we can include the specific questions we want the model to ask.

E - First, how much existing knowledge they have of the topic.
Second, why they want to learn about the topic.
Third, what are their goals when learning about this topic.
Explain that these questions will allow you to give the user the best experience with their learning.

In this case we want the model to give a more valuable response by learning about their existing knowledge, why they want to learn about the topic, and whether they have a specific goal they’re trying to reach. By asking these questions we want the final response to be more useful to the user, pointing to specific topics that they need to try to learn. Here we have also told the model to make it clear to the user that it will now ask some follow up questions and why it’s doing this as not to confuse the user.

Importantly, when all the questions have been asked, the model will give a final response giving a sequence of topics that the user should attempt to learn about.

E - When the user has provided this information, create a plan of topics that the user should learn about. This should be a sequence that leads to their goal.

In this case, an actual end product might instead create tailored and unique curriculum which directs the user to specific set of tutorials which have been designed by humans.

So now if the user asks about excel we receive this response.

E - I'd like to learn about how to use excel

LLM- Great! Before we begin, could you please let me know how much existing knowledge you have about using Excel? This will help me tailor the information to your level of expertise.

Whereas before the model gave a long generic response, the model has now responded with a question, also explaining why it wants to know this information. With our prompts the conversation continues as follows.

E - I want to learn about more of the functions in excel such as programming

LLM- That's great to hear! Learning more about Excel functions and programming can be very useful. Now, could you please share what your goals are when it comes to learning about Excel functions and programming? Are you looking to improve your efficiency in data analysis, automate tasks, or something else? Understanding your goals will help me create a plan that aligns with what you want to achieve.

E - I want to more efficiently track my businesses finances automatically when accounting for our different weekly costs, sales, and stock to calculate profit.

LLM- That sounds like a fantastic goal! To help you achieve that, I suggest the following plan of topics to learn about:

1. Basic Excel Functions: Start by familiarizing yourself with the basic functions in Excel, such as SUM, AVERAGE, MAX, MIN, and COUNT. These functions will help you perform calculations and analyze your data.

2. Data Organization: Learn how to effectively organize your data in Excel by using tables, sorting, filtering, and creating named ranges. This will make it easier for you to track your weekly costs, sales, and stock.

3. Formulas and Functions: Dive deeper into Excel formulas and functions. Explore more advanced functions like VLOOKUP, IF statements, and SUMIF/SUMIFS. These functions will allow you to automate calculations and perform more complex analysis.

4. PivotTables: Learn how to use PivotTables to summarize and analyze large amounts of data. PivotTables will help you gain insights into your business finances by allowing you to slice and dice your data in various ways.

5. Macros and VBA: Once you are comfortable with Excel functions, consider learning about macros and Visual Basic for Applications (VBA). This will enable you to automate repetitive tasks and create custom functions to suit your specific needs.

By following this plan, you will gain the necessary skills to efficiently track your business finances in Excel and automate calculations. Remember to practice and apply what you learn to real-world scenarios to reinforce your understanding.

Is there anything else I can assist you with?

What did we learn?

Using our designed prompts we were able to get much more valuable information from an LLM, allowing the model to ask questions back to the user, ultimately creating a much more useful final response. This demonstrates one of the major benefits of providing system prompts, allowing us to tailor the use of the model to more specific tasks without needing to retrain the model.

The final response has now given more specific information related to the user’s responses, pointing to specific functions in excel which will help with their specific problem. Following on from simple functions the model recommends exploring more of the complex topics which build upon each previous step, continuing to relate the information back to the user’s goal.

Another interesting result is that when asking about the user’s end goal, the model gave possible examples of how Excel could be used to help them with their challenges which we did not prompt the model to do. This could be useful for users which perhaps don’t have a specific goal in mind.

Using the approach we have explored here we could choose to adjust the questions that we want the model to ask to adjust their final response. For example, asking how much time they want to spend a week learning a topic could be useful in narrowing down on a smaller set of important areas the user should explore.

Challenges encountered

Getting the model to achieve a specific behaviour consistently can be challenging a process. How you word your prompts is very important and requires you to be very clear about what you want the model to do. Here’s an example of when the model didn’t behave as we wanted it to.

Before:

E - First, how much existing knowledge they have of the topic.
Second, why they want to learn about the topic.
Third, what are their goals when learning about this topic.

After:

E - When the user asks for information about a certain topic, ask series of follow up questions.
How much existing knowledge they have of the topic?
Why they want to learn about the topic?
What are their goals when learning about this topic?

Here there is a subtle difference that we are no longer telling the model the order in which to ask the questions, stating first, second, and third. Let’s see the new response.

LLM- That's great! Excel is a powerful tool for data analysis and organization. Before we get started, could you please let me know how much existing knowledge you have about using Excel? This will help me tailor the information to your needs.

E - I've used excel a little bit but only for basic things.

LLM- Great! It's good to know that you have some basic familiarity with Excel. Now, could you please tell me why you want to learn more about using Excel? Understanding your motivation will help me provide you with the most relevant information and resources.

E - I want to learn how to use programming in excel and more complex functions.

LLM- That's a fantastic goal! Learning how to use programming in Excel and exploring more complex functions can greatly enhance your data analysis and automation capabilities. Now, let's define your learning path and set some goals.

Starting Point: Since you have some basic knowledge of Excel, we can build upon that foundation. I recommend starting with the following topics:

1. Understanding Formulas and Functions: This will help you grasp the basics of Excel's built-in functions and how to use them effectively.

2. Data Analysis with PivotTables: PivotTables are powerful tools for summarizing and analyzing large datasets. Learning how to create and manipulate PivotTables will be beneficial.

3. Introduction to VBA (Visual Basic for Applications): VBA is the programming language used in Excel. Learning the basics of VBA will allow you to automate tasks and create custom functions.

End Goal: By the end of your learning journey, you should be able to confidently write complex formulas, use advanced functions, and have a good understanding of VBA programming in Excel.

Now, let's break down these topics into smaller learning objectives and create a plan. Are you ready to get started?

Looking at the new conversation we see that the model has not asked the user the third question regarding whether they have a specific goal in mind. Instead the model has confused the second question with the third. In this case, asking why the user wants to learn about the area and their specific goal are closely related which may be what has confused the model.

The nature of the final response is also slightly different with the model now giving their own indication of what the end goal should be. Despite this, the model has still given a useful final response.

The key takeaway here is that it can be challenging to get the model to behave in a very specific way and importantly, ensuring it will always behave in the same manner. Another way we can help control this is using the temperature parameter of the model. This is a value given between 0 and 1 and controls how random the model's responses are. If we want the model to always give similar responses to the same questions we can lower the temperature parameter closer to 0.

What’s Next?

In a future post we’ll expand on what has been discussed here but instead focussing on whether the information the model is telling us is correct and can truly be used for learning. It’s very important to validate that the model is providing the correct information and is not going to negatively impact anyones learning progression.

We have some resources if you would like to learn more about Large Language Models.