On Monday (13th May 2024) OpenAI had their Spring Update - watch here if you’re interested. They announced a few things - one of which was a new model GPT-4o (where the o stands for Omni). Some amazing features being added to the free version of ChatGPT and a desktop version of ChatGPT have been launched. But this blog is going to focus on using the GPT-4o model in your own applications, like DOT-ed.

How Easy is it to Switch?

For DOT-ed we are using the OpenAI APIs (this means writing code, rather than just playing around with ChatGPT). Having already done the switch from GPT-4 to GPT-4-turbo last year, I was hoping this would be as easy.

Which it was - it was literally as easy as deleting some text and replacing it with an “o”.

How Do the Models Compare?

To determine the difference between the previous GPT-4-turbo model, and the new GPT-4o model, I decided to run through the DOT-ed flow with exactly the same responses from the learner.

In case you don’t know, our platform DOT-ed has a chatbot Dotly who asks a few questions to get to know our learners, then personalises a learning journey using our content. It is really several chatbots all working together in the background.

Part 1 - Getting to Know the Learner

Screen showing a chat between a learner and Dotly

Learner: I'm a data analyst - I really want to become better at google sheets.

Old Model	New Model
That's fantastic, Google Sheets can be a powerful tool for data analysis! To tailor your learning experience, could you tell me a bit about your existing knowledge of Google Sheets? Are you a beginner, intermediate, or advanced user?	Great to hear you're interested in Google Sheets! How much experience do you have with Google Sheets so far?

Learner: I think I'm intermediate - I can use basic formulas and pivot tables. But I feel slow when I use it

Old Model	New Model
Understood, it seems like you have a good foundation to build upon. Let's make sure we speed things up for you! Now, could you share why you want to become better at Google Sheets? Is this for a specific job role, personal development, or another reason?	Got it, Laura. It's great that you already have some skills! What motivates you to improve your Google Sheets skills? Are there specific tasks or projects you're looking to tackle more efficiently?

Learner: For my job

Old Model	New Model
Great, enhancing your skills for work is always a good investment. Can you tell me if there's a specific skill within Google Sheets or a particular type of project you're aiming to master for your job?	That makes sense. Improving your skills can definitely help with efficiency at work. Do you have any specific goals or particular topics within Google Sheets that you want to focus on? For example, advanced formulas, automation, or data visualization?

Learner: I'd like to build really interactive spreadsheets. And be able to use functions much better so I don't have to do as much manual stuff

Old Model	New Model
That sounds like a productive goal! Interactive spreadsheets and efficient function use can definitely streamline your work. As for your learning style, how do you prefer to absorb information? Through videos, hands-on practice, reading blogs, or perhaps a combination of these?	Fantastic goals to have! Building interactive spreadsheets and mastering functions will definitely save you time. How do you prefer to learn new skills? Do you enjoy watching videos, reading blogs, hands-on coding, or any other method?

Learner: Anything with examples

Old Model	New Model
One final question: is there anything else you'd like to add or any other specifics you think are important for your learning journey?	Examples are a great way to learn! One final question: Is there a particular project or type of analysis you're currently working on, or anything else to add that might help me understand your learning needs better?

Learner: I don't have a lot of time so quickly and short content is better if possible

Thoughts on the responses

The new model is more succinct - the majority of its replies are shorter, more direct and have more accessible language (eg not asking if someone is a beginner or intermediate, or using the term learning style). On this last point, I wish I hadn’t decided to use exactly the same learner responses, as I definitely wouldn’t have said I was intermediate if Dotly hadn’t worded it that way! This feels really important for our goals - getting learners to tell us as much as about their learning needs in their own words.

Interestingly the overall sentiment of all responses is very similar, but I think the new model sounds more natural.

Another plus to the shorter replies is the price - in very basic terms, there is a price per word

Part 2 - The Output - The Learner Profile

After the chatbot gets the needed information from the learner, it creates the learner profile and a search string (which is used to return the most relevant content from our database).

	Old Model	New Model
topics	Google Sheets	Google Sheets
backgroundKnowledge	intermediate	intermediate - basic formulas and pivot tables
motivation	job	improve efficiency at work
goals	interactive spreadsheets and efficient function use	build interactive spreadsheets, master functions to reduce manual tasks
preferredLearningMethod	good examples	anything with good examples
summary	The learner is an intermediate Google Sheets user, looking to improve their ability to create interactive spreadsheets and use functions more effectively for their job. They value learning through examples and are seeking quick and concise content due to time constraints.	Intermediate data analyst looking to become more efficient in Google Sheets for work by learning to build interactive spreadsheets and master functions. Prefers content with good examples and in short, quick formats.
searchString	Google Sheets advanced functions interactive spreadsheets	Google Sheets interactive spreadsheets advanced functions automation example-based learning short content
anythingElse	prefers quick and short content due to time constraints	prefers short content due to time constraints

It is worth remembering that both these models were given exactly the same information about the learner. The ability to infer some additional context seems to be a feature of the new model - for my job resulted in the motivation being “job” and “improve efficiency at work”. This also suggests the old model was only using the direct answer to questions to inform this learner profile, but the new model is using the entire conversation. Similarly for the anything else the new model has changed the learners words “quick and short”, recording only short. This is useful, but it will be interesting to see if it sometimes misses a level of detail or nuance by doing this (don’t worry I’m already thinking up some tests for this…)

Part 3 - The Search Results - Relevant Content

The search string is also different - with the new model being more explicit with the type of content.

Both search strings return all the Google Sheets content (above other content like Python, SQL, ChatGPT, machine learning). They also had the same top results - Google Sheets - Functions - which makes sense given what the learner said.

But that was it, the order of the rest is very different. Interestingly, the new model which mentions example-based and short has prioritised our shorter videos and follow along blogs (rather than our info blogs). I’m quite impressed with this, as the other search string did return some longer information blogs quite high up due to the content.

Part 4 - The Final Output - The Learning Journey

Finally, the learner profile and the relevant content (the search results) get sent to another chatbot whose job is to create the best learning journey.

The chatbot was given the same content (although in a different order), but a different learner profile, and the models made quite different learning journeys.

	Old Model	New Model
Learning Journey Name	Interactive Spreadsheets and Function Mastery	Mastering Google Sheets Functions and Interactive Spreadsheets
Learning Summary	The learner will advance their Google Sheets skills, focusing on creating interactive spreadsheets and efficient function use, with a preference for examples and quick learning content.	You will learn to build interactive spreadsheets and master advanced functions in Google Sheets, focusing on practical examples to improve your efficiency at work.
Learning Plan	Intro to Google Sheets - A quick follow-along video to refresh your understanding of Google Sheets fundamentals. Google Sheets Functions - SUMIF - Learn to selectively sum data based on specific conditions. Google Sheets Functions - INDEX - A text guide on retrieving specific data using INDEX function. Google Sheets - Functions - CONCATENATE - Learn to merge text from different cells. Google Sheets Functions - AVERAGEIF - A guide on computing conditional averages. Google Sheets - Examples - COUNTIF - Video tutorial on counting occurrences in datasets. Google Sheets - Formatting - Alternating Colours - Enhance your data presentation with formatting techniques in a follow-along video. Google Sheets - Examples - Basic Text Formulas - A video on text manipulation in spreadsheets. Google Sheets - Formatting - Tables - Video on formatting tables for better readability.	Intro to Google Sheets (Video): A quick follow-along video covering the basics of spreadsheet terminology, navigation, and cell operations. Suitable for refreshing fundamental skills. (Content ID: 20) Google Sheets Functions - SUM (Text Guide): Learn how to use the SUM function to efficiently total numbers in your spreadsheets. Ideal for mastering basic data manipulation. (Content ID: 1) Google Sheets Functions - AVERAGE (Text Guide): Understand the AVERAGE function to calculate the mean of datasets, useful for data analysis tasks. (Content ID: 7) Google Sheets - Formatting - Tables (Video): Learn to format tables for readability, covering techniques like text wrapping, resizing columns, and freezing header rows. (Content ID: 60) Google Sheets - Examples - COUNTIF (Video): This tutorial provides practical examples for using the COUNTIF function to tally occurrences within a dataset. (Content ID: 57) Google Sheets Functions - VLOOKUP (Video): Learn how to use the VLOOKUP function for vertical lookups, manipulating date formats, and handling errors. (Content ID: 55) Google Sheets - Functions - INDEX (Text Guide): Learn to retrieve specific data from a spreadsheet using the INDEX function for enhanced data manipulation. (Content ID: 45) Google Sheets - Functions - MATCH (Text Guide): Learn to locate the exact position of data in your Google Sheets using the MATCH function. (Content ID: 43) Google Sheets - Examples - Basic Text Formulas (Video): This covers basic text functions like LENGTH and TRIM, helping you clean and format text data efficiently. (Content ID: 58) Google Sheets Functions - IF (Text Guide): Master the use of the IF function to set conditions and automate responses in your spreadsheets. (Content ID: 11) Google Sheets Functions - AVERAGEIF (Text Guide): Learn to use the AVERAGEIF function to compute conditional averages of datasets. (Content ID: 14) Google Sheets - Functions - CONCATENATE (Text Guide): Learn to merge text from different cells using the CONCATENATE function. (Content ID: 44) Google Sheets - Tips - Freezing Rows and Columns (Video): A short video tutorial on keeping header information visible while scrolling through your spreadsheet. (Content ID: 59)
Skills	Creating interactive spreadsheets; proficient in Google Sheets functions	Advanced Google Sheets user
Certificate Text	Advanced Google Sheets functions, interactive spreadsheets, data manipulation, automation	Mastery of advanced Google Sheets functions and interactive spreadsheet building

One thing I’ve noticed with switching these models is the same tweaks you need to make in the instructions - in this case, I need to add something about not including the content ID.

Some other differences between the outputs:

there were more bits of content included in the new models new
the bits of content included by the old model seem to be more intermediate but not in the perfect order (I would put COUNTIF before SUMIF or AVERAGEIF)
both start with an intro (I do enjoy that Dotly does not trust people when they say intermediate)
the new models output has a more logical order, and includes some more basic functions at the beginning
the summary for the new model says you and focuses on the learner, rather than giving a general summary (something which came through in the demos in the update)

I actually prefer the skills and certification title output by the old model, this is because the new model is focusing on the learner rather than wording for a certification. This will need some tweaking in the instructions for the new model to give slightly different responses for these outputs.

Conclusions

GPT-4o felt faster, shorter spoken and more natural. I’m keen to tweak our platform a little bit to utilise this new model more.

It was very easy to start using this new model - but it is really important to test out how any new models impact the output of any chatbots or platforms.

O O O - It’s Magic, or is it? Testing out GPT-4o