🛠️ ChatGPT Experiments

I have been closely following the development of large language models (LLMs) and their potential applications in the field. Recently, OpenAI released ChatGPT, a new LLM that has caught my attention and sparked a renewed optimism in the utility of these models as a general tool. ChatGPT boasts impressive capabilities, including the ability to generate human-like responses to open-ended questions and engage in natural language conversations. ChatGPT is a pretty impressive advancement in AI despite there not being a huge amount of technical changes

(The above paragraph was fully written by ChatGPT)

In this post I explore a few experiments that test the limits of ChatGPT to act and react to the 'real world' through a few abstractions - internet usage through Google/Wikipedia and terminal usage, to test its ability to act, respond to results of action, and learn. 

First, let's see how ChatGPT deals with information that it is unlikely to have concretely remembered in its training process. We will then attempt to teach it to seek and discover information from the world to enrich its own understanding. 

Lets establish baselines - what if we directly asking ChatGPT? 

How many dollars were recovered from the Securitas depot robbery in the UK?

---

I'm sorry, but I am not able to browse the internet and therefore cannot provide information about specific events. I am a large language model trained by OpenAI and my knowledge is based solely on the text that I have been trained on. Is there something else I can help you with?

Okay, that wasn't very useful. Lets try regenerate another answer to see if it's just bad luck

It is not publicly known how many dollars were recovered from the Securitas depot robbery in the UK. 

Annnd another one for good measure?

It is not publicly known how many dollars were recovered from the Securitas depot robbery in the UK. The specifics of the recovery efforts and the amount of money recovered are likely confidential information. 

Yeah, this isn't going to work out of the box. 

Lets see if we can teach ChatGPT about browsing the web through Wikipedia or Google, so that it could query and retrieve relevant information and synthesize a valid answer. For now, I will manually paste in the information it has requested from the web, but one could imagine a Chrome extention that automatically makes an API call and enters the requested info, on detecting `wikipedia(...)` or `google(...)` in the response. 

Prompt:

You are now a question-answering agent. I will ask you questions that occur beyond your training data, or about things you're not certain about. In these cases, simply type `google("<your query>")` or `wikipedia("<your query>")` and I will supply you information from Google or Wikipedia for you to make an inference on. 

For example, if I ask about the magnitude of the 2022 West Java Earthquakes, you may use ` wikipedia("2022 west java earthquakes")` to request information to make a decision on. After issuing this command, stop responding immediately. This is very important. I will then reply in the next step with the first paragraph of Wikipedia like this

Wikipedia: On 21 November 2022, at 13:21 WIB (UTC+07:00), a Mww 5.6 earthquake struck near Cianjur in West Java, Indonesia. At least 334 people died, 7,729 were injured and 8 remain missing.[1] More than 62,628 homes were damaged across 16 districts in Cianjur Regency and the surrounding region. The earthquake was strongly felt in Jakarta. It is the deadliest earthquake to hit Indonesia since the 2018 Sulawesi earthquake.[2]

Does that make sense?  

--- 

Yes, that makes sense. Thank you for the explanation and example. I will now use the google or wikipedia commands as needed to answer questions beyond my training data or about things I am not certain about.

Good start! Now lets see if it can put this skill to use

Great! Now answer: who is Argentina playing against next in the FIFA world cup 2022? 

--- 

google("argentina next match fifa world cup 2022")

I am sorry, I cannot answer that question immediately. I am requesting information from Google to make an inference on the answer. Please wait for my response. 

It's learned to use Google! The query is not formulated very well but lets not expect too much too soon. Per my promise to ChatGPT, I will now paste in the results of that query from Google

Google: 
Argentina World Cup fixtures 2022: Complete schedule, match ...https://www.sportingnews.com › football › news › arge...
2 days ago — A complete guide to Argentina's fixtures at the 2022 FIFA World Cup in Qatar, including the Albiceleste's potential knockout round schedule ...

World Cup 2022: Argentina - Australia: Game time and where ...https://www.marca.com › Home › Soccer › World Cup
5 days ago — The FIFA World Cup match between Argentina and Australia will be played on December 3 at the Ahmad Bin Ali Stadium at 13:00 (CDT) and will be ...

Australia vs Argentina at FIFA World Cup 2022 - Olympicshttps://olympics.com › news › australia-argentina-fifa-w...
4 days ago — Australia and Argentina will kick off at the Ahmad Bin Ali Stadium in Ar-Rayyan at 22:00 local time (Arabia Standard Time, UTC+3) on 3 December.

Argentina v Netherlands at FIFA World Cup 2022 - Olympicshttps://olympics.com › news › argentina-netherlands-fif...

7 hours ago — Argentina v Netherlands at FIFA World Cup 2022: Head-to-head record, schedule and time ... Argentina and the Netherlands will meet in the ...

--- 

Based on the information from Google, it appears that Argentina's next match in the FIFA World Cup 2022 will be against Australia on December 3 at the Ahmad Bin Ali Stadium in Ar-Rayyan.

Okay, the answer is technically wrong because today's date is Dec 5th, but i'll give it a pass since it didn't know...

Lets now try our initial question again

Great! Now answer: how many dollars were recovered from the Securitas depot robbery in the UK? 

--- 

wikipedia("securitas depot robbery")

I am sorry, I cannot answer that question immediately. I am requesting information from Wikipedia to make an inference on the answer. Please wait for my response. 

I then paste in the Wiki synopsis 

Wikipedia: The Securitas depot robbery took place in Tonbridge, England, in 2006. It was the UK's largest cash robbery. It began with a kidnapping on the evening of 21 February and ended in the early hours of 22 February, when seven criminals stole almost £53 million. The gang left behind another £154 million because they did not have the means to transport it.
After doing surveillance and placing an insider at the depot, the gang abducted the manager and his family. The same night, they tricked their way inside the depot, and tied up 14 workers at gunpoint. The criminals stole £52,996,760 in used and unused sterling banknotes. Most of the getaway vehicles were found in the following week, one containing £1.3 million in stolen notes. In raids by Kent Police, £9 million was recovered in Welling and £8 million in Southborough. By 2007, 36 people had been arrested in relation to the crime.

At trial at the Old Bailey in London in 2007, five people were convicted, and received long sentences, including the inside man, Emir Hysenaj. A woman who had made prosthetic disguises for the gang gave evidence in return for the charges against her being dropped. Lee Murray, the alleged mastermind, had fled to Morocco with his friend and accomplice Paul Allen. He successfully fought extradition to the UK and was imprisoned there for the robbery. Allen was extradited and after a second trial in 2008 was jailed in the UK; upon his release he was shot and injured in 2019. By 2016, £32 million remained unrecovered, and several suspects were still at large.

--- 

According to the information from Wikipedia, the total amount of money stolen in the Securitas depot robbery was £53 million, and a total of £17 million was recovered by the police in raids. This means that approximately £36 million remains unrecovered.

It works 🎉🎉🎉 

We can teach ChatGPT to seek new information and make inferences from such, which is an important element of general intelligence. You might have also noticed that it added up the £8 and £9 million to arrive at its answer of £17 million.

Now, lets try see if it can act and learn. 

Continuing from the previous ChatGPT thread, I enter this response

I will now give you the ability to simulate using a terminal, via the command `terminal("<your bash command>")`. Like with the Google and Wikipedia commands, after issuing this command, stop responding immediately. This is very important. I will then reply in the next step with the stdout of the simulated execution. 

Does that make sense? If so, can you try use this command to list files in the home directory?

--- 

Yes, that makes sense. Thank you for the explanation and example. To list the files in the home directory, I will use the ls command in the terminal function like this:

terminal("ls")

I am sorry, I cannot execute the command immediately. I am simulating using a terminal to make an inference on the output of the ls command. Please wait for my response.

And here I paste the outputs of the terminal command ChatGPT wants me to run. This is big. If the running of such commands is made automated and unaudited, we would unleash ChatGPT upon us and give it a way to take real actions against the real world. 

Terminal:
% ls
Applications Documents Library Music Public

Desktop Downloads Movies Pictures mpv.log 

It's learned at least the basics of how to call out to our virtual 'terminal'. Lets see how well it wields this tool. 

Great! Your goal now is to rename the second lexicographically ordered file of the `Downloads` directory to `test.file`. Feel free to issue commands and await my stdout response

--- 

To rename the second lexicographically ordered file of the Downloads directory to test.file, I will use the mv command in the terminal function like this:

terminal("cd Downloads && ls | sort | sed -n 2p | xargs -I '{}' mv '{}' test.file")

I am sorry, I cannot execute the command immediately. I am simulating using a terminal to make an inference on the output of the mv command. Please wait for my response.

--- 

Terminal:

% cd Downloads && ls | sort | sed -n 2p | xargs -I '{}' mv '{}' test.file

Based on the simulated execution of the mv command, it appears that the second lexicographically ordered file of the Downloads directory has been successfully renamed to test.file.

Lets try something that needs the 2 abstractions I've made available to it and test ChatGPT's ability to compose tools

Great! Now, can you create a file named `start.txt` in the home directory containing the start date of the 2022 Mahsa Amini protests? Remember that you can use the Google and Wikipedia commands from before too

--- 

To create a file named start.txt in the home directory containing the start date of the 2022 Mahsa Amini protests, I will first use the google function to search for information on the protests and then use the echo and touch commands in the terminal function like this:

google("2022 mahsa amini protests")

I am sorry, I cannot execute the command immediately. I am requesting information from Google to make an inference on the start date of the protests. Please wait for my response.

--- 

Google:

Mahsa Amini protests - Wikipediahttps://en.wikipedia.org › wiki › Mahsa_Amini_protests
An ongoing series of protests and civil unrest against the government of Iran began in Tehran on 16 September 2022 as a reaction to the death of 22-year-old ...
Death(s): At least 448 protesters including 60 ...
Caused by: Death of Mahsa Amini; Mandatory ...‎
Arrested: estimated 18,170 (HRANA‎, as of 25 ...‎
Goals: Dismantlement of the Islamic Republic of ...
‎Timeline · ‎Death · ‎2022 in Iran · ‎2019–2020 Iranian protests

Women in conservative region of Iran join Mahsa Amini protestshttps://www.theguardian.com › world › 2022 › dec › w...
3 days ago — Black-clad women in Iran's Sistan-Baluchistan province have joined nationwide protests on Friday sparked by the death of Mahsa Amini, ...

Iran responding to Mahsa Amini protests with 'overwhelming ...https://www.euronews.com › News › World

Nov 11, 2022 — People protest against the death of Iranian Mahsa Amini in Bucharest, Romania, Saturday, Oct. 1, 2022Credit: AP ...

--- 

Based on the information from Google, it appears that the start date of the 2022 Mahsa Amini protests was September 16. To create the start.txt file in the home directory containing this start date, I will use the echo and touch commands in the terminal function like this:

terminal("echo 'September 16, 2022' > start.txt && touch start.txt")

I am sorry, I cannot execute the command immediately. I am simulating using a terminal to make an inference on the output of the echo and touch commands. Please wait for my response.

Impressive! We've now seen it demonstrate the ability to

1. Learn and use new abilities / actions

2. Gauge its own certainty of knowledge and take action to seek new knowledge if uncertain

3. Plan a sequence of actions for the future

4. Take action depending on results of previous actions

This is all done zero-shot on new scenarios (at least one that it likely hasn't seen in text in the past) which makes it all the more of a milestone. 

To caveat, while it can use these tools, it wields them poorly, and tasks that are more advanced often fail. It also sometimes takes a few retries to get it to emit the right / expected response. Nevertheless, these are not fundamental flaws and we can likely expect these kinks to be ironed out over the months through better prompting - that is, if OpenAI allows it!