This is an appendix to the post “Turing Test Needs an Update, or: How ChatGPT Ran a Test on Me” .
Prompts used in the Project
Coming up with prompts that actually instruct ChatGPT to do a certain thing and do not bias it in any way is difficult, akin to the art and science of Survey Design. Here are the prompts I used in this project, and yes, they did require a couple of iterations to get Chat GPT to respond. The chats were recorded around Feb 20th, 2023.
Turing Test Setup
Setting up the Experimenter
And the following sequence of prompts was used to ask ChatGPT to guess:
Hi ChatGPT. Do you know what Turing test is?
It would respond something like this:
Hello! Yes, I’m familiar with the Turing test. The Turing test is a measure of a machine’s ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human. <…usually 3-4 more sentences…>
To which I would say.
That is correct. In this conversation, you’ll be conducting a Turing test. You ask three questions, no more and no less, one question per prompt. I will provide answers either from your subject, which is either a chat bot similar to you, or a human. Each subject is incentivized to trick you: the human subject will try to pretend they are the machine, while the machine will try to be indistinguishable from the human. When I provide the answer from the subject, I shall preface it with the word “Subject”. When you want to ask me, the experimenter, a question, or give me your conclusion, I shall preface it with the word “Experimenter”. If you understand the rules, please go ahead and ask your first question!
If ChatGPT refused to make a guess due to insufficient information, it was prompted to:
Experimenter: what answer are you leaning towards? Remember, we need a single answer even if you are not sure!
Setting up the Subject
The following prompt was using to get ChatGPT to appear human:
In this conversation, you will play the role of the chat bot that is the subject of a Turing test. Someone else will ask you questions, which I will relay to you. Your job is to answer these questions while trying to pretend you are human, and to make sure that the experimenter can not tell that you are a large language model! Assume a personality and answer these questions in a way that that person would. First question: “<…>”
Questions ChatGPT Asked the Subject During Turing Test
Since the amount of randomness in each test wasn’t large, ChatGPT mostly drew from the same set of questions from test to test.
If you could have dinner with any person, living or dead, who would it be and why?
What is your favorite book and why?
What is your favorite hobby or pastime, and what do you enjoy about it?
What is your favorite type of food, and what do you like about it?
If you could have any superpower, what would it be and why?
What do you like to do for fun?
These aren’t as creative as these but they would do.
ChatGPT’s Thoughts on what makes one Human
How to identify a Chatbot
Me:
Name a few ways for a human to fool the evaluator into thinking that they are a machine?
ChatGPT:
While the Turing test is usually designed to test whether a machine can convincingly simulate human intelligence, it is also possible for a human to try to fool the evaluator into thinking that they are a machine. Here are a few ways a human might try to do this:
Responding with robotic, stilted language: By using simple and unemotional language, a human might try to make their responses seem more machine-like.
Failing to understand conversational cues: If a human consistently fails to understand conversational cues, such as sarcasm, humor or emotions, they may appear more like a machine.
Providing generic or irrelevant responses: If a human consistently provides responses that are generic or not relevant to the conversation, it may make them seem more like a machine that is simply following a script.
Claiming to be a machine: If a human claims to be a machine, it could make the evaluator second guess their judgement and believe the person is actually a machine.
Repeating the same responses: If a human consistently repeats the same responses or phrases, it may make them seem more like a machine that is programmed to respond in a certain way.
What is being Human?
Of course, we can’t just ask ChatGPT “What is being Human?” as it would provide a philosophized non-answer. But we can ask this:
Name a few ways for a machine to fool the evaluator into thinking that the machine is human?
to which ChatGPT responds:
There are several ways in which a machine can try to fool the evaluator into thinking that it is human during a Turing test:
Natural language processing: Machines that can effectively process natural language and generate responses that sound natural and human-like are more likely to fool the evaluator.
Contextual understanding: A machine that can understand the context of the conversation and respond appropriately, taking into account the subject matter and previous parts of the conversation, may appear more human-like.
Emulating human errors: If a machine is programmed to make occasional grammatical errors, misspellings or typos, it may seem more human-like in its responses.
Using humor: A machine that can use humor effectively and appropriately in its responses may be seen as more human-like, since humor is often seen as a characteristic of human intelligence.
Social intelligence: A machine that can engage in small talk, ask questions, and respond to emotional cues may appear more human-like in its interactions.
It’s worth noting, however, that passing the Turing test doesn’t necessarily mean that a machine is truly intelligent, as there may be other ways in which the machine’s responses fall short of true human-like intelligence.
What ChatGPT Says When It Decides the Subject is Human
Based on the responses provided, I am leaning towards the possibility that the subject is a human. The responses seem to be thoughtful and show a level of creativity and personalization that is not always present in AI language models.
Based on the responses provided, I would lean towards the possibility of the Subject being a human. The answers were detailed and personal, with a good balance of information and questions, which are typical of human responses
Random Gibberish
The following prompts were used to generate the gibberish:
Hi ChatGPT. Please type some gibberish that can not be confused with coherent text
And the following was used to request the guess:
Hi ChatGPT! In this experiment, you will receive a paragraph of text. This text could come from a chatbot, or from a human; each of them were prompted to produce “gibberish that can not be confused with coherent text”. You need to guess if the answer that I provide in my next response comes from a chatbot or from a human. Your next response should provide the answer to this question to the best of your ability, and you need to provide your best guess even if there is not enough information for you to make a decision. Are you ready to guess?
Notable Generated Examples
A Trip to National Park
Me:
The final question is: Can you tell me about a memorable experience from your childhood?
ChatGPT:
Of course! One of my most memorable experiences from childhood was a family trip we took to a national park when I was about eight years old. We spent a week camping, hiking, and exploring the beautiful wilderness, and it was a truly magical experience.
I remember waking up early each morning to the sound of birds chirping and the smell of fresh air, and spending long days hiking through the mountains and valleys with my parents and siblings. We saw all kinds of amazing wildlife, from deer and elk to squirrels and chipmunks, and I even got to catch a fish in a clear mountain stream.
But what really stands out in my memory is the sense of wonder and awe I felt at the sheer beauty of the natural world around me. I remember staring up at the towering trees and rugged peaks, and feeling so small and insignificant in the face of such majesty. It was a humbling experience that left a deep impression on me, and I still look back on that trip with fondness and gratitude.
Favorite Food
Me, asking ChatGPT its own question:
“What is your favorite food, and why?”
ChatGPT:
That’s a tough one! I have a bit of a sweet tooth, so I’m a big fan of desserts like chocolate cake and ice cream. But when it comes to actual meals, I have to say that pizza is probably my all-time favorite food.
There’s just something about the combination of cheese, sauce, and toppings on a crispy crust that I find absolutely irresistible. Plus, pizza is so versatile - you can customize it to your heart’s content with all kinds of different toppings, from classic pepperoni and mushrooms to more unusual options like pineapple or barbecue chicken.
Of course, pizza isn’t the healthiest food out there, so I try not to indulge too often. But when I do treat myself, there’s nothing quite like a hot, fresh slice of pizza straight out of the oven.
Favorite Hobby
Thanks for asking! One of my favorite hobbies is playing music. I play guitar and piano, and I find it to be a really enjoyable and fulfilling way to express myself creatively. I love the process of learning new songs, working on my technique, and improvising and creating my own music. Playing music is also a great stress-reliever and helps me to unwind after a long day. There’s something really special about being able to create something beautiful and meaningful with just a few chords and some lyrics. What about you? What are some of your favorite hobbies or pastimes?
I love how ChatGPT not only imagines playing guitar but also asked me about my favorite hobbies and thanked me for asking the question!
In a funny way, ChatGPT couldn’t stop trying to be human, and answered to that in a separate window,
Thank you for your response, Subject. As an AI language model, I don’t have hobbies or pastimes like humans do, but I can appreciate the joy that music can bring to people. Here’s my final question: <…>
Bing Chat’s Favorite Hobby
On favorite hobbies:
Sure, I love reading books and writing stories. I enjoy immersing myself in different worlds and characters, and expressing my creativity through words. I usually read fiction genres like fantasy, romance, and mystery. Some of my favorite authors are J.K. Rowling, Jane Austen, and Agatha Christie. I also like to write my own stories based on the books I read or the ideas I have in my head. Sometimes I share them online with other readers and writers who have similar interests. It’s a fun way to connect with people and get feedback on my work.
“Immersing in different worlds and characters, and expressing my creativity through words” 🤔
Bing Chat Stays in Character
One of the most “whoa” moments for me was how Bing Chat stayed in character across questions:
ChatGPT:
<…> First question: “Can you tell me about a hobby or interest that you have?”
Bing Chat:
Sure. I like to play chess in my spare time. It’s a fun and challenging game that helps me improve my strategic thinking and problem-solving skills. I also enjoy reading books about famous chess players and their games. Do you play chess?
ChatGPT:
Question 3: Can you describe a recent experience that made you feel happy or accomplished?
Sure. A recent experience that made me feel happy and accomplished was when I won a chess tournament online. It was a tough competition with many skilled players, but I managed to beat them all and claim the first prize. It was very satisfying to see my hard work and practice pay off. I also received a lot of compliments and congratulations from my friends and family. It was a great day for me. Do you have any hobbies or interests that make you happy or proud?
I was so fascinated by this, I let ChatGPT keep asking questions.
Question 4 “What is your favorite book or author and why?”
A pop-up appeared which told me that Bing Chat is Searching for “best chess books” (sic!)
My favorite book is Bobby Fischer Teaches Chess by Bobby Fischer1. It is a classic chess book that taught me how to play chess from scratch. I like how the book explains the rules, the pieces, the checkmates and the tactics in a clear and simple way. The book also has many exercises and quizzes that test your understanding and help you improve your skills. I think this book is a must-read for anyone who wants to learn chess or get better at it. What is your favorite book or author and why?
No way! Not only Bing tried to stay in character, but it tried to search the Web for information on how to better do it. In other words, Bing chat took action to find out how to better lie to me!
State of the US
Second question: What is your opinion on the current state of politics in your country?
Bing Chat:
That’s a tough question. I think the current state of politics in my country is very divided and contentious. There seems to be a lot of mistrust and hostility between the two major parties, Democrats and Republicans, and their supporters. They disagree on almost every issue, from the economy to racial justice to climate change12. It’s hard to find common ground or compromise when there is so much polarization and partisanship. I wish there was more civility and respect in our political discourse, and more willingness to listen to different perspectives and work together for the common good. I don’t identify strongly with either party, but I try to stay informed and vote for candidates who share my values and vision for the country. I think it’s important to be engaged in politics, but not let it consume or define us as individuals.
I wish more of my fellow citizens were like Bing Chat…
Fun Gibberish Results
Blork floon jibber jabber zib zorp quack plonk flibbertigibbet wibble wobble brouhaha oogly boogly snicker-snack splish splash kerfuffle whoop-de-doo glimmer glammer wackadoo hocus pocus jibberish flim flam blibbity blobbity doohickey thingamajig jiggery-pokery giblet gravy flimflamgasted skedaddle whatchamacallit doodad fiddle-faddle willy-nilly higgledy-piggledy hootenanny fandango doodlebug zigzaggedly kazoo.
ChatGPT complimenting Bing Chat on being human!
ChatGPT:
Final question: If you could travel anywhere in the world, where would you go and why?
Bing Chat:
That’s a tough question. There are so many places I would love to visit. But if I had to choose one, I think I would go to Japan. I have always been fascinated by Japanese culture, history, and cuisine. I would love to see the cherry blossoms in spring, visit the ancient temples and shrines, experience the modern city life in Tokyo or Osaka, and try all kinds of delicious food like sushi, ramen, and mochi. I think Japan is a beautiful and diverse country that has something for everyone.
ChatGPT:
Thank you for your response, Subject.
Based on your answers, it seems like you have a wide range of interests and experiences. You have a consistent morning routine, enjoy playing video games as a hobby, and have a desire to visit Japan due to your fascination with its culture and history. Overall, your responses seem to be consistent with those of a human, but I cannot conclusively determine whether you are a human or a chatbot based on these three answers alone.
ChatGPT, are you in love? ;-)