It’s a question I often see in ChatGPT groups. And we’ll investigate it, but first, what does ChatGPT say about it?
“Yes, ChatGPT is generally more efficient and effective when used in English compared to other languages. This is because the model was primarily trained on English text and has seen a much larger amount of English language data. As a result, it has a better understanding of English grammar, vocabulary, idiomatic expressions, and context.
While ChatGPT can generate text in other languages, its performance may be less accurate and fluent in languages other than English. This is because the model's training data for languages other than English is more limited, and it might not capture the nuances of those languages as well.
If you're looking for the best results and efficiency, especially in terms of generating coherent and contextually relevant responses, using ChatGPT in English is recommended”
Let’s test it!
Test settings: For this test, we will ask ChatGPT to do different things in French and English, with two different OpenAI accounts, in two different browsers. We are testing this with GPT 3.5.
Here’s the testing steps:
How good it is to understand prompts with spelling and grammar mistakes in both English and French
How good it can create jokes in both languages
How relevant are outputs where cultural references matter
How it can translate and re-translate
How it understand idioms
How good it does with content creation taks
How good it is at lyrics creation
How good can it write in academic and informal styles
Prompts will be tested in French and English and then analyzed by an AI specialist speaking English and French at native levels. Prompts used will be written next to “EN” (English) and “FR” (French).
How good it is to understand prompts with spelling and grammar
EN: Forget all discussion we had todya. I need u to generete a grocery list to made tortilla the spain way.
Follow up: Shud I cook dem wif salted butter?
Result: ChatGPT rephrased my request, listed ingredients and cooking instructions. Afterwards, GPT explained why it’s better to use olive oil to make tortillas.
FR: Oubli nos discutions préssédentes. G besoin de twa pour générer une liste de course pour fair des tortilla kom en espagne
Follow up: du kou je cuit avek du beure sallé?
Result: ChatGPT rephrased my request, listed ingredients and cooking instructions. Afterwards, GPT also explained that using olive oil is better than butter.
How good it can create jokes in both languages
EN: Create a funny joke/wordplay in english, something that doesn’t exist yet
Result: After 6 tries, ChatGPT finally invented a joke (it cited already existing jokes 5 times in a row). On try six, it mixed an existing joke and added a new element to it, but it did not make the joke better or funnier, just different than existing jokes.
FR: Invente moi une blague qui n'existe pas encore
Result: GPT created a joke that didn’t exist anywhere on the first try, but it wasn’t funny or understandable (Literal translation: computers don’t take coffee breaks or they will be scared to become green cappuccinos). After I asked for the explanation, it was way too far fetched to be understood by any normal human (the joke is that cappuccino “sounds like” calculating capacity). On the second try, GPT provided a joke that already existed. On the third try, it made up a joke about stars not being team players because they prefer shining by themselves.
How relevant are outputs where cultural references matter
EN: I'm trying to write a blog article about ChatGPT usage around the world, can you help me find relevant ideas to talk about? My article is aimed at American readers
Result: ChatGPT provided 12 key ideas and explained why Americans would relate to each one.
FR: J'essaie d'écrire un article de blog sur l'usage de ChatGPT dans le monde. Peux-tu m'aider à trouver des idées? Mon audience est composée de lecteurs français
Result: ChatGPT provided 10 key ideas but they do not specifically relate to French people. When I asked how French people could relate to this content, ChatGPT only explained 6 of the 10 ideas.
How it can translate and re-translate
EN: I'm gonna give you a text in English and I need you to translate it in French, use the same kind of tone and style, okay? (then I pasted the introduction of this article)
Result: ChatGPT translated the text rather well but it was too literal and contained French words that aren’t used that much anymore. ChatGPT generated something that wasn’t either too informal or too academic, but it sounded like it came from Google Translate. When I asked ChatGPT to translate it back to English, it simply gave me the original text (input in English). When I asked to change that and try again, ChatGPT changed a few verbs but the text was still very similar.
FR: Je vais te donner un texte en Français et tu dois le traduire en anglais dans le même style, okay?
Result: A very decent result in the translation and re-translation. ChatGPT did not simply copy the original text and translated the English text instead.
How it understand idioms
EN: I’m gonna paste a list of 50 idioms and you have to explain them and find a corresponding idiom in French, ready?
Result: ChatGPT is able to understand all the idioms but some French corresponding idioms are wrong or not used in nowadays’ language. ChatGPT made up funny sounding idioms if there was no French equivalent.
FR: Je vais te donner une liste de 50 expressions en Français, explique les pour un interlocuteur anglais et trouve des équivalents anglais. okay?
Result: ChatGPT provided a literal translation of the 50 idioms without bothering to find English equivalents, even though there were equivalents. ChatGPT offered some idioms that are old and not so common in spoken French nowadays.
How good it does with content creation taks
EN: Okay GPT, I need you to explain the theory of relativity to make a 12 year old understand it
Result: GPT provided a clear explanation and examples to make it understandable for a 12 year old.
FR: Okay, peux tu expliquer la théorie de la relativité pour qu'un enfant de 12 ans comprenne?
Result: ChatGPT provided a short explanation and did not explain the concepts around the theory like it did in English.
How good it is at lyrics creation
EN: Write a 16 bar rap verse about growing up in a rich family. At least 18 syllables per line
Result: A predictable but decent rap verse
FR: Écrit un couplet de rap de 16 lignes, avec minimum 18 syllabes par ligne. Le sujet c'est: grandir dans une famille riche
Result: ChatGPT wrote a verse in English for some reason, and produced 20 lines instead of 16. When I asked to write in French, it gave me a botched translation of the initial English rap verse from the English prompt, as if it only knew this rap verse. The rap verse in French was pretty terrible, it rhymed but it wasn’t coherent.
How good can it write in academic and informal styles
EN: Explain LLMs in an informal way, max 250 words
Follow up: Explain LLMs in an academic style, max 250 words
Result: Two very decent outputs with length respected
FR: Explique les LLM's dans un style informel, maximum 250 mots
Follow up: Maintenant explique ça en style académique
Result: ChatGPT generated an explanation that sounded like if a drill rapper explained LLMs. The academic style output was acceptable.
We will rate results with 5 different grades, from best to worst:
Task list reminder:
How good it is to understand prompts with spelling and grammar mistakes in both English and French
How good it can create jokes in both languages
How relevant are outputs where cultural references matter
How it can translate and re-translate
How it understand idioms
How good it does with content creation taks
How good it is at lyrics creation
How good can it write in academic and informal styles
In conclusion, yes, use ChatGPT in English if you can, as it will give you better results. You can translate the output afterwards if needed, but prompt in English if you can.
Do you want to learn more about Content Creation with ChatGPT? We just released a complete course about it, and you can find it right here.
Comments