Simulating Dialogues and Characters in ChatGPT-4
A Comparative Analysis of Performance in English and Portuguese
Fabiano Rodrigues de Souza (first author),
Denise da Vinha Ricieri, Raphaela V. G. Barreto, Adriana M. G. de Farias
·
CoBICET 2023 â IV Brazilian Interdisciplinary Congress on Science and Technology, Online, AugâSep 2023
·
Peer-reviewed full paper
Research Question
How does GPT-4's output quality differ between English and Portuguese under an
identical structured prompt, and what mechanism explains the difference?
Methodology
- Subject: GPT-4 (LLM
code-davinci-003) accessed via ChatGPT, April 2023, US-geolocated account.
- Prompt class: In-Context Learning (ICL) â chosen to leverage prompt-side context over pre-trained generalization alone.
- Prompt structure (Prompt-EDU framework): three components â context · purpose of intended generalization · command key.
- Task: simulated dialogue between Paulo Freire (Brazilian educator) and Mark Zuckerberg (US Meta CEO) on AI in education â selected to control for cultural-data balance, cross-domain generalization, and pre-cutoff name recognition.
- Conditions: identical prompt run in PT-BR and EN-US; cross-translation validation via GPT-4 itself; re-analysis after translation.
- Analytical dimensions: content (substance, detail) · context (in-frame fidelity) · language (vocabulary, complexity).
Key Findings
- English outputs measurably richer in detail and referential scope than Portuguese under identical prompts.
- Portuguese outputs concise and shallow â only essentials, lower vocabulary diversity.
- Context preservation robust in both languages â ICL generalized framing successfully across translation.
- Root cause traced to English-native pre-training and a hidden double-translation pipeline for non-English prompts.
- Stronger, systematic language-native prompt engineering required for non-English deployments.
- Findings mapped to Hovy & Prabhumoye 5-source NLP bias taxonomy (data, annotation, input rep., model, task design).
Implications for Practice
Any organization deploying LLMs in non-English markets needs (1) language-native, systematic prompt engineering, (2) explicit multilingual evaluation pipelines, and (3) human-in-the-loop review of model outputs.
Educators integrating AI into learning experiences must develop AI literacy, structured prompt-design competency, and an awareness of translation-induced bias before relying on LLM-generated content.
The paper concludes: ChatGPT itself acknowledges its outputs are not 100% reliable and require human review.
Technical Concepts Engaged
AI & NLP:
LLM GPT-4 Transformer Attention Mechanism Tokenization Embeddings Autoregressive Decoding NLP Multilingual NLP Translation Bias Hallucination
Prompting & Methods:
In-Context Learning (ICL) Prompt Engineering Prompt-EDU Framework Fine-tuning Cross-lingual Evaluation
Education & Ethics:
AI in Education Bloom's Taxonomy Human-in-the-Loop Responsible AI Equity in AI Instructional Design
Selected References (full list in original paper)
Vaswani et al., Attention is all you need (2017) ·
Liu et al., Pre-train, prompt, and predict (2023, ACM Comp. Surv.) ·
Wang et al., LLMs are implicitly topic models (arXiv 2301.11916) ·
Liang et al., GPT detectors are biased against non-native English writers (arXiv 2304.02819) ·
Hovy & Prabhumoye (2021) on NLP bias sources ·
Kasneci et al., ChatGPT for good? (Learning & Individual Differences, 2023) ·
OpenAI Inc., GPT-4 Technical Report (arXiv 2303.08774, 2023).