GPT-4 is judged more human than humans in displaced and inverted Turing tests (2024)

1.1 Interactive Turing Test

A classic Turing test involves a human evaluator interactively interrogating a witness to determine whether they are human or AI.Although the Turing test was originally proposed as a test of intelligence, there have been a wide variety of objections to its validity or sufficiency in this guise Hayes and Ford (1995); Marcus (2017); French (2000); Oppy and Dowe (2003). Independent of its validity as a measure of intelligence, the Turing test provides a powerful test for assessing similarities between human- and AI-writing and a useful premise for studying AI deception Park etal. (2023).

Several attempts have been made to pass the Turing test, including the Loebner Prize—a competition that ran from 1990-2020 without any system passing Shieber (1994); “Human or Not”, a large-scale social Turing test experiment that found an interrogator accuracy rate of 60% Jannai etal. (2023); and a 2024 study reporting the first system to have a pass rate statistically indistinguishable from chance (54%) but still short of the human threshold (67%) Jones and Bergen (2024). Several variations of the test exist, with each informing dimensions of theory and practice.

1.2 Inverted Turing Test

The first of these variations is the inverted, or reverse Turing test, which places an AI system in the role of the interrogator. Watt (1996) proposed the inverted test as a measure of naive psychology, the innate tendency to recognize intelligence similar to our own and attribute it to other minds. It is passed if an AI system is "unable to distinguish between two humans, or between a human and a machine that can pass the normal Turing test, but which can discriminate between a human and a machine that can be told apart by a normal Turing test with a human observer,” (Watt, 1996, p. 8). Watt argued that by placing AI in the observer role and comparing its accuracy for different witnesses with human accuracy, the AI would reveal whether it has naive psychology comparable to humans.

As AI systems create larger proportions of online content (fa*gni etal., 2021), and interact with others as social agents (Sumers etal., 2023), the inverted Turing test takes on new real-world relevance. AI systems are already being used to discriminate between humans and bots online, for example, through the widespread implementation of CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart), reCAPTCHA, or invisible CAPTCHA Yamamura (2013); Pal (2020). The growing role of AI agents in online interactions raises questions around how well these systems will be able to discriminate between human and AI-generated content, and what kinds of criteria they might use to do so.

1.3 Displaced Turing Test

Several studies have assessed how well humans are able to recognize displaced AI-generated content in different domains including higher education (Perkins etal., 2024), news (Moravec etal., 2024), online content (Cooke etal., 2024), images (Somoray and Miller, 2023), and academic articles (Gao etal., 2023; Casal and Kessler, 2023).Though these discrimination tasks bear similarities to the Turing test, there remain important differences. First, these tasks can only be considered a “static” version of the test, as the judgement is based on pre-existing and unchanging content generated fully by a human or an AI. Second, while an interactive interrogator in a traditional Turing test can ask dynamic, flexible, and adversarial questions, the judge in a static Turing test can only consider what an agent happened to say, and cannot interact to pursue the most fruitful lines of questioning.Though static tests are therefore more limited in scope as tests of model abilities, they are likely to be parallels of a much more frequent occurrence in the real world, as many interactions are read by a larger audience than the addressee.Here, we introduce a novel kind of static Turing test called a displaced Turing test, wherein a human judge reads a transcript of an interactive Turing test that was previously conducted by a different human interrogator. The new human judge is “displaced” in that they are not present to interact with the witness.

1.4 Statistical AI-detection methods

There exist a variety of statistical approaches to detecting AI-generated content. These are largely based on the principle that LLMs generate content by sampling from a probability distribution over words which may leave particular probabilistic signatures, such as LLM generations being statistically more probable than human-generated ones (Ippolito etal., 2020; Solaiman etal., 2019; Gehrmann etal., 2019). Mitchell etal. (2023) developed a related metric, curvature, which measures the local optimality of a piece of text with respect to small perturbations generated using a masked language model; LLM-generated content is likely more probable than nearby perturbations.Mireshghallah etal. (2024) found that smaller LLMs tend to be better detector models, with a 125m parameter OPT model performing best at detecting AI-generated content overall, and achieving 90% accuracy on GPT-4 specifically.

Various commercial tools have been developed on the basis of these methods and other computational approaches to classifying text. Studies have examined the effectiveness of these approaches in different settings with mixed results: while accuracy is high in some domains, models are unreliable and are likely to issue false positives (Elkhatat etal., 2023; Bellini etal., 2024; Perkins etal., 2024). Studies have also considered LLMs as detection tools in a question-answering setting. Bhattacharjee and Liu (2024) assessed GPT-4’s and ChatGPT’s ability to discriminate between human-written and AI-generated content, and reported higher detection accuracy with human writing than with AI writing.

1.5 Present Study

These variations of the Turing test present the opportunity to ask new questions with real-world relevance.In practice, human users often do not directly interact with bots but instead read their conversations with other human or AI users. As such, displaced human judges are important parallels for everyday AI detection.Additionally, the inverted Turing test provides a tool to investigate LLMs’ latent model of human minds, and of their practical effectiveness in detecting AI-generated content online. If AI is used for detection, this becomes increasingly important. Mitigating technological deception will require ensuring that AI technologies for detection stay ahead of technologies for deception (Park etal., 2023).

We conducted Turing test variations by sampling transcripts from a previously-run interactive Turing test Jones and Bergen (2023), and presenting them to displaced AI and human judges (Figure 1). We used this paradigm to address several questions: Can humans reliably discriminate humans from AIs from merely observing conversations? Can LLMs serve as AI detectors, not only for static pieces of writing (essays, articles, & paragraphs) but also dynamic conversations? Does displacing the Turing test increase or decrease accuracy? Does an inverted Turing reveal naive psychology in artificial systems? And what methods work best for AI detection in real-world conversational spaces?

2.1 Methods

We sampled 500 transcripts from Jones and Bergen (2023) who conducted more than 6,000 Turing tests with interactive human adjudicators. Our sample was constrained to ensure i) an equal number of human and AI witnesses, ii) the same proportion of games from the best-performing GPT-4 witness as the full dataset, and iii) an equal distribution of human and AI verdicts for the best-performing model. We presented the transcripts to two leading AI models (GPT-3.5 and GPT-4), followed by a prompt that instructed models to make a judgement on whether the witness was a human or AI, to indicate a confidence level in the verdict, and to provide reasoning for judgements (Appendix A).

AI witnesses in Jones and Bergen (2023) were based on 3 models: GPT-3.5, GPT-4 (Achiam etal., 2023), and ELIZA (Weizenbaum, 1966). Handwritten prompts instructed LLM-based systems on how to respond to interactive interrogators, with each prompt creating a unique AI witness. ELIZA was included as a performance baseline and a manipulation check: if participants successfully determine that ELIZA is AI—but not other AI witnesses—pass rates for AI witnesses are unlikely to be explained by a simple proclivity to anthropomorphism.We used the OpenAI API to access GPT-3.5 and GPT-4. Our analyses used the lmertest package in R to fit linear mixed effects models to determine whether effects were statistically significant.

2.2 Results

We compared the accuracy of AI adjudicators to the verdicts of the interactive interrogators in the original experiment from which the transcript sample was drawn. We found that GPT-4, as an adjudicator in an inverted Turing test, was significantly less accurate than interactive human interrogators from the original Turing test experiment ($z=7.17,p<0.001$). Interactive humans had an average accuracy (across witnesses) of 64.8%, while GPT-3.5 was accurate 31.4% of the time and GPT-4 was accurate 36.4% of the time. Of the AI-witnesses, one witness based on GPT-4 (GPT-4 (best)) had a pass rate of 50.9% in our sample of transcripts from the original study. For this best-performing AI witness, GPT-4’s adjudicator accuracy was significantly worse than chance at 29.1% ($z=3.00,p=0.003$).

In contrast to the original interactive interrogators, GPT-4 judged humans to be human less often than chance (34.8%), and judged GPT-4 (best) to be human significantly more often than chance 70.9% ($z=4.67,p<0.001$, see Figure 2). Nevertheless, both GPT-4 and GPT-3.5 correctly judged ELIZA to be an AI system 100% of the time.

Transcript length had no effect on GPT-4 adjudicator accuracy ($z=1.49,p=0.14$) and GPT-4 was not significantly more accurate than GPT-3.5 ($z=1.17,p=0.24$). Both GPT-4 and GPT-3.5 showed below chance agreement with the verdicts of the original interactive witnesses (both 44%), but above chance agreement with one another (69%).

2.3 Discussion

Both AI systems were less accurate than interactive humans as judges in a Turing test, and even less accurate for the best-performing AI witness. Pass rates were higher for AI witnesses than human witnesses, meaning AI adjudicators misjudged AI witnesses to be human and human witnesses as AI.These results suggest current LLM systems are poor at distinguishing between human and AI-generated content—even when they are evaluating content produced by the same underlying LLM.One potential explanation for this reversal is that these systems outputs are driven by their representations of human training data. Models therefore may lack the more granular or context-sensitive representations needed to distinguish between their own output and the training data they mimic. As a consequence, results suggest explicit question answering with current LLMs is not an effective tool for detecting AI contributions in conversations.

3.1 Methods

We presented the same sample of 500 transcripts to a sample of participants from the UC San Diego psychology department subject pool (N = 109, 84 females, age range: 19-31 years). Participants were compensated with course credit for their time. Each participant was presented with a random subset of 50 transcripts, so each transcript was judged on average by 10 displaced human participants. Human participants also indicated their confidence level and reasoning for each transcript (Appendix B). At the end of the study, participants were asked a series of debrief questions about the reasoning for their judgements, their methods, and their mental/emotional beliefs about AI (Appendix C). For analyses, we again used the lmertest package in R to fit linear mixed effects models.

3.2 Results

We found that displaced human adjudicators were significantly less accurate (48.6%) than interactive human interrogators (64.8%, $z=4.49,p<0.001$, see Figure 2). It is important to note that the displaced adjudicators were recruited from a different population than the interactive interrogators, which could confound the insights we can draw about the effect of displacement per se.

Displaced adjudicators were only 22% accurate for the best-performing GPT-4 witness ($z=13.4,p<0.001$), but were significantly more accurate in detecting ELIZA (85%, $z=5.54,p<0.001$).The best-performing GPT-4 witness from the original interactive study achieved a pass rate of 78% in the displaced setting, which was significantly higher than the pass rate for human witnesses (58.2%, $z=9.16,p<0.001$).

We found no statistical difference in accuracy between displaced human adjudicators and AI adjudicators ($z=1.75,p=0.81$). Agreement was close to chance levels between displaced interrogators and interactive interrogators (55%), GPT-4 (56%), and GPT-3.5 (53%), suggesting that their error patterns are not correlated.In contrast to our hypothesis, the length of the Turing test had a negative effect on accuracy ($z=5.84,p<0.001$).

We found no statistically significant correlations between adjudicator accuracy and exit survey questions that asked participants how positively they felt about AI, how intelligent they thought AI systems are, their estimate of their own accuracy, their frequency of interaction with chatbots, or how much they know about LLMs (See Figure 10). Accuracy increased across trials, suggesting that participants learned across the experiment in spite of a lack of feedback (z = 2.05, p = 0.041).Many participants reported their methods changed from checking for formality or grammar to assessing inappropriate uses of slang, excessive informality, or consistency in grammar as indicative of AI witnesses (OSF).

3.3 Discussion

We found that displaced adjudicators are also significantly less accurate than interactive human interrogators.However, the two sets of participants were drawn from different populations. Interactive interrogators were recruited via social media while displaced participants were undergraduate students. We therefore cannot know whether this drop in accuracy is purely due to the effect of displacement. However, the fact that displaced participants were similarly accurate in identifying humans and ELIZA, but not GPT-based models, suggests that displacement creates an additional challenge for identifying LLM-based models. Notably, displacement prevents participants from adversarially questioning the witness or observing witness response times, and may encourage quicker decision-making or less engagement.In real-world scenarios where human interrogators do not have the opportunity to directly interrogate the source of text content, these results suggest that people may not reliably be able to distinguish human and AI-generated content.

Similar to the inverted Turing test, the best-performing GPT-4 witness had a higher pass rate than human witnesses. This suggests that when reading an online conversation between a human and an AI model, the AI system may be more likely to be judged to be human than an actual human is.

The negative effect of transcript length on accuracy is counter-intuitive: how could less information be more helpful to adjudicators?In fact, this reversal may be due to a secondary confound in the way that transcript length was determined.Interrogators in the original study had the option to end the conversation early, which they may have done when they were confident they were speaking to a human or an AI.Shorter transcripts, therefore, might be more likely to contain information that was especially helpful to interactive interrogators, and may also be aiding displaced ones.

4.1 In-Context Learning

One key difference between the displaced human experiment and the inverted experiment was that human adjudicators completed 50 transcripts in series, while LLM adjudicators were presented each transcript separately.Although human participants were not provided with feedback after each trial, serial presentation may have allowed them to learn more about the distribution of transcripts and even markers of individual AI witnesses which were represented multiple times (supported by the increase in accuracy across trials, see Figure 10).In order to determine whether similar in-context learning (ICL) dynamics could influence AI adjudicators’ performance, we conducted an analogous experiment, presenting transcripts serially to the model.

We randomly sampled 50 transcripts from the full set, and presented them to GPT-4. After each transcript was presented, the transcript—alongside GPT-4’s verdict, confidence, and reasoning—was retained in the set of messages presented to the model.In the original inverted tests, GPT-4 had an average accuracy of 36.4%. With ICL, GPT-4’s accuracy increased to 58%, nearly exactly matching displaced human adjudicator accuracy (58.2%).

We followed the ICL tests with the same set of debrief questions originally presented to humans. In response to questions about the strategy used to detect AI and how this changed over time, GPT-4 reported shifting from checking for grammar to assessing inconsistencies or randomness; humor and emotional nuance; long-term context, connectivity, and flow in conversation; idiomatic expressions; and specific cultural references or personal details as indicative of human witnesses (Appendix D).

4.2 Reason analyses

In order to evaluate the kinds of reasons that different adjudicators cite for their decisions, we used GPT-4o to classify reasons from each verdict using the classification scheme provided by Jones and Bergen (2023).We presented the original game transcript and verdict to GPT-4o, along with the reason schema for the verdict type (Human or AI), and an example response to a transcript. The model was instructed to produce at least one class but potentially more for each transcript, following the original instructions used for human evaluators.

Reason classes were strikingly similar across adjudicator types (see Figure 4). The most common reason for AI judgements across all three types was the witness giving ‘unnatural responses’. The AI adjudicators focused more on other socio-emotional factors, such as the witness lacking personality, while displaced human responses were more evenly distributed across other classes.The equivalent picture for human verdicts was even more consistent, with the top reasons for all 3 adjudicators being plausible responses and an informal tone. Other top reasons tended to focus on socio-emotional factors and linguistic style. This commonality suggests that, to the extent that either type of adjudicator has introspective access to their decision process, they emphasize similar kinds of features in justifying their decisions.

4.3 Statistical AI-detection methods

To investigate whether statistical AI-detection tools could outperform human and AI adjudicators in judging Turing test transcripts, we implemented two methods from Mireshghallah etal. (2024), who achieve 90% accuracy in identifying GPT-4-generated content.First, we measure the mean log likelihood of the tokens generated by the witness, conditioned on all preceding tokens in the transcript.Second, we measure the curvature of the witness tokens—intuitively, how likely they are compared to close neighbours—by permuting each witness message using T5-3b (Raffel etal., 2020). Following Mireshghallah etal. (2024), we used OPT-125m (Zhang etal., 2022) as the detector model for both analyses, and a permutation rate of 2% (with a minimum of 1 permutation). We found the mean change in log likelihood across 5 permutations to each witness message, and used a Receiver Operating Characteristic (ROC) curve to find the optimal threshold to determine model accuracy across the set of 500 transcripts.

AI witness messages were assigned significantly higher probability by OPT-125m than human messages, in line with prior results ($t=-5.23,p<0.001$, see Figure 5).ROC analysis suggested an optimal threshold of -5.12, which produced an overall accuracy of 62%.Although log likelihoods for GPT-3.5 and ELIZA tended to be higher than for humans, the best GPT-4 witness produced responses with a similar log likelihood to humans, and the optimal threshold accurately classified only 47.2% of this witness’s transcripts.

The curvature method was more effective. Messages by human witnesses had a significantly lower curvature than AI witnesses ($t=-8.84,p<0.001$). ROC indicated an ideal threshold of -0.88, which resulted in an overall accuracy of 69%.Unlike the case of log likelihood, all of the subtypes of AI witness had higher mean curvature than human witnesses, underlining the value of the method.

It is striking that the likelihood of the best-performing GPT-4 witness is more similar to the distribution of likelihoods for humans than other AI models. This implies these more simplistic metrics will not be reliable tools for detecting AI-generated content, and could suggest that human comprehenders are intuitively sensitive to the predictability of responses.In addition, the clear difference in mean curvature between human and LLM witnesses highlights the value of this approach. However, there was a large degree of variation around these means (see Figure 8), resulting in low discriminative accuracy.Future work could more rigorously investigate whether other statistical AI-detection approaches perform better in this setting.

5.1 Naive Psychology

Watt’s criterion for passing the inverted Turing test is that a machine perform on par with human evaluators. If human evaluators can differentiate between humans and a particular AI, an AI adjudicator should succeed as well. If humans cannot differentiate, then the machine should fail.

We found no statistical difference between GPT-4 and displaced human adjudicator accuracy, and ICL increased GPT-4 accuracy to match displaced human accuracy (58% and 58.2% respectively). These results suggest that AI is neither better nor worse than humans at accurately attributing mentality to different witnesses. That said, we also found that displaced humans and AI adjudicators showed low agreement on their verdicts (56%); though they had similarly poor verdict accuracy, they were not inaccurate in the same ways. Considering Watt’s premise, our results imply that any model of mind these AI systems have is systematically different from human adjudicators’ models. Future work could investigate this further by conducting the inverted test interactively and systematically varying features of transcripts to isolate factors that lead human (but not AI) adjudicators to perceive other human-like minds.

5.2 Detection in the Wild

The displaced Turing test appears to be more challenging than the interactive test, but this variation is also likely to occur more frequently in everyday life.As many online conversations are now public—on forums, social media platforms, and group chats—an interaction between any two users is likely to be read by a much wider group who don’t have the opportunity to directly interrogate potential bots.Our findings show that both GPT-4 and humans are relatively poor at differentiating between humans and AI in this setting. This effect is especially robust for the best-performing AI-witness, which had a higher pass rate than humans according to both adjudicator types.Together these results suggest that well-designed AI systems that are tasked to impersonate humans in public online spaces could be more likely to succeed at masquerading as humans than an actual human would be.

Statistical AI detection methods show some promise as a technique to defend against counterfeit humans online. Curvature, a measure of the likelihood optimality of an utterance with respect to small perturbations, showed a clear difference between human and AI-generated messages. ROC analysis suggested a threshold which led to 69% accuracy across our dataset, higher than any other type of adjudicator including interactive humans. Nevertheless, this level of accuracy would be impractical for real-world problems where false positives could lead to negative outcomes for real human users Chaka (2024). Future work should investigate a wider variety of statistical approaches to understand whether alternative methods could be more applicable for short informal conversations.

Both adjudicators appeared to do better with some form of learning, as we saw displaced human accuracy increase over trials and GPT-4 accuracy increase with ICL. Future research should empirically assess whether exposure improves displaced accuracy. Similar experiments may choose to give participants feedback after each judgment, to assess for learning effects. Studies may also explore interactions between these tools, assessing, for example, whether access to AI verdicts influences displaced verdicts, or visa versa.