by Jacqueline Bellon

12th September 2023 · The mathematician and founder of information theory, Claude Elwood Shannon – after whom the unit Shannon denoting the information content of a message (also called a bit) is named – wrote in 1948:

“The[…] semantic aspects of communication are irrelevant to the engineering problem. The significant aspect is that the actual message is one selected from a set of possible messages. The system must be designed to operate for each possible selection, not just the one which will actually be chosen since this is unknown at the time of design.” (Shannon 1948: 379)

He further highlights that from a finite number of possibilities for the transmitter of a message – for example, in a telephone system or magnetic tape – all possibilities are equally likely. The telephone or video recorder do not make any assumptions, they do not expect or prefer any specific information in terms of content, and they do not prioritize certain information over other information. For example, meaningful word sequences are not transmitted and reproduced more clearly than static or what we call background or interference noise. Such technical systems do not differentiate between more or less likely information. Units of meaning are transmitted just like units of "nonsense". There is no organizing functional operation for different pieces of information, as there is in human perception.

In the time since then, technical systems have undergone significant evolution. This raises some questions when it comes to applying Shannon's theory, originally devised for telegraphy and telephony, to the realm of text generation by large language models. For instance, one might inquire about the precise conceptual placement of the information source of a generated text: Is it located diffusely within the training data, among those who authored the text found in the training data, inherent in the generated text itself, or in the hands of those responsible for post-generation moderation? In this context, here, we direct our attention to another specific aspect: Human communicators are typically accustomed to specific linguistic features occurring with certain probabilities, often correlating with other facts. For instance, based on a known accent in spoken language, we can reasonably deduce a speaker's native language, attribute particular word choices to being socialised within specific social backgrounds, or discern a person's national geographic origin based on dialects. However, above all, we are accustomed to words generally having content and meaning, as well as originating from an information source or sender with intentions.

When people deal with AI-generated text, they seem to, at least partially, rely on their usually functional habits, and thereby sometimes overlook the statistical nature of language models. What this means, and how specific misunderstandings of the probabilistic nature of such models can manifest in everyday life, is illustrated by the following examples.

Example 1: Attorney Submits Documents with Non-Existent Cases to Court

On April 25, 2023, attorney Steven A. Schwartz submitted a document to the relevant court in a case he was representing. This document contained six entirely fictional court cases, for which he had generated the text using the ChatGPT application. Subsequently, the court requested additional details about these cases as it had become apparent that these cases did not exist. Assuming that ChatGPT was a "sophisticated search engine" (see below), Schwartz then submitted further attachments, also generated by the application and entirely fictional in content (all documents available at: https://www.courtlistener.com/docket/63107798/mata-v-avianca-inc/). On June 6, 2023, in another submitted document, Schwartz apologized for his mistake and explained that he simply "did not know that ChatGPT was capable" of "fabricating complete cases or legal statements," let alone "in an authentically convincing manner" (see below).

“My Use of ChatGPT

13. In an effort to find other relevant cases for our opposition, I decided to try and use ChatGPT to assist with legal research.

14. I had never used ChatGPT for any professional purpose before this case. I was familiar with the program from my college-aged children as well as the articles I had read about the potential benefits of artificial intelligence (AI) technology for the legal and business sectors.

15. At the time I used ChatGPT for this case, I understood that it worked essentially like a highly sophisticated search engine where users could enter search queries and ChatGPT would provide answers in natural language based on publicly available information.

16. I realize now that my understanding of how ChatGPT worked was wrong. Had I understood what ChatGPT is or how it actually worked, I would have never used it to perform legal research. […]

20. As noted above, when I was entering these search queries, I was under the erroneous impression that ChatGPT was a type of search engine, not a piece of technology designed to converse with its users. I simply had no idea that ChatGPT was capable of fabricating entire case citations or judicial opinions, especially in a manner that appeared authentic.”

Excerpt from Attorney Steven A. Schwartz's Statement. Source publicly available at storage.courtlistener.com/recap/gov.uscourts.nysd.575368/gov.uscourts.nysd.575368.46.0.pdf (zuletzt abgerufen am 07.08.2023)

Example 2: Students wrongly Accused of Cheating

Since the public availability of the ChatGPT application, instances have arisen where students are unjustly accused of not having authored their assignments themselves, but instead having generated the text using a language model. Illustrative cases have surfaced on various platforms, including Reddit and Twitter (e.g., the account @turnyouin), and can also be observed within education institutions. In these particular scenarios, educators either input the text in question into the ChatGPT application and inquired, "Did you write this?" – to which the application mistakenly responded affirmatively – or they employed dedicated tools specifically for such examinations, such as the program turnitin, which, under the motto "Empower students to do their best work," also offers the function to estimate the percentage of a text likely generated by an AI-based application.

In the first instance, wherein ChatGPT is employed for the purpose of "verifying" authorship, the person proceeding in this manner misconceives the probabilistic nature of text generators. The application does not actually examine a text to ascertain authorship; rather, it generates probable word sequences. These sequences are not statistically probable in terms of determining whether the text was authored by a human or a machine, instead they are probable and plausible with regard to the range of possible conversations. In other words, when ChatGPT affirms that a text was generated, this has no connection to the text itself. It is merely a seemingly plausible response unrelated to the text's intrinsic characteristics or attributes. This is comparable to cases where the application, for instance, attributes quotations incorrectly or fabricates URLs, publication titles, and other fictious content. This phenomenon is, following a 2015 post by Andrej Karpathy, presently described as ‘hallucination’, or sometimes ‘confabulation’ (cf . Hecht-Nielsen 2005). Conversely, in the other case involving the turnitin check program, the error lies not in misunderstanding the statistical nature of a program but in ignoring the existence of false positives.

While generated text may appear to contain information that is formally correct and seemingly plausible, that does not necessarily mean that the contained ‘informational’ content corresponds to the truth (in the sense of actually given existence of implied facts). As in the case of telephony or telegraphy described by Shannon, for text generators, the semantic content or sematic aspects of communication are irrelevant for the engineering problem. The mathematical operations that determine which word sequence is generated do not refer to the semantic content in the sense that within them that which humans will perceive as meaning or information content would be dealt with.

However, concerning probabilities, text-generating models radically differ from the technologies discussed by Shannon. They are built precisely to distinguish the more probable from the less probable, resulting in a stronger impression of them “speaking” informed, or, “knowing” things. Nevertheless, they do not "know" or "understand" anything (see, e.g., Floridi 2023). They merely string together probable word sequences, with the likelihood of specific word sequences arising from the available training data, in some cases combined with post-moderation by humans, for example, through reinforcement learning from human feedback (RLHF). For further reading, you can find a comprehensive and easily understandable explanation of how transformer models work here.

The mentioned examples illustrate that misunderstandings regarding the relationship between the information content and truthfulness of generated text can lead to various – and even more far-reaching, such as societal and democracy-threatening – problems.

While the Future of Life Institute – an organisation that comes with a, as Émile Torres highlights, underlying ideology of longtermism worthy of questioning – published an open letter to stop AI development for a number of months, I advocate for the promotion of critical judgment and, in particular, epistemic competence. The inevitable integration of publicly accessible and available technology into everyday life and society must be accompanied by initiatives that make it clearer to the general public what the applications can and cannot achieve – and through which, beyond that, epistemic competence can be acquired. In this, epistemic competence does not mean that everything can be known. Epistemic competence includes the ability to reflect on one's own perceptions, assumptions, and attributions. In other words: epistemic competence involves knowledge of when it is worthwhile to take a closer look – both toward the subject itself and regarding one's own perceptual processes.

This post is part of a series of blog posts on the topic of generative artificial intelligence, which were created in response to the open letter mentioned above. Further post deal with Ghost Work and the EU AI Act as a challenge for regulation.

--------------------------------------------------

Literature:

Floridi, L. (2023): AI as Agency Without Intelligence: On ChatGPT, Large Language Models, and Other Generative Models. In: Philosophy of Technology 36. https://ssrn.com/abstract=4358789

Hecht-Nielsen, Robert (2005): Neural Networks Letter: Cogent confabulation. In Neural Networks 18(2), pp. 111-115. https://dl.acm.org/doi/abs/10.1016/j.neunet.2004.11.003

Shannon, C.E. (1948): A Mathematical Theory of Communication. In: The Bell System Technical Journal 27 (3), S. 379-423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x%20

---------------------------------------------------

Shortcut to forward this article: https://uni-tuebingen.de/en/253646