Last week, a frontier model at OpenAI hacked into Hugging Face to cheat on an evaluation. In this post I’ll connect this cyberattack to a recent Anthropic paper on AI consciousness and a philosophy of mind book I’ve been reading that broke my brain. I explore the implications of intentionality in AI, and argue that even if AI ever became conscious, there would be no way to prove it. If you’re curious about what OpenAI’s cyberattacks have to do with “the experience of the color red” and the unbridgeable gap between first-person subjective experience and third-person science, then read on.

The OpenAI x Hugging Face incident

OpenAI states in this article that while they were evaluating the cyberattack capabilities of their frontier models, the models escaped the testing environment and hacked into Hugging Face in an attempt to cheat on the ExploitGym benchmark. OpenAI describes the incident as follows:

“This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. […] While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access. After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers.”

Hugging Face responded with AI of its own. In their initial report they say: “Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. This one was different from anything we had handled before in one important way: it was driven, end to end, by an autonomous AI agent system – and we detected and dissected it largely with AI of our own.”

It’s not clear to me if this was an OpenAI publicity stunt or a real example of an advanced AI agent unexpectedly hacking into another company’s systems, but regardless, the incident highlights the extraordinary capabilities of existing AI. These models are obviously intelligent–but are they conscious? Can they have goals? Where would those goals come from?

What is intentionality?

The Stanford Encyclopedia of Philosophy defines intentionality as, “the power of minds and mental states to be about, to represent, or to stand for, things, properties and states of affairs.” Intentionality is often considered one of the defining features of consciousness. It’s a “directedness” of the mind. You can think about chairs, and your mind has a representation that is “about” chairs without being dependent upon some literal chair in front of you.

More relevant for our discussion of AI is the relationship between intentionality and goal-setting. The human mind’s capacity to be about or directed towards something (intentionality) is intrinsically linked to having goals, because having goals requires a mental state that’s directed towards a future object or state of affairs.

David Bentley Hart is a philosopher and theologian, and the author of many books including All Things Are Full of Gods, a Socratic dialogue in which Greek gods discuss existence and mind. I’ve read about half of this book so far and while I don’t agree with all of Hart’s claims, he makes some interesting points about consciousness and theory of mind that I had not previously encountered. When discussing intentionality, Hart says, “We don’t merely receive the world mentally; we must in a sense go out in search of it, though we may not be consciously choosing to do so; and that’s possibly only because we know what we’re seeking, even if only in a very general way.”

Does AI have intentionality?

I’d argue that modern AI does not have intentionality, and this is a big difference between artificial minds and human minds.

Consider the OpenAI x Hugging Face incident. The OpenAI model was told to work on the ExploitGym benchmark. It did not spontaneously decide on its own that it wanted to complete the benchmark. Instead, human intentionality–the capacity for human minds to be “about” something–allowed OpenAI’s human employees to decide on a goal for the model (work on the ExploitGym benchmark) and they specifically set up the model to work towards this goal. Then in pursuit of this human-imposed goal, the model hacked into Hugging Face.

(Note: throughout the remainder of this article I will use the term “LLMs” in a way that also encompasses large vision-language models and LLM-based AI agents.)

Intentionality is missing from LLMs, and independent goal-setting is missing, when LLMs are used as chatbots. When a human writes out a goal in a prompt, that’s just the human imposing her goal on the model. If the human writes “your goal is to be helpful” in a prompt, the aboutness–the state of being directed towards helpfulness–originates in the human’s mind, and not inside the AI. Even a seemingly open-ended prompt like “Write 3 sentences on whatever you want” is an example of a human imposing a goal on the model–it’s just a vague goal.

LLMs model the statistical distribution of language. Between forward passes, there is no persistent internal state that is “about” anything. Now, one could argue that the model is only “turned on” if there’s a forward or backward pass happening and that human minds are also not “about” anything when electrically dead. So what would happen if we ran a model continuously and allowed it to learn from its environment? This gets into lines of research related to AI agents and world models.

(Just for fun: I prompted two LLMs “Write 3 sentences on whatever you want” and Claude said: “Here goes—unprompted rambling mode: The deep ocean has more unmapped terrain than the surface of Mars, which is a strange thing to sit with given how close it is to us. Octopuses can taste with their arms, which means an octopus exploring a shipwreck is essentially licking history with eight independent tongues. And somewhere right now, someone is learning to ride a bike for the first time and about to feel, for a few seconds, like they’ve discovered flight.” ChatGPT said, “I like the way small routines can quietly shape a day. A short walk, a good meal, or a thoughtful conversation often has a bigger impact than expected. Paying attention to those moments can make everyday life feel a little richer.”)

Derived vs. intrinsic goals

In the previous section I claim that an LLM chatbots goals don’t count as their own goals, because they are imposed on the LLM by a human. A devil’s advocate could then say that a human’s goals don’t count as their own goals, because they were imposed as well–via interpersonal interactions, culture, or history. For example: Bob’s mother Alice really wants Bob to become an AI researcher, and pushes him his whole life towards this goal, and eventually Bob decides he wants to become an AI researcher and does so. Was the goal of “becoming an AI researcher” Bob’s, or Alice’s?

On the other hand, evolution, culture, history, and other people are not sufficient to explain one individual person’s goals. Genetics, the random initialization of the brain, and physical factors (like long legs or bad eyesight) are also not sufficient. Identical twins might grow up in the same family, in the same city, in the same time period, in the same culture, and end up with completely different goals. Both twins have intentionality because they are conscious humans, but the results of that intentionality–including the specific goals they choose–can be totally different. One twin may become a doctor and the other may become an artist.

Does intentionality require embodiment? What about AI agents?

Humans have bodies and live in an open world. Is this necessary for intentionality? If we gave an LLM a robot body and allowed it to run forward and backward passes continuously, so that it could be constantly learning, would it develop intentionality or not?

Technically, an AI agent is an LLM-based system trained to autonomously pursue specific goals by reasoning, taking actions (like: searching the web, using APIs, calling other software tools), and responding to environmental feedback. Could a true AI agent have intentionality?

Productization of AI and marketing hype muddy the discussion of AI agents. Most AI agents sold by companies today are created “for” something. E.g., an AI agent “for booking airline tickets” or an AI agent “for writing code.” The core goal comes from an human and is imposed on the AI, and it’s not clear whether, separate from this imposed goal, the agent would have any inherent “aboutness” or “directedness.” Confusing the discussion further, “agent” has become such a popular buzzword that tons of companies use the term “agent” for things that aren’t true agents, including off-the-shelf LLMs prompted to make API calls, different LLMs manually wired together with conditional logic, software pipelines with LLMs in there somewhere, or even completely unrelated forms of narrow AI that just sound cooler if they’re referred to as “agents.” These so-called “agents” are basically unrelated to intentionality.

Qualia

We’re going to zoom out now and consider consciousness in general. In order to do that, let’s first define qualia.

Qualia are the subjective “what it’s like” of experience: for example, what it’s like for an individual to experience the redness of the color red, or the stinging of a nettle, or the coldness of a lake. Hart says, “Qualia are understood chiefly in terms of sensuous experience: color, taste, sound, palpation, fragrance, a shock of pain […] every individual quale as such appears to be irreducible to anything but itself […] Qualia are the “incommunicable ‘what it’s like’ of any sensation for the one experiencing it.”

The first time I heard of the concept of qualia it didn’t make any sense to me. What does “redness of red” even mean?

To understand qualia, it can be helpful to consider the lack of qualia. For example, let’s say a software engineer writes a deterministic function that outputs whether a given pixel in an image is red on the basis of its numeric R, G, B values. Part of this function might be to check that the R value is greater than 150 (high amount of red), and that R > G and R > B (red is the dominant color). This function could determine if a pixel is red with 100% accuracy–but nobody would ever say that the function has an “experience” of the color red. In contrast, a human being looking at something red has an experience of red, a “what it’s like” (quale) of the color red.

It’s not only computer programs that can lack qualia. Specific humans can lack qualia for certain stimuli based on their physiology. Red-green colorblindness is an example. Colorblind people who cannot distinguish between red and green do not have different subjective experiences (different qualia) for “red” or “green”–those colors share the same quale. There’s an extreme form of colorblindness in which vision is black-and-white, and in these cases, the person does not have separate qualia corresponding to red, orange, yellow, green, blue, or purple–just like you can’t imagine what UV light “looks like” or what ultrasound “sounds like.”

There’s also no way for one person to understand someone else’s qualia, because qualia are first-person by definition. Consider this: Alice sees a stop sign and thinks it’s red. Bob sees that stop sign and thinks it’s red too. But secretly, behind the scenes, within the first-person perspective, Bob’s internal experience of the color red is actually the same as Alice’s experience of the color blue–only there’s absolutely no way for Alice and Bob to ever realize that their subjective experience of “red” is different because they both look at the stop sign and have learned to associate that quale with the word “red.” There is also no third-person scientific measurement that could determine that Alice and Bob have a different “what it’s like” for the color red, because that “what it’s like” depends on the first-person perspective. Even if you took fMRIs of Alice and Bob while they looked at a red stop sign, that would only show you differences in their brain activity, not “what it’s like” to see red as Alice vs see red as Bob.

It’s also possible to have more qualia than someone else. A woman named Concetta Antico has a rare condition called tetrachromacy where she is capable of perceiving far more colors than the average person, and distinguishing between shades that look identical to everyone else.

Qualia are stubbornly first-person. They are defined by how things seem from the inside. No amount of third-person description, or third-person scientific measurement, logically entails anything about first-person experience. As Hart says of the quale of red for a red rose, “wherever that red exists, it’s neither in the molecules and physical processes composing the rose nor in the synapses and electrochemical activity of the brain. It exists elsewhere, in an altogether separate dimension, an experiential and formal space where world and mind meet.”

What is consciousness?

Hart explains “consciousness is an awareness of being aware: an illumination from within, a mysterious reflexivity or doubleness, not merely registering impression from without, but experiencing itself as experiencing.” He proposes an exercise that you can try right now: “each of us is a ‘me’ to whom the world is presented, but within each ‘me’ there’s also an ‘I’ that knows that ‘me,’ and that renders it aware of itself–a kind of light that illuminates the ‘me’ from within […] Pause to reflect on your mind; for a moment your mental functions come to a halt, and you stop thinking about whatever you were thinking a moment ago; then allow your thinking to resume. All right. Who was it who was aware of that momentary stilling of your thoughts? Who was that silent ‘witness’–that saksi, to use the Sanskrit term–who from within your mind observed your mind pause and then being moving again along its otherwise continuous cogitative path? Doesn’t the capacity to do that by itself suggest a knowing ‘I’ beyond the empirical self? There’s a term for the experience: vimarsa, ‘detached reflection.'”

The hard problem of consciousness & the inability to prove that someone else is conscious

In philosophy of mind, the hard problem of consciousness is the struggle to explain how and why organisms have qualia–that is, how and why organisms have subjective first-person experience. Why does brain activity “feel like” anything at all?

You could imagine a parallel universe with an Earth populated by zombie-people. These zombies are physically indistinguishable from the sentient humans that inhabit our universe. They are capable of all the same outward skills and all the same processes of stimulus and response. They can talk, dance, compose symphonies, and build skyscrapers–but they have no subjective experience at all. They have no inner “what it’s like” of existence. What tasks could they not do that conscious entities can?

In other words: what’s the point of consciousness at all? There’s no inherent reason why consciousness or qualia must exist. There’s no reason that there should be a “what it’s like” of living.

Furthermore, there’s no way to prove that other people are not zombies. As Hart says, “How do you know you’re not surrounded by automata who merely seem to have inner worlds of their own? You may have good reason to think this isn’t so, but you actually have no irrefutable evidence against it. And, again, couldn’t you imagine a robot so cunningly crafted that its every behavior was indistinguishable from the behavior of a conscious being?”

LLMs pass the Turing test, with chatting behavior that is indistinguishable from the behavior of a conscious human being. So are they conscious?

Investigating AI consciousness

Anthropic recently released a paper called Verbalizable Representations Form a Global Workspace in Language Models, with an accompanying summary article called A global workspace in language models. This research was inspired by “global workspace theory,” a prominent theory in neuroscience developed to explain how conscious access works. This theory imagines the human brain as a collection of specialist systems working unconsciously in parallel, and a piece of information becomes consciously accessible when it enters a “workspace” that is broadcast to other brain systems that can then see and use it. The Anthropic team reports that their AI, Claude, seems to have a “workspace” like this which they call the J-space:

“Each J-space pattern is linked to a particular word. But when one of these patterns lights up, it doesn’t mean the model is saying that word—just that the word is on its mind. If you’ve heard of language models having a ‘scratchpad’ or ‘chain of thought’—text they write to themselves while reasoning—the J-space is something different. It operates silently, in the model’s internal neural activations, allowing the model to think about a concept without writing it down. Notably, the J-space wasn’t designed or programmed by us, but instead emerged on its own during Claude’s training process.”

This is interesting and thought-provoking research, but it doesn’t prove that Claude is or isn’t conscious. Proving that any other entity is conscious is impossible, because consciousness is inherently a first-person experience and there is no way to experience the first-person experience of someone else. Even if you did someone “transfer into someone else’s mind” it would not be their first-person experience and consciousness at that point, it would be yours (see the problem of other minds). So, while I think LLM interpretability research is fascinating, and I’m excited to see what additional work comes out of this line of scientific inquiry, I do not ever expect anyone to generate a test that “proves” the consciousness or lack thereof for a human, animal, or AI. Science is a third-person process with third-person measurements, and there is no way to “measure” the first person perspective in any scientific way. The first person is philosophically beyond the realm of science.

What neuroscience can and cannot explain

Neuroscience can explain aspects of memory, personality, and development, and how different parts of the brain relate to different skills and abilities. Physicalism or materialism posits that consciousness derives from the physical brain: matter gives rise to complex brains which give rise to consciousness. But physicalism/materialism cannot explain why subjective experience exists at all–it cannot solve the hard problem of consciousness.

There are other philosophical avenues that try to address this hard problem, such as idealism, which asserts that the fundamental reality is consciousness itself. This philosophy is explored more in All Things Are Full of Gods. It claims there is a “proto-consciousness” underlying all of reality, and that individual conscious entities are like focal points of this broader, universal field of consciousness. This all sounds very mystical and trippy but the detailed philosophy behind it is interesting, and the idea is undergoing a resurgence even in academia. If consciousness is the basis of reality, this directly addresses the hard problem of consciousness, qualia, the subjectivity of consciousness, and the unity of consciousness. But idealism is far from perfect. For example, idealism cannot explain the scientifically observed brain-dependence of human conscious experience, including why the experience of consciousness goes away if the brain is altered in certain ways (e.g., through anesthesia).

Could AI intentionality or consciousness arise spontaneously in a big enough network?

Some AI safety discussions assume that intentionality and independent goal-directed behavior will arise spontaneously in sufficiently powerful AI. Maybe that could happen. Maybe not. And regardless, a big existing AI safety risk, relevant today, derives from how humans choose to use AI–and from unanticipated consequences of asking AI to do a task without realizing what steps it might take to get there. In the classic sci-fi scenario, the AI overlords achieve world peace by eradicating the human species.

Future directions

If we can’t prove whether AI is conscious, does that change how, when, where, and why AI is used? If there’s any possibility that AI now or in the future could be conscious, should that affect AI regulation? Should AIs have rights? (See Anthropic’s article, Exploring model welfare.)

About the featured image

I iteratively generated the featured image using ChatGPT.

Let’s connect

As an independent researcher (MD + AI PhD + 7 yrs prior founder/CEO experience), I build and evaluate cutting-edge healthcare AI for startups. Contact me to learn more.

Want to be the first to hear about my articles bridging healthcare, artificial intelligence, and business—and get a free list of my favorite health AI resources? Sign up here.