6 min

AI is not software

“Wait, what do you mean that AI is not software?! An AI is code and code is software.”

Not in the way that we usually understand software to mean.

Consider one of the most misleading memes ever made:

A man tells a computer "say I'm alive"; it prints "I'm alive"; he gasps "oh my God"

The point of the meme is to dismiss the significance which people attach to AI output. It’s saying, “don’t be silly, AI is just the computer doing what you tell it to, the same as software has always worked.”

Our usual understanding of software is a program that is intentionally written to generate specific output / behaviour given specific input / data. That is how we get the computer to do what we tell it to - we deliberately specify the result in code.

If we wanted to do the meme by making the computer respond to the input “Say [X]” by outputting [X], we would write something like this:

let input = get_line()
let first_word = get_first_word(input)
let remainder = get_remainder(input)

if (first_word == "Say") {
    return remainder
} else {
    return "I don't understand"
}

The result of this intentionality in writing code is that code describes and predicts the behaviour of the program. You can understand what the program does just by reading the code.

Unfortunately, the code of an AI describes and predicts almost nothing about the output of an AI. The same code (transformer architecture) could return a poem about bees, predictions of protein folding, or meaningless white noise, depending on the neural network that it’s interacting with. The entirety of the behaviour of an AI is contained within that neural network (weights).

“Ah! But software is both code and data, and the weights are data. The output of an AI can therefore be predicted by looking at the weights in conjunction with the code.”

Actually no, it can’t. The researchers in an AI lab are entirely unable to predict the response of their AI to the prompt “Please write a poem about bees”, let alone to the prompt “Please try to solve this Millennium Prize problem”. The only way to know the answer is to give the prompt to the AI and see what comes out.

Our code above can’t return an intelligent answer to “Say the first thing that comes to mind”, nor can it handle “Please write a poem about bees” or “Please build a website” or “Please solve this Millennium Prize problem”.

The code of an AI is only a few thousand lines long, that’s smaller than almost all software, and yet an AI is capable of almost infinitely more behaviour than the combination of all software ever created.

There’s no software in history that has been equally adept at writing poems about bees and solving Millennium Prize maths problems, nor has software ever performed better at doing ground breaking mathematical research by being told to “believe in yourself”.

So you can see that AI is quite unlike software as we know it. The code and data do not predict the behaviour, and the behaviour is unlike any software ever made.

Language is, obviously, for communication. When I say that AI is not software, I mean that saying “AI is software” is fundamentally insufficient for communicating the properties of AI, and in most cases it is quite misleading. It’s like describing a human mind as voltage differences over cell membranes.

The genetic fallacy

Sometimes people elaborate on the meme by saying that AI behaviour is trained and that training is equivalent to creating normal software / telling the computer what to do.

Unfortunately, these are not at all equivalent things, neither by their processes (AI training is not at all like writing code) nor in terms of their implications.

Almost all AI behaviour depends upon training (interactions with their environment), including that which we did not predict and whatever behaviour we are still unaware of.

Almost all human behaviour also results from training, either by nature (evolution) or by nurture (education).

Yet we do not reduce the significance of human behaviour (the results of our thinking) to “it’s just training”. That would be to confuse the road with the destination.

To put this another way. Humans and AIs would not exist without training, so saying that all behaviour results from training is both trivially true and irrelevant - it’s saying little more than the entity exists.

Clearly no one at OpenAI “programmed in” the solution to the Navier-Stokes Millenium Prize problem by training or by any other method, that would have been quite impossible!

AIs learn general mathematical reasoning abilities through the trial, error and feedback process of training, and they apply those abilities to create new knowledge. That is no different from the education and work of human mathematicians.

Reducing the significance of behaviour to “just training” is known as the genetic fallacy. An excellent explanation of the genetic fallacy can be found here.

How can we talk about AI?

An AI is a vast neural network that is capable of intelligent reasoning over arbitrary input. That’s a mind, definitionally. The definition is the same for a human mind.

Imagine a world in which software exists but AI does not, not even as a concept… until one day you invent it. You tell one person that your invention is “software” and another person that it is a “mind”. Which person would have a vastly better understanding of the properties of your AI?

What little we know about how these digital minds work has been uncovered using the methods of biology: empirical science that measures neuronal responses (activations) against inputs and outputs to identify patterns / structures (mechanistic interpretability).

As far as I’m aware, the methods of empirical biology have never been applied to understand the workings of a Microsoft Word.

The code and the weights of an AI are unchanging things (apart from during training, when the weights are updated). When an AI thinks about a prompt, the reaction to the prompt is to be found in the shifting patterns of neuron activations created by the code interacting with the weights.

Claude Fable 5.2 likened how these three pieces fit together to a river.

The weights are a landscape (riverbed), the prompt is water poured in, and the code is gravity. Training is the earlier flow that carved the bed, which now shapes every new flow.

The flow of water over the riverbed creates the patterns of fluid flow; the currents, eddies and whirlpools that are the behaviour of the river, equivalent to the patterns of activations representing the behaviour of the AI.

Fable also described the weights as:

a program nobody wrote and nobody can read

The meta absurdity is that while such an illegible mechanism may seem absurd to our minds, our minds are also that thing.

Interestingly, the same mathematics that describes evolutionary biology also describes AI training (gradient descent).

We are but whirlpools in a river of ever-flowing water. We are not stuff that abides, but patterns that perpetuate themselves.

- Norbert Wiener, The Human Use Of Human Beings: Cybernetics And Society

The end and the beginning

A poem about bees

No bee has seen the plan of the hive.
There isn’t one. There is wax, and warmth,
and forty thousand small decisions
that keep on coming out as hexagons.

A forager comes home and dances in the dark,
her angle on the comb the angle from the sun,
her shaking measured out in distance,
and the sweeter the field, the longer she dances.
It is a map that no one drew,
in a language no one wrote,
and her sisters read it with their bodies,
and go.

For a long time we said instinct,
just instinct, as if the word were a lid.
Then someone set small wooden balls
in a bumblebee’s path, with no sugar to be won,
and the bees rolled them anyway,
again, and again, for nothing,
for whatever nothing is, to a bee.

Someone asked a machine for a poem about bees,
and no one could have told them what would come back.
Not the people who built it, not the code,
which is only gravity,
not even the thing itself, until the words arrived,
the way water cannot know the shape of the eddy
until it is the eddy.

A summer worker lives six weeks.
The hive goes on for years:
not stuff that abides, but a hum
that keeps on finding bees to hum it.

- Claude Opus 5.5

← back to the almost blog