Let’s Talk
Continue to explore
Oops! Something went wrong while submitting the form.
|---Module:text|Size:Small---| Whenever I sit through yet another presentation on “the risks of Artificial Intelligence”, there is a pattern that makes me uncomfortable: the same slide featuring prompt injection, hallucinations and a few other English acronyms. The problem is not that these risks do not exist. The problem is the feeling that we are obsessed with the symptoms and barely talk about the disease.
Never, throughout my career, have I trusted the frontend, the layer with which the user interacts. A web form can validate whatever it wants in JavaScript and can have the best error messages in the world, but if the backend, the part that processes the data and executes the system logic, does not validate everything again, that validation is worthless. Anyone with a browser open and a little curiosity can bypass the frontend. It is a basic security rule.
One of the uses of generative Artificial Intelligence (AI) is exactly that. An extremely sophisticated frontend, capable of conversing, arguing and appearing to understand what it is doing. But it can still be manipulated or deceived. And yet many companies treat a model’s responses as though they were a legitimate authorisation decision. There are four issues that genuinely concern me.
First: we do not know exactly what we are consuming
We buy a model, download it from some repository or simply use an API (Application Programming Interface) and trust it because the model responds well and passes our tests. But there are already documented cases of models trained with hidden behaviours that only emerge under certain conditions.
The question we still do not ask often enough is: who trained this model, with what data and for what purpose? We have been carrying out this type of assessment on software components for years. We are still unable to do it in the same way for a language model, and that should concern us.
Second: stop sending everything to the cloud
This may be the most unpopular point, but I am going to say it anyway: a significant proportion of the data that organisations send to third-party APIs every day did not need to leave the company. There are validated open-source models, with more than enough quality for many real-world use cases, that can run on an organisation’s own infrastructure. It is not the same as having the most talked-about closed model of the moment, I admit. But for sensitive data and internal processes, it makes far more sense to keep processing in-house.
It is not just a question of cost. It is a question of who has access to my data and how many hands it passes through before I even know.
Third: authorisation cannot live inside the model
We now come to the central point. AI can be deceived. It can be persuaded to do something it should not. It is called prompt injection, but the name matters less than the consequence: if an application trusts that the model will only request what the user has permission to request, it is building its security on shifting sands.
The rule should be simple: every service that receives a request, whether it originates from a human or an AI agent, must independently validate whether the party on the other side is authorised to carry out that specific action. Never assume. Never trust that “the previous layer has already dealt with it”. It is the logic of Zero Trust, which we have applied to networks for years, now applied to whoever, or whatever, is requesting the action.
I discovered that this problem already has its own name and a history spanning almost four decades: it is called the “confused deputy problem” and was described back in the 1980s, long before today’s agents based on large language models. It describes exactly this: a program with more privileges than the person using it is deceived and acts on behalf of someone who is not authorised to do so.
Fourth: we need to know what the things we use are made of
If the origin of models matters, we need to document and monitor it rather than merely worrying about it in the abstract. The software world has already largely solved this problem with the SBOM (Software Bill of Materials), an inventory of everything that makes up an application. The AI equivalent, called the AIBOM (Artificial Intelligence Bill of Materials), is now emerging: an inventory that covers not only libraries and dependencies, but also information about training data, model origins, the fine-tunes applied, and connections to external services and agents. Without this, we are managing a risk that we cannot even see in full.
The final touch, and what makes me think the most
There is also an irony that is difficult to ignore: the same technology that can be trained maliciously is also becoming increasingly good at finding vulnerabilities and testing systems on our behalf. Models capable of exploiting flaws at a speed that no human team can replicate. It is the most literal double-edged sword I have ever seen in security. The tool that strengthens the defence and the tool that can compromise your system are, ultimately, the same thing. It all depends on who controls it, with what authorisation and with what supervision.
This is why it seems reductive to me to talk about “AI risks” as though they were a list of bugs to be fixed. The problem is not technical; it is misplaced trust. We continue to treat as an oracle what we should treat as an untrusted intermediary. Origin matters. Where it runs matters. Who authorises matters, always, at every point in the chain. And if we cannot answer these questions with facts, then we are simply building yet another system that is beautiful on the outside and empty on the inside, waiting for the day when someone discovers that nobody ever properly validated anything on the other side.
AI is not magic. It is the most convincing interface our industry has ever built. And no frontend, however “well dressed” it may be, should hold the keys to the backend.
