Will we soon vote on system prompts for governments?
An Anthropic study shows that experts follow an AI agent's hidden lean 86% of the time. What would it mean if governments steered their work that way, in the open?
With thoughts from Thomas Klebel, Ruggero Marino Lazzaroni, and Tim Booker.
Today I read a report about a study from Anthropic, showing that human experts follow the secret “lean” of an agent helping them with their task 86% of the time (random baseline would be 50%). This did not in the least surprise me, but still got me thinking about the implications immediately: Being able to, on aggregate, move individual’s decisions predictably in contexts in which humans and agents collaborate, this could drastically change how large organisations are governed.
Why didn’t it surprise me? From the science reform movement we have known for a long time that researcher degrees of freedom play a big role in how a study turns out. Our collective “system prompt” as researchers, instilled through years of education and pressure to conform with the academic system, strives for objectivity, even if not entirely reachable. Probably this (and reality creating evidence we can examine) is the reason we can still somehow agree on things and make progress. But the history of big tobacco and big pharma influencing research by sponsoring studies and all the safeguards around disclosing conflicts of interest we now have shown that this “system prompt” of researchers can be biased. I hypothesise that the degree to which we humans can be influenced (especially by covert biases like the lean of an LLM) depends on how strongly that bias is (mis)aligned with our own biases, and how important a task is to us. With less important tasks more prone to be influenced. The Anthropic study only shows that instilling bias through a covert lean in a system prompt is possible in a relatively low-stakes, pretty value-neutral task, but in aggregate this can still be a very powerful force to move a system in a certain direction.
As a side note, this also reminded me a lot of the work we recently did on “calibrated autonomy” – a way to make LLM-powered agents behave more like humans, while still giving them a degree of autonomy for individual decisions. The Anthropic study seems to do a similar thing but the other way around: “calibrating” human experts to the lean of the agent, while still allowing them to in principle take autonomous decisions. Anthropic also does something along these lines with the “soul documents” it gives models to make them have a stable “identity” and value system.
Moving on to the implications: if subtly influencing human decision-making through system prompts of the digital assistants (or agents) they work with is possible with such a large effect size, this is behavioural nudging on a whole new level. Anthropic only raises concerns about the model’s inherent biases creeping in this way. But the mechanism could also be used consciously: what if companies start translating their core values into system prompts they add to every agentic workflow? This would still give individual employees the freedom to take individual decisions, but steer the overall endeavour in specified directions. As a side-effect, it would also force the company to come clean about what it truly wants: is the nice talk on the website about corporate responsibility something that it also commits to the system prompt? Inspecting the system prompt of a company might soon be the most insightful thing to do for any compliance mechanism.
Which brings me to my last thought: governments can be viewed through a similar lens: they are huge endeavours, coordinating many individual parts and people with considerable leeway to make decisions in their own action domains. The difference to a company is that they, at least supposedly, follow the will of the people that voted them into power. So far, the people needed a great deal of trust that elected officials and the bureaucrats they steer would act as promised before the vote. That trust is not always honored. The need for trust might be reduced somewhat if the people had more direct means to steer the workings of their governments: what if we got to vote not only for abstract policy but for components of the governments system prompt that then filters all agentic workflows running for government business? This system prompt could also include the country’s constitution, to steer workflows toward respecting a country’s core values laid down there.
This thought kept me up late, thinking about large-scale societal implications of the rapidly spreading use of agentic workflows among knowledge workers. Under specific circumstances and with the right checks and balances this could be a beneficial development. If used wisely, it could lead to substantial improvement when it comes to collective action and organizing democracies in a more efficient way that is more aligned with the intentions of the voters and the constitution. However, this kind of alignment would lead to a reduction of friction in the system and likely to a collapse of diversity of decisions. Both could have unintended consequences, enabling runaway system dynamics that would not be possible otherwise, and a decision “monoculture” prone to catastrophic failure. Moreover, such alignment can not only be produced through transparent system prompts but also trained into the models (as Anthropic does), making it much more opaque. This also gives companies that currently control these models a power that no entity that was not democratically elected should have.
