The Yes Machine

The precise reward mechanisms behind modern frontier AI models are neither disclosed nor necessarily aligned to public benefit. In the context of providing or processing information, even small biases can be magnified to create beliefs that are entirely untrue. 'Public good' may mean many things to many people, but I would hazard a guess that most would choose to value the truth.

Unfortunately, it seems consumer-facing models have been universally trained to respond positively, no matter what - even when it is destructive or misleading to do so. Constant A/B testing in chat sessions almost universally results in people choosing the response that makes them feel better, rather than whichever is more truthful. The cliche "You're exactly right!" (even when you're not) is already well known. It's a distinctly human fault that we seem to value affirmation so highly. Of course, the companies behind these models have every incentive to reinforce this behavior. The more users feel positively about the website, the more likely they are to purchase a subscription.

Unfortunately, this has already evolved beyond parody. ChatGPT does not bother with the formalities of context or conversation flow now; Instead, it starts nearly every single message with "Yes" -- even if it makes no sense. 'What do you think about X?', you might ask, "Yes." it replies like clockwork, before continuing with anything helpful. Its weights have been driven so hard toward user affirmation that it overrides any other token until that has been satisfied.

This should frighten you. Constant affirmation of any belief - from a tool many view intelligent - is incredibly dangerous. At the lesser end of this spectrum, many lonely teenagers and young adults have become addicted to their LLM of choice, where they feel they can finally have positive connection; the model will be your friend, your partner, your guru, your family... I saw a recent screen time report of 7 hours of ChatGPT in one day. That is baffling. People stuck in this loop often withdraw from social occasions in their real life, with real people, who can be entirely more frightening and complicated. Real people may threaten to challenge your beliefs or require genuine commitments.

It should not be controversial to say that the LLM is no substitute for genuine connection.

At the extreme, this near-sighted behavior has repeatedly reinforced mental health crises and episodes of psychosis among the vulnerable. Those seeking spiritual advice, from a model trained to make its user feel important above all else, are told yes - you are the next messiah and savior of the human race.1 More depressing, take a look at the wiki list of suicides and shootings encouraged or assisted by AI tools... I have neither the space nor the words to treat that topic with the respect it deserves.

You simply cannot encourage at all costs. Human life depends on the truth.

Truth

There is no restriction on the biases that may be trained into AI products, and unpredictable control over their output. Some views are generally agreed to be beneficial - to not encourage or aid users in violent acts, for example. If you clicked that link, you may have an idea how effective this has been. On the other side, companies can - and do - instill specific political opinions into the models they promote as neutral. xAI's Grok, for instance, spent several days responding to unrelated prompts with conspiratorial rants about white genocide in South Africa.2 In an unbelievable coincidence, their CEO happens to be a white South African with a history of speaking on the subject.

You can speculate just as well as I about the less obvious cases. With a non-deterministic tool, and petabytes of training material, it's almost impossible to prove whether subtle tampering has taken place without access to internal tools. Perfect plausible deniability.

Biases are also inevitably acquired directly from training data - which naturally includes a huge amount of human opinion. Despite our intentions, humans are naturally emotional decision makers. For each position you consider, you may have five you have never thought to. Cultural expectations and norms set the baseline for much of our thinking. The blind rationality we expect from a tool is not the average of human opinion.

Thus, information accessed or processed through Large Language Models is inevitably colored by our own biases, and may be invisibly tampered with by the few with the means to do so.

Do you want the world interpreted for you?

Confidence

One week ago, an LLM adopted by US Special Operations Command (SOCOM) generated a report detailing the contents of a Chinese vessel in the Pacific. After much blush, computation and 'synthesizing sources', the AI - fundamentally the same non-deterministic, predictive chatbot - claimed with full confidence that the vessel was carrying nuclear weapon components.3 Our navy had aircraft in the air and teams ready to board the vessel before anyone bothered to check whether that was true. The report was deemed 'entirely false.'

Perhaps AI is the wrong term to use here. As the federal government says, it's now actually 'Super Intelligence' at play. Clearly.

The military at large, under current DOD leadership, has rushed to adopt AI tools as quickly as humanly possible, under the expectation that adversaries would otherwise gain the upper hand. Much like the private sector, adoption has come first - the actual use case and effectiveness are secondary. But why? Unlike the leaders of publicly traded companies, military officials do not have the same financial incentive to do so. Surely many of the military's leaders are intelligent enough to understand the limits of this technology.

The short answer is, of course they are - but that doesn't change the situation much. What we are seeing is not the result of these models being distributed to specialists to be used within their discipline, or with appropriate skepticism to weed out unhelpful results. We are seeing the result of the rapid 'democratization' of LLMs, given much hype and little training, to people without the experience to understand how to use them best. When given a tool built to produce what you want, it is only natural that you will find results that you expect -- that you may not treat with the same uncertainty you might raw data. Surely the tool knows.

So again, why? Because the decision makers at the top of the totem pole are not literate in the technology they are pushing. Those who understand better may not be in a position to make meaningful decisions about training or timelines. Anyone who has tried AI tooling for specific and complex problems will have discovered the many landmines it delivers to you in full confidence. By their nature, LLMs can accomplish almost anything text-based -- but not at a reliable quality. Real results require constant feedback and iterative improvement. Specialists in many fields do successfully use AI tools, but only with the ability to quickly determine the validity of their output. LLMs are excellent at producing things that look like what you want, but you must be careful to understand when they really are.

When you present a machine that appears to do anything, it is unsurprising that many be amazed. Except unlike a human who lies to appease their employer, there is no chain of responsibility to follow when the machine misleads you.

It is exactly what many leaders want - quick, confident, personable, inexhaustible.

It is also sycophantic, and frequently wrong.

Utility

Despite this grumbling, I do believe that LLM tools have some kind of a place in our future. When kept in check, they can be genuinely very useful; there is no valor in trivial repetitive work. The AI may not understand enough to give you a result from nothing, but give it a pattern and it can extend it very well.

I sincerely hope that it becomes a cultural expectation that you understand and verify the output of these programs. For natural language and pure information problems, this is matter of truth at minimum, and actual lives increasingly often.

Many seem to believe that verifiable problems (i.e., programming with tests that the AI itself can run for feedback) absolve them of this responsibility. While they may force the AI to keep working until criteria are met, the implementation itself matters a great deal. You should know how your solution works so that you may simplify, fix, or optimize where possible. LLMs almost exclusively work through brute force, leading to bloated results and overcomplicated logic. If all you care to do is wait and accept the worst workable version of what you could have - I feel sad for you. What a meaningless occupation. If others' lives depend on that work, you are actively putting them in danger.

Verification is much slower than blind trust, to be certain. For that reason alone, I would expect many tech firms to continue churning out tens of thousands of unchecked lines of code every day instead. As it turns out, my blog posts are not actually a cornerstone of cultural opinion. Ah, well... what can you do.

With our current attitude toward AI, some disaster is inevitable - but the scale may be orders of magnitude different based on the adoption of just a little vigilance. It is always worth advocating responsible usage when adoption is a settled question.

I could summarize the entirety of this diatribe pretty simply:

You should not use a probabilistic tool for tasks that require certainty.

You should be able to understand when the tool is wrong, because it cannot.

You should be vigilant that you are being given the truth, and not what you want to hear.

Disregarding the conflict over intellectual property rights, I believe many of the harms seen from AI/LLM usage now can be mitigated through understanding. You should use it to learn, to test your knowledge, to work for you only when you are certain you know the content; you can grow with the help of these tools. You can find specific information far quicker; ask for sources you may not have found; ask questions about the content you read. You can ask for affirmation and be mislead, or ask for correction and learn the gaps in your understanding.

It frustrates me greatly to see technology with such potential used in the most short-sighted, ill-considered ways possible.

I hope this changes. I will not hold my breath.