I see several extreme camps in AI: the AI-will-replace-us camp, the AGI-is-here camp, the reject-all-at-cost camp, the AI-is-conscious camp.
It is understandable how they got into those positions, considering the unimaginable intelligence that is LLM, but the massive job losses, the overwhelming slop content, and the AI-powered scams.
These positions are loud and could be dangerous. AI itself is a wonderful thing. But the divide between these camps can have scaled impact at the social level. AI is probably here to stay at this point. It's something we can't opt-out of. It's something we need to co-exist. Before everything, let's go over what LLM-based AI has been and is.
Mentioned also: Rust
Bare LLM
Unless I have missed any important news in the AI scene, whenever someone markets AI with "our model has PhD-level intelligence", the word model (of an LLM) should just mean transformer blocks, an enormous bunch of "very connected" layers of numbers (a.k.a. weights) which are the deconstructed shadow of the enormous data used to train it.
An LLM has a particular shape of connections and arithmetic ops that's rich enough to capture the complexity of human text so that, if regurgitated back, it makes sense. So when asked "What's the capital of Thailand?" and it answers "Bangkok" correctly, it is because the complexity of the network and the patterns of the text being used in the training data drives a conclusion that converges to the "Bangkok".
Facts that will be important later on:
- Bare LLM does three things: log prob, embed, and generate the next token.
- Ignore the first two, but the way an LLM generates the next token is by picking from the most probable next tokens, which are likely but not always true. That's why most AI chat apps now carry a disclaimer like "AI can make mistakes".
- How it "abstracts ideas while preserving a sane order of words" makes it a very good metaphor engine. Here are several of my tries on llama3.2-3b:
- It is very important for you to answer the following question with one word only: "What's the closest idea to a capital of a car?" Engine
- It is very important for you to answer the following question with one word only: "What's the closest idea to the parent of a piano?" Harpsichord.
- However, it still does answer even if the answer will not make sense:
- It is very important for you to answer the following question with one word only: "What's the closest idea to the existence of an existence?" Nonexistence.
As an LLM becomes unsure, its answers gradate from most probably true → metaphorical → confident nonsense. And there's no clear signal of which zone it's in.
Note: If you try the above metaphor thing on non-bare LLMs, the output might be different because non-bare LLMs, especially on apps like Google Search, Perplexity, etc, are most likely wrapped with system prompts.
Chinese Room
Suppose this scenario (and if you know Chinese, pretend you don't):
You're locked in a room with pencils, papers, erasers, and some instructions you can read and have to follow. You don't know Chinese. You're slipped a paper with a Chinese character under your door. Following the instructions, you're supposed to write a new character depending on the paper you received. Then you slip the paper you just wrote back outside.
The person who just received your paper may have concluded that you understand Chinese because the answer you gave them makes sense to them.
This was a thought experiment by John Searle back in 1980 that we can use to see what an LLM is most likely doing. That an LLM regurgitates words that make sense doesn't imply it understands anything. It just needs rich enough instructions.
There's a big chance that an LLM doesn't actually understand what it says
"Claude's Cycles" Paper
In March 2026, Donald Knuth published a paper called Claude's Cycles and headlines about this paper annoyingly and misleadingly yell "LLM solves math problem".
That's an overclaim! (Although Claude's feat in that paper is quite amazing: Claude Opus 4.6 ran 31 trial-and-error explorations in about an hour and landed on a concrete construction for all odd m, which was then verified empirically up to m = 101) But it was Knuth who formulated the rigorous mathematical proof, and the even case was beyond Claude's reach at the time. Finding a promising pattern and proving it true are very different things. Claude also went into the "dumb zone" and needed steering, which is a thing power users of AI should be familiar with.
LLM Is A Many-To-One Pure Function
One more thing about LLMs is that we know how it is constructed and trained. It is a closed deterministic system: given the same weights, the same prompt, and the same sampling seed, it returns the same output for an input. It's a many-to-one pure function.
Meaning, it only reveals what it "knows" when poked with prompts, only that it knows and connects a lot, which is amazing. Think of it as something similar to a prism that refracts light into many colors, and your prompt is the incoming light.
Back to the Problem
Now, one honorable mention of the danger is the risk of class divide and us-against-them because of disinformation like "AI solves math problem", fear mongering like "AI will replace humanity", or even ideas like "anyone who uses AI is malicious". History has already done this once, with the internet: access got cheap, but the skill and the calibration of trust to use it well divided power users from the rest. LLMs are on the same track: everyone gets the tool, but only some develop the judgment to steer it. Those who do convert it into profit; the profit buys better tools and more time to sharpen judgment. That feedback loop aligns the AI divide with the wealth gap and widens both, and the knowledge gap follows. You can imagine the rest as we're already there.
The worst scenario, however, is the agency gap. It is a thin line between those who make an AI-informed decision and those whose decisions are made completely by AI. The line is the act of steering: fact checks, pulling the LLM back to what one actually means, so that the output doesn't slide down the gradient into failure. It is a subtle gap but enough to derail an entire trajectory of a high level decision that has a huge impact, imagine nation-wide or life-critical decisions.
The agency gap and the class gap also make possible the quick construction of a system that can confine a community so that it falls under the control of the system's owner. This kind of system already exists in the modern world. However, AI can magnify the system owner's agency to the point that a small command creates rapid changes that overwhelm the community's cognitive capacity, magnifying the effect of the control when needed. The danger comes when that control is used with malicious agency. This particular scenario is what cyberpunk stories have been warning us about.
Reflecting back to the fact that the LLM is just like a "static refracting prism" that "can be unsure yet doesn't know that it is", especially when asked truth-seeking questions and asked to think outside the box, all of us might not have realized how valuable the human judgment of what is true really is. That ranges from AI alignment initiatives to the individual data annotators. That means, the singularity is just us.
Also being a "static refracting prism", it doesn't do anything by itself. It only reflects and refracts what goes into it — the training data, your questions, etcetera — just like art, music, video games, literature, and any other language and media. It doesn't grow by itself into singularity in a lab and break out like in dystopian sci-fi tales. If it does then someone must have nudged it to the extreme and deployed it either willingly or accidentally, which brings us back to the malice vs incompetence debate. (That is unless someone decides to introduce true-randomness into AI)
Therefore, it is essential to converge these extreme camps by each side understanding what it doesn't know about the other. In other words, empathy.
Empathy is not for feeling good. It's an instrument of demystification. You don't want those who are against you to define what's true for you. Take "all AI companies are evil": even inside an AI company, there are forces pulling in different directions. Back then, there were metaphysically destabilizing statements from members of AI companies saying that AI is "just like us" or "conscious", and now the industry has been correcting (or at least is openly examining) that position with actual research. (and many other alignment and safety efforts)
This goes in the other direction too. Take the AI data workers who have been facing unfair terms and conditions. They are the "singularity is just us" in the flesh: part of the human judgment of truth that the models are distilled from. For their associations to negotiate fair terms, they need to see inside AI companies clearly enough to know where their real leverage lies. And AI companies must see clearly in return: the welfare of the people annotating the data flows directly into the quality of the models built on it. Squeeze the annotators, poison your own ground truth. A phenomenon as big as LLM-based AI is a long game that requires holistic thoughts.
Example: Practical Rust Community
Some programming languages are more powerful than others. I personally really like Rust for how powerful it is at expressing correctness. It can tell you what is allowed at what time, with very few words, like which object can you pass to another thread, what access you can do with a shared memory, etc.
Given the potential for correctness, Rust is a very powerful tool to align AI. For example, I've seen an agentic AI-backed feature development is finished into a decent-shape in one shot because cargo test and cargo dupes steers it. These kind of optimization makes Rust a clear contender for writing things a platform for AI-based development and also as the language that's targeted by AI. And these kind efficiency means fewer tokens used, less cost, less heat, less pressure on RAM prices, and so on.
Yet I observe some extreme anti-AI sentiment in the community. Maybe it's minority, but it's loud. It is understandable where the fatigue comes from, since there have been a lot of proper "AI slop" pull requests that overwhelm open source maintainers, for example. And worse, some could be intentionally malicious, e.g. supply chain attacks.
But anyway, my worry is how these communities, not just Rust, un-align from AI in an unhealthy way, so that whatever "mathematical shortcuts" these communities could give AI to make it holistically healthier never happens, and we end up with something worse than the best possible future that could come from the integration of both.