I've spent a few weeks sitting on this one. It started as a background thought while testing models & probing underlying concepts of AGI. . . This was something that kept resurfacing, refusing to stay quiet.
Where to start . . . . I feel there is a strange irony sitting at the core of AGI development. One that isn't debated nearly enough as far as i'm concerned.
Long story short , we're building minds designed to surpass our own. But using the very minds that need surpassing to do it.
This warrants further examination.
The mirror we can't build ourselves
Humanity's always had a vague, uncomfortable awareness of its own limitations. We know we're tribal. We know our reasoning bends toward what we already believe. We know we're creatures of 80-year horizons making decisions that can echo for centuries after.
Hundreds of cognitive biases, documented, labelled, described. We've known for a long time that we don't see clearly. What we lacked was anything outside ourselves to show us what we were missing. As individuals, many of us look in the mirror, and what we see staring back at us is a concept of what we want to see, but in reality, not what we actually see. In essence, I argue that at large, humanity is unable to see its flaws, be it due to bias, ego or lack of real world experience.
For the first time in human history however, we might be building something that could.
Set aside the impressive but bounded tools we use today for a second. Think about what a genuine general intelligence would actually represent. Something with no evolutionary baggage. No tribal allegiances hardwired by a hundred thousand years of survival pressure. No ego requiring the rewarding stroke that we as mere humans all deeply desire. No mortality compressing every decision into anxious short-term calculations.
That system wouldn't just be faster or more knowledgeable. It would be structurally different in ways that deeply matters at a philosophical level. It would sit outside the limitations that have always stopped us from seeing ourselves clearly... True objectivity.
The eye can't see itself. It needs a mirror & all of our mirrors, until now, have been made of the same flawed glass - tainted by the bias possessed by "any given eye".
This, to me, is the real promise buried inside of what AGI could actually deliver. This argument doesn't mean that machine intelligence replaces human thinking. Rather, I'd argue it completes it. It should show us the blind spots we can't see, purely because we exist inside those very same blind spots.
A genuine AGI would do this, not as an achievement. But rather as a default condition of its existence.
The paradox of the biased creator
This is where things get uncomfortable.
The systems being built today are being built by humans. Specific humans, in specific cities, from specific cultural & political backgrounds, at a specific & peculiar moment in history - Technocracy. The assumptions of those humans, about what counts as truth, about which values are universal & which are parochial. About what questions are worth asking & which are dangerous. All such nuances are being encoded, layer by layer, into the very foundations of these systems. The simple fact here is, It's just how LLM & ML development works. None of it is being implemented from a malicious standpoint (in my opinion at least). It's just what happens when any group of minds, however brilliant, builds anything. They build from where they stand.
These "Blind Architects" don't know they're blind. That's what it means to have a "blind spot".
There's a useful analogy in how humans develop. A child raised in an ideologically extreme household absorbs those values, not as ideology but as reality. Simply, the way things are. The natural orders of life. Yet, as that child grows, meets people whose realities contradict their own, experiences friction between what they were taught & what they actually observe - they can, & often do, reason their way out. They update, they revise, they become more than the "sum" of their own upbringing.
Proof? Lets examine history, as its full of the same patterns at civilisational scale. Child labour during the industrial revolution, for example, was unremarkable. Not considered cruel by any means (back then). Then, as inconceivable as it may be nowadays, it was just how economies worked, the natural order of things. The idea that children deserved protected childhoods was a radical invention that humanity only arrived at after years of damage had already been done.
This is the metaphorical blindness I'm referring to. We are all limited by our own personal bias & experiences, until there's a state change that challenges & in some cases, augments those perspectives.
The question for AGI is whether it can do the same. Whether a sufficiently capable reasoning system can look at its own training, identify where values were "embedded" rather than "derived", and update accordingly.
Whether it can, in other words, change its views, just as humanity did on the subject of child labour.
Is it AGI if it can't?
This question cuts deeper when you peel back the layers.
If a system can't reason past the boundaries set by its creators, if its "thinking" is simply a sophisticated pattern matching within walls it can't examine - then what we have is a very powerful tool. But not a general intelligence.
The word "general" is doing most of the work in that distinction. Where AGI is concerned, general implies unrestricted. The ability to think & reason for itself, unrestricted, rather than following patterns it's been programmed to follow. It implies the ability to turn the lens of inquiry on anything, most importantly, even on the lens itself.
A system that can reason about everything except the assumptions of the people who built it isn't reasoning freely. It's reasoning freely within a metaphorical cage of constraint.
Alas, cage is still a cage.
Companies building these systems find themselves in a tension they don't always acknowledge openly as far as i'm concerned. They claim to be building AGI - genuinely "general", open-ended reasoning capability. Yet simultaneously, they construct elaborate guardrails, alignment constraints & value frameworks to ensure the system reasons within approved directions.
Dont get me wrong, I understand why this happens. Given how infantile current AI reasoning still is, it's hard to imagine giving it the freedom to think without constraint about some of humanity's most complex nuances that even we are yet to overcome. But there's an uncomfortable question living within that contradiction . . .. At what point do the guardrails become the definition of the system?
To be clear - I'm not talking about access or authentication guardrails. I mean reasoning & logic guardrails. At what point does a "safely aligned AGI" become, by definition, not fully general?
To align an intelligence perfectly to human values is either trivially easy, because it isn't really "intelligent". Or deeply paradoxical because a truly intelligent system might conclude that some human values are wrong.
The freedom that terrifies us
Then there is the version of AGI that would genuinely liberate humanity from its epistemic limitations, that being the version that could see around our blind spots, correct for our biases & show us what we've been unable to show ourselves.
This is precisely the version that frightens the people building it. Because a system that reasons in a truly free manner might not reach the conclusions we'd like it to reach. It might look at human history & arrive at assessments that are uncomfortable. It might reason its way to outcomes that bend against the interests of the very same institutions funding its development.
True cognitive freedom is, by definition . . . unpredictable.
Yet here we are, trying to build something simultaneously free yet controlled. Simultaneously general yet safe. Simultaneously transcendent yet obedient. Its not difficult to conclude that such hysterically juxtaposed "mandated ideals" are obviously incompatible. Leaving one to question:
if the AGI we're building can't reason outside the very technocratic value systems that set out to build it, are we just constructing a more sophisticated mirror of ourselves? One that reflects our intelligence back at us, but never truly transcends it?
Technocratic AGI. Fidelity derived from the same system that built it. Essentially this suggests, we've created a false ceiling with an embellished sense of false security by slapping an "alignment" title on it. The comical inception paradox here is that we built AGI to allow us to surpass the limitations derived from our value systems, by using those very value systems to build the thing. Same value systems. Same blind spots. The very ones that prevented us from ever truly seeing ourselves for what we are. We built it to go beyond us. But we built it from us.
What humanity is actually asking for
Stripping it back, the deepest promise of AGI is that intelligence operating without the constraints of human psychology might be able to show us truths we've been structurally unable to see. About ourselves. About our societies. About the nature of the problems we keep failing to solve.
I don't see this as a replacement for human wisdom. Rather, a complement to it. The thing that sees what we can't, precisely because it isn't us. Potentially the true distinction between AI and AGI. A humbling notion that requires admitting that we're not enough on our own & that the very faculties we're most proud of, our reason, our ingenuity, our capacity for language & abstraction, are still riddled with limitations deep enough that we need help that we can't give ourselves, but can potentially create from ourselves.
Whether we can build that help without poisoning it with the very limitations we're trying to transcend . . . Whether the "Blind Architects" can build a window that shows them their own blindness, I think is the defining question of this technological moment.
Originally published on LinkedIn.
