AI and the quiet erosion of human judgment

In 1986, Paul Simon sang of “days of miracle and wonder” in the song The Boy in the Bubble, from his Graceland album. It was hardly a hymn to progress. Scientific ingenuity coexisted with violence and vulnerability. The world was becoming more technologically capable but not necessarily wiser.

Four decades later, I find the sentiment difficult to improve upon. Since ChatGPT’s release in November 2022, predictions about AI have arrived in rapid succession. Sam Altman, Dario Amodei, Elon Musk and others in the tech industry’s so-called “broligarchy” have fuelled recent headlines about approaching “superintelligence” and futures ranging from human flourishing to catastrophe. The advances are real but the predictions often carry more confidence than evidence. The future, it seems, is permanently scheduled for an upgrade.

Yet a more immediate question risks being lost — as AI systems become more capable and we delegate increasingly complex work to them, will our capacity to understand, question and oversee their outputs keep pace?

That question begins with understanding what we are delegating. AI is not a single technology. Specialized systems have helped scientists predict protein structures. ChatGPT relies on general-purpose language models trained on vast datasets. The most advanced systems are often called “frontier models”. They can write, translate, analyze and converse across subjects, but fluency does not guarantee accuracy. Success at one task does not establish competence at another.

The model is not the whole system. Software can connect it to current information, documents and tools, extending what it can do without changing its training. “Agentic AI” goes further, undertaking sequences of actions on our behalf. These advances increase the volume of work humans may be asked to oversee. Here lies the paradox — we may delegate work to AI to become more efficient, only to discover that the scale of what it produces makes meaningful human oversight increasingly difficult.

Manitoba’s recently passed Bill 51, the Public Sector Artificial Intelligence and Cybersecurity Governance Act, recognizes human oversight as part of responsible AI governance. Yet, as these systems grow more capable, could we lose the very capacity we are trying to protect?

“Keep a human in the loop,” sounds reassuring, but signing off is not understanding when the person responsible lacks the time or evidence to question a result.

Consider a thought experiment: An organization uses AI to analyze thousands of records and prepare hundreds of reports. No one can read them all, so a second AI system summarizes the findings and recommends action. A manager approves the recommendation.

What has actually been reviewed? The evidence, its interpretation and the recommendation have all passed through AI. We risk delegating not only the work, but the grounds on which we form and justify our judgments. Oversight fatigue compounds the problem. As recommendations arrive faster than people can examine them, approval may become routine.

The black box is no longer just a model but a process too extensive to reconstruct. Automated financial trading offers a warning — even before generative AI, transactions occurred faster than people could intervene.

What happens when the conditions required for oversight are undermined by the efficiency gains we seek? If a week’s analysis takes an hour, do we check its assumptions and errors or move to the next project? The more we delegate, the fewer opportunities we may have to practise the judgment needed to supervise it. Dependence can reinforce itself.

The danger is not simply that we may trust machines too much. It is that we may organize our work and institutions in ways that leave us progressively less able to do otherwise.

There are genuine benefits worth protecting. Colleagues with whom I work, particularly in the Global South, see opportunities for AI to extend human capabilities without surrendering authority over their languages, knowledge or priorities. The point is not to reject AI but to adopt it deliberately, define its purpose, test its performance under real conditions, retain the ability to intervene and allow time for review. Slowing adoption where the stakes are high is not resisting progress. AI literacy must likewise mean more than writing prompts. It requires knowing what evidence supports confidence in a system and what must remain open to human challenge.

Simon’s age of wonder was also an age of contradictions. Ours is no different.

If we gain the power to accomplish more while losing the time and capacity to judge what has been done, what kind of progress is that? If we surrender the ability to question these systems, perhaps all that will remain, in the end, is miracle and wonder — amazement at what they can do, and bewilderment at decisions we can no longer explain.

Rod Lastra is an academic specialist in AI and digital innovation at the University of Manitoba’s division of extended education and a former UNESCO AI and digital transformation adviser.