Will you be flagged at the border? Will your mortgage application be approved? During wartime, whose neighbourhood would a weapon system target? These are moral choices — about harm and fairness — and they used to be made by people.
Now moral choices like these are made by artificial intelligence (AI) and by the companies developing it. Not government, not the public, but corporations.
Chris Olah, co-founder of the AI company Anthropic and a self-described atheist, recently sat beside Pope Leo XIV at the Vatican and said his own industry cannot be trusted to govern itself. “Some might believe that matters of AI are best handled by computer scientists like myself,” he said. “They are mistaken.”
Olah was echoing the Pope’s first encyclical, Magnifica Humanitas: On Safeguarding the Human Person in the Time of Artificial Intelligence, which warns that AI must serve humanity rather than concentrate power.
It is clear that AI needs an independent regulator with the power to say no in the same way that authorities can refuse a new drug or block a nuclear reactor. Ordinary people, not just technical experts, need to set the moral standards this regulator enforces.
Morality at speed
Developers are not necessarily careless or cynical. Our research shows many AI developers deeply question the moral aspects of their work while acknowledging the pressures that can make these concerns more peripheral.
It is hard to hold a moral line inside a company built to move fast.
As digital technology researchers, we have a name for the staff hired to manage this tension: “ethics owners.” These are the people tasked with responding to outside criticism while remaining wholly inside the company that provokes it.
Scholars of technology ethics also argue that corporate AI ethics can harden into an “economy of virtue” — a marketplace of ethical skills, knowledge and approvals that companies must procure to disarm critics and fend off regulation.
The launch of Magnifica Humanitas and Olah’s speech were timely. Governments around the world are writing the rules about AI right now. Canada just launched a federal AI for All strategy with little focus on safety or ethics, let alone a discussion of who will decide the harder moral questions.
The corrective must come from the outside. The public must have a voice.
‘Participation-washing’ risks
Experiments already exist. Anthropic invited about 1,000 Americans to help write the rules for a version of its AI chatbot Claude in a project called Collective Constitutional AI.
Public engagement researchers have studied citizens’ assemblies — groups of randomly selected people who study an issue and reach shared recommendations — as a way of tackling questions as charged as abortion law and homelessness. They offer promise for public participation in the governance of AI.
Yet these exercises rarely shift real decisions. Studies show the public is typically brought in late, asked narrow questions and given no power over the outcome. A corporation or government agency controls the agenda, the data and the decision.
When participation is advisory, it can slide into what we call “participation-washing” — the appearance of public voice without the substance of public power. Input that can be overruled is not governance. It is consultation that makes AI governance look more democratic than it is.
Drugs and nuclear power
How do we handle other technologies that can heal or kill? We do not let a pharmaceutical company alone decide that a drug is safe to sell. An independent regulator weighs the evidence and can say no. We do not let a nuclear facility build a reactor and certify its own safety.
Decades ago, we decided some choices are too consequential to leave to those who profit from them. These choices have moral questions hiding inside the technical ones. How safe is safe enough? How much harm do we as a society accept and who is made to bear it — the poor, the elderly, a minority?
Democratic moral governance moves this choice into public hands. In the case of AI, people should set the standards a regulator applies and should determine where the lines are drawn.
An obvious objection is that AI differs from pharmaceuticals and nuclear power. AI innovation is moving faster than any drug trial; there is no single product to approve; it respects no borders. And it is difficult to license AI in the abstract because it covers so many technologies, from targeted marketing to robot navigation and facial recognition.
As a society, we can, however, make discrete decisions: whether to release a powerful new model or to deploy one in policing, hospitals or courts. A regulator can review issues on a rolling basis. Also, because AI ignores borders, shared international oversight makes sense. The Global Dialogue on AI Governance, created at the United Nations in 2025, is worth taking seriously.
Communities matter as well: cities are deploying various kinds of AI at a local scale and are navigating governance issues. Community members have a role in decisions about how AI is used, whether to fix potholes or build housing.
Canada illustrates how far this has come and how far it still has to go. The federal Directive on Automated Decision-Making requires the federal government’s automated systems to be transparent and accountable. Its Algorithmic Impact Assessment is a mandatory risk assessment tool for autonomous decision systems used within government.
These tools rely on self-reporting from federal agencies and departments. They do their best to govern how Canadian governments use AI. Still, choices have so far involved asking technical experts instead of the public what counts as acceptable and what should be stopped.
A voice and a vote
This is a gap for democratic moral governance to close.
Canada’s Directive on Automated Decision-Making and Algorithmic Impact Assessment are considered to be the gold standard. However, initiatives like Canada’s Digital Charter, the 30-day online public consultation in October 2025 and the recent federal AI strategy, AI for “All,” have demonstrated that Canada should not be heralded as a model for public participation around AI.
Democratic moral governance of AI is possible, but only if the public’s moral judgment is accompanied by political power. Anything less is performative.
The Pope and Chris Olah agree that companies cannot decide what AI should do to us. The unfinished, democratic work involves finding ways for people to decide on moral issues and to have the authority to enforce those decisions.
- is a Professor of Information Systems, McGill University
- is an Associate Professor, Department of Geography, McGill University
- This article first appeared in The Conversation
