Your money, postcode, friends and family can make all the difference to how the criminal system treats you.

The New South Wales police recently scrapped a widely condemned program known as the Suspect Targeting Management Plan. It used algorithmic risk scores to single out “targets”, some as young as ten years old, for police surveillance.

But similar programs remain in place. For instance, Corrective Services NSW uses a statistical assessment tool called LSI-R to predict whether prisoners will re-offend.

“High-risk” prisoners receive “high-intensity interventions”, and may be denied parole. The risk scores are calculated from facts such as “criminal friends”, family involvement in crime or drugs, financial problems, living in a “high crime neighbourhood” and frequent changes of address.

A predictive algorithm is a set of rules for computers (and sometimes people) to follow, based on patterns in data. Lots has been written about how algorithms discriminate against us, from biased search engines to health databases.

In my newly published book, Artificial Justice, I argue the use of tools that predict our behaviour based on factors like poverty or family background should worry us, too. If we are punished at all, it should be only for what we have done wrong, not for the cards we have been dealt.

Algorithms are watching us

Algorithms generate risk scores used in criminal justice systems all over the world. In the United Kingdom, the OASys (Offender Assessment System) is used as part of the pre-sentence information given to judges – it shapes bail, parole and sentencing decisions. In the United States, a tool known as COMPAS does something similar.

Risk scores are used beyond criminal justice, too, and they don’t always need computers to generate them. A short survey known as the Opioid Risk Tool helps doctors in Australia and across the world decide whether to prescribe pain relief for acute and chronic illness, by predicting whether patients will misuse their medications.

Predictive algorithms literally save lives: they are used to allocate donor organs, triage patients and make urgent medical treatment decisions. But they can also create and sustain unjustified inequalities.

Imagine that we develop an algorithm – “CrimeBuster” – to help police patrol crime “hot spots”. We use data that links crime to areas populated by lower-income families. Since we cannot measure “crime” directly, we instead look at rates of arrest.

Yet the fact that arrest rates are high in these areas may just tell us that police spend more time patrolling them. If there is no justification for this practice of intensive policing, rolling out CrimeBuster would give these prejudices the status of policy.

Algorithms are judging us

The trouble deepens when we use statistics to make predictions about intentional action – the things that we choose to do.

This might be a prediction about whether someone will be a “toxic” employee, commit crimes or abuse drugs.

The factors that influence these predictions are rarely publicised. For the British sentencing algorithm OASys, they include whether someone has been the victim of domestic violence.

The American COMPAS system captures parental divorce and childhood abuse. The Opioid Risk Tool asks whether the patient’s family has a history of substance abuse, and whether the patient (if female) has a history of “preadolescent sexual abuse”.

In each case, these facts make it more likely that someone will go to prison, miss out on medical treatment, and so on.

We all want to have the chance to make choices true to who we are, and meet our needs and goals. And we want to be afforded the same choices as other people, rather than be singled out as incapable of choosing well.

When we punish someone because of facts they can’t easily influence, we do just this: we treat that person as if they simply cannot help but make bad choices.

We can’t lock people up just in case

The problem isn’t the use of algorithms per se. In the 19th century, Italian physician Cesare Lombroso argued we could identify “the born criminal” from physical characteristics – a misshapen skull, wide jaw, long limbs or big ears.

Not long after, British criminologist Charles Goring ran with this idea and argued that certain “defective” mental characteristics made “the fate of imprisonment” inevitable.

Algorithms simply make it much harder to see what’s going on in the world of crime risk assessment.

But when we look, it turns out what’s going on is something pretty similar to the Lombroso-Goring vision: we treat people as if they are fated to do wrong, and lock them up (or keep them locked up) just in case.

Public bodies should be required to publish the facts that inform the predictions behind such decisions. Machine learning should only be used if and to the extent that these publication requirements can be met. This makes it easier to have meaningful conversations about where to draw the line.

In the context of criminal justice, that line is clear. We should only deal out harsher penalties for bad behaviour, not other physical, mental or social characteristics. There are plenty of guidelines that take this approach, and this is the line that Australian institutions should toe.

Once penalties for their crime have been applied, prisoners should not be treated differently or locked up for longer because of their friends and family, their financial status or the way in which they’ve been treated at the hands of others.