AI Act: EU must ban predictive AI systems in policing and criminal justice

Article by Fair Trials

Fair Trials, European Digital Rights (EDRi) and 49 other civil society organisations launched a collective statement to call on the EU to ban predictive policing systems in the Artificial Intelligence Act (AIA).

Artificial intelligence (AI) systems are increasingly used by European law enforcement and criminal justice authorities to profile people and areas, predict supposed future criminal behaviour or occurrence of crime, and assess the alleged ‘risk’ of offending or criminality in the future.

However, these systems have been demonstrated to disproportionately target and discriminate against marginalised groups, infringe on liberty, fair trial rights and the presumption of innocence, and reinforce structural discrimination and power imbalances.

Griff Ferris, Legal and Policy Officer, Fair Trials stated:

“Age-old discrimination is being hard-wired into new age technologies in the form of predictive and profiling AI systems used by law enforcement and criminal justice authorities. Seeking to predict people’s future behaviour and punish them for it is completely incompatible with the fundamental right to be presumed innocent until proven guilty. The only way to protect people from these harms and other fundamental rights infringements is to prohibit their use.”

Fundamental harms are already being caused by predictive, profiling and risk assessment AI systems in the EU, including;

Discrimination, surveillance and over-policing

Predictive AI systems reproduce and reinforce discrimination on grounds including but not limited to: racial and ethnic origin, socio-economic and work status, disability, migration status and nationality. The data used to create, train and operate AI systems is often reflective of historical, systemic, institutional and societal discrimination which result in racialised people, communities and geographic areas being over-policed and disproportionately surveilled, questioned, detained and imprisoned.

Infringement of the right to liberty, the right to a fair trial and the presumption of innocence

Individuals, groups and locations are targeted and profiled as criminal. This results in serious criminal justice and civil outcomes and punishments, including deprivations of liberty, before they have carried out the alleged act.  The outputs of these systems are not reliable evidence of actual or prospective criminal activity and should never be used as justification for any law enforcement action.

Lack of transparency, accountability and the right to an effective remedy

AI systems used by law enforcement often have technological or commercial barriers that prevent effective and meaningful scrutiny, transparency, and accountability. It is crucial that individuals affected by these systems’ decisions are aware of their use and have clear and effective routes to challenge it.

Sarah Chander, Senior Policy Adviser, EDRi stated:

“AI for predictive policing is reinforcing racial profiling and compromising the rights of over-policed communities. The EU’s AI Act must prevent this with a prohibition on predictive policing systems.”

The signatories call for a full prohibition of predictive and profiling AI systems in law enforcement and criminal justice in the Artificial Intelligence Act. Such systems amount to an unacceptable risk and therefore must be included as a ‘prohibited AI practice’ in Article 5 of the AIA.

Read the statement here.

Signed by:

  1. Fair Trials (International)
  2. European Digital Rights (EDRi) (Europe)
  3. Access Now (International)
  4. AlgoRace (Spain)
  5. AlgoRights (Spain)
  6. AlgorithmWatch (Europe)
  7. Amnesty Tech (International)
  8. Antigone (Italy)
  9. Big Brother Watch (UK)
  10. Bits of Freedom (Netherlands)
  11. Bulgarian Helsinki Committee (Bulgaria)
  12. Centre for European Constitutional Law – Themistokles and Dimitris Tsatsos Foundation (Europe)
  13. Citizen D / Državljan D (Slovenia)
  14. Civil Rights Defenders (Sweden)
  15. Controle Alt Delete (Netherlands)
  16. Council of Bars and Law Societies of Europe (CCBE) (Europe)
  17. Czech League for Human Rights (Czech Republic)
  18. De Moeder is de Sleutel (Netherlands)
  19. Digital Fems (Spain)
  20. Electronic Frontier Norway (Norway)
  21. European Centre for Not-for-Profit Law (ECNL) (Europe)
  22. European Criminal Bar Association (ECBA) (Europe)
  23. European Disability Forum (EDF) (Europe)
  24. European Network Against Racism (ENAR) (Europe)
  25. European Sex Workers Alliance (ESWA) (Europe)
  26. Equinox Initiative for Racial Justice (Europe)
  27. Equipo de Implementación España Decenio Internacional Personas Afrodescendientes (Spain)
  28. Eticas Foundation (Europe)
  29. Fundación Secretariado Gitano (Europe)
  30. Ghett’Up (France)
  31. Greek Helsinki Monitor (Greece)
  32. Hackney Account (UK)
  33. Helsinki Foundation for Human Rights (Poland)
  34. Homo Digitalis (Greece)
  35. Human Rights Watch (International)
  36. International Committee of Jurists (International)
  37. Irish Council for Civil Liberties (Ireland)
  38. Iuridicum Remedium (IuRe) (Czech Republic)
  39. Liga voor Mensenrechten (Belgium)
  40. Ligue des Droits Humains (Belgium)
  41. La Quadrature du Net (France)
  42. Novact (Spain)
  43. Observatorio de Derechos Humanos y Empresas en la Mediterránea (ODHE) (Europe)
  44. Open Society European Policy Institute (Europe)
  45. Panoptykon Foundation (Poland)
  46. PICUM (Europe)
  47. Racial Justice Network (UK)
  48. Refugee Law Lab (Canada)
  49. Rights International Spain (Spain)
  50. SHARE Foundation (Serbia)
  51. Statewatch (Europe)
  52. ZARA – Zivilcourage und Anti-Rassismus-Arbeit (Austria)