Algorithmic decision-making
The use of Automated Decision Systems (ADS) and mathematical models to automate conclusions, recommendations, or actions that affect human lives, often reducing the role of human oversight.
Critical Dimensions
- Algorithmic Bias: Systematic and unfair discrimination resulting from biased training data or model design.
- Transparency and Explainability: The “Black Box” problem, where the logic behind a decision is inaccessible to users or regulators.
- Algorithmic Accountability: The difficulty in assigning legal or ethical responsibility for errors or harms caused by automated processes.
- Automation Bias: The human tendency to over-rely on or trust automated suggestions, even when they are erroneous. This cognitive bias is critical in high-stakes environments like aviation, where human operators may defer to AI systems despite contradictory evidence.
Case Studies & Failures
- Robodebt Scheme: Australia’s unlawful automated debt recovery program.
- Air India Crash: Human Responsibility and Evolving AI Cognition in Aviation: Analysis of the Air India crash highlights the tension between human responsibility and evolving AI cognition. The incident underscores how automation bias can lead to catastrophic failures when human oversight is compromised by over-trust in automated systems.