WHEN CODE GOVERNS LIVES: ALGORITHMIC JUSTICE AND INEQUALITY IN INDIA

HIMAANI, STUDENT, B.A LLB , NATIONAL UNIVERSITY OF STUDY AND RESEARCH IN LAW, RANCHI

Abstract

The rapid integration of artificial intelligence and algorithmic decision-making systems into governance and commerce has created unprecedented challenges for the protection of constitutional rights in India. This paper examines the intersection of algorithmic governance and fundamental rights under the Indian Constitution, with particular emphasis on Articles 14, 15, and 21.⁶ The study argues that algorithmic bias is not merely a technical problem but a structural one, deeply rooted in India's social hierarchies, including caste, gender, religion, and class.¹ Recent research demonstrates that popular AI models reproduce harmful stereotypes about Indian castes and marginalised communities, threatening to automate and amplify existing social inequalities.⁷ The paper analyses how current legal frameworks, including the Digital Personal Data Protection Act 2023 and the AI Governance Guidelines 2025, address these challenges while identifying critical gaps in enforcement mechanisms.⁸

It explores how constitutional guarantees of equality and non-discrimination must be reimagined to account for indirect algorithmic discrimination and the opacity of automated decision-making.⁶ The study draws on comparative analysis of global approaches, particularly the European Union's risk-based regulatory model, while emphasising the need for India-specific solutions that reflect the country's unique social context.⁹ The paper proposes a comprehensive framework incorporating mandatory algorithmic audits, diverse training datasets, transparency requirements, and strengthened judicial oversight.⁵ It argues that effective regulation requires ongoing institutional capacity rather than one-time legislation, demanding specialised bodies capable of monitoring AI systems throughout their lifecycle.This research contributes to the growing discourse on digital constitutionalism in India, bridging technology policy and constitutional law to ensure algorithmic systems operate within constitutional limits while promoting substantive equality.

1. Introduction

Digital systems now control major parts of Indian life. Aadhaar determines who gets welfare benefits. Algorithms decide loan approvals. Police departments use software to predict crimes. These automated systems are based on the premisethat computers can make fair and objective decisions. But this idea ignores a harsh reality. The data fed into these systems carries inherited biases, and the technology itself reflects cultural assumptions.¹

Two questions become critical here. When an algorithm denies someone their rights, who takes responsibility?² How do we protect vulnerable people while still using technology for efficiency? Indian law has not caught up with these problems yet.

2.The Problem of Algorithmic Bias in India

2.1Data Distortions and Representational Failures

Training data in India has serious representation problems. Most models work best for middle-class men who are digitally active.² Half of the country does not even have internet access. These people simply do not exist in the datasets that train algorithms making decisions about their lives.¹

Consider the datasets that train most AI systems globally. ImageNet and Open Images are two massive image libraries used worldwide. Researchers found that 60 per cent of their images show European and American contexts. Another 32 per cent specifically come from the US. India, with its 1.4 billion people, appears in just 2 per cent of these images. When Indian developers tried to fix this model through Staqu, they trained the model on Bollywood celebrities and regional film stars. This led to even more bias, as most of the celebrities are fair-skinned and able-bodied. So, the bias just shifted the form without resolving the issue.²

2.2 Sources of Bias

Bias enters algorithms through three main doors. First, historical data reflects past discrimination. Amazon built a hiring tool that had to be removed because it favoured men over women. The algorithm learned from years of hiring records, where companies chose men more often. It treated that pattern as correct.¹

Secondly, programmers have their own blind spots. The developers make choices based on their experience and worldview. These choices shape how the algorithm makes decisions. The assumption that awareness alone can mitigate bias is a fundamentally flawed premise; most cognitive biases operate at the subconscious level and influence decision-making even before it reaches the threshold of conscious deliberation.²

Furthermore, there is a phenomenon commonly referred to asthe black box problem. In certain instances,the algorithm produces biased results despite being trained on unbiased data.² The programmer cannot determine how the system arrived at that conclusion. The decision-making process remains concealedwithin complex layers of computation that even domain experts struggle to understand. 

2.3 Indian Context and Discriminatory Proxies

Western fairness models focus on race and gender as the primary categories for analysis. However, these categories fail to capture the nuanced social structure that defines India.Caste hierarchies, religion, tribes, and linguistic communities operate through distinct mechanisms and carry significant societal implications.Standard algorithms lack the understanding to detect or address these dimensions properly.²

In the Indian context, names act as a marker of social identity to reveal caste and religion. The surname Banerjee signifies that the individual belongsBengali upper caste community. Similarly, Khan indicates Muslim identity. An algorithm trained on an Indian dataset inevitably replicates these identity signals.²

Additionally, occupation still connects to caste for many traditional jobs. Manual scavenging and butchery remain associated with specific communities. What someone buys can indicate religion - beef or pork purchases mark religious identity. How far women travel reveals gender inequality. Indian women use public transport more and travel shorter distances because of safety concerns and economic constraints.²

Language creates its own barriers. Only 10 per cent of Indians understand English well. Yet many AI systems operate mainly in English. India has 30 languages spoken by over a million people each. Caste-based slurs fill online spaces, but content filters often miss them because they are trained on English-language hate speech.

3. Real-World Consequences

3.1 Aadhaar and Welfare Exclusion

Aadhaar was meant to reduce corruption in welfare distribution. Instead, it has excluded the very people it should serve. Santoshi Kumari died of starvation after her family's ration card was cancelled for not linking with Aadhaar. Shrimati Devi lost her Rs. 1000 monthly pension when a banking error transferred it to someone else. This violated her right to life under Article 21 of the Constitution.³

These are not isolated incidents. The computerised screening system catches genuine beneficiaries in its net while trying to stop fraud.The system gives primacy to urban upper-caste groups who have better access to documentation and biometric enrolmentcentres.²

3.2 Predictive Policing and Facial Recognition

In Maharashtra and Delhi, police have begun using predictive models. These models study past crime data to identify high-risk locations and individuals.²However, the main problem lies in the data itself. Historical records indicate more arrests in Dalit and Muslim neighbourhoods, as these areas always faced heavier policing. So, the algorithmexhibits the same pattern, leading to increased surveillance of the same community.²

Reports highlight that facial recognition software fails to identify darker skin tones. If the algorithm itself misidentifies someone, then there is a high possibility of false arrest.¹ The presumption of innocence means nothing if an algorithm marks you as a suspect. Hence, the AI make the inequality widespread instead of reducing it.²

4. Legal and Regulatory Gaps

4.1 Inadequacy of Existing Framework

The Digital Personal Data Protection Act of 2023 focus entirely on privacy. It says nothing about algorithmic fairness. The legislation doesn’t require companies to audit their system. The Act even exempts publicly available data from scrutiny, even though such data embeds most of the historical prejudices.³

Compare this to the EU AI Act, which requires risk assessments and algorithmic audits for high-stakes systems. California is also proposing anti-discrimination provisions that are specifically tailored for AI. Meanwhile, India has not implemented either approach. The Economic Advisory Council suggested a Complex Adaptive System Framework for AI governance, yet it remains theoretical without any legal enforcement mechanism. 

Currently, no legal mechanism exists to challenge a biased algorithmic decision. If a loan app rejects you because of your zip code or name, you have no recourse. The company does not have to explain the decision. You cannot demand an audit of their algorithm.⁴

4.2 The Need for Transparency and Accountability

Fintech startups give instant loans based on automated scoring, Recruitment platforms filter resumes through algorithms, and police departments deploy facial recognition, and none of these systems faces meaningful regulation.⁴

European regulation bans fully automated decision-making systems wherein fundamental rights are involved. The question for India is whether to follow suit. At the very least, human oversight must be integrated into these algorithm processes. Someone should be able to explain the reasoning behind the algorithm's decision. Most importantly, there should be accountability – a designated individual or entity that bears the responsibility when AI produce bias result.

5. Towards Algorithmic Justice in India

5.1 Contextualising Fairness

Western fairness frameworks do not fit India. Dr B.R. Ambedkar's social justice vision was a direct result of the anti-caste movement and is based on a premise that is radically different from the distributive justice theories that inform much of the modern computer science literature on algorithm fairness.¹Addressing algorithmic bias in India requires a contextual interpretation of caste and patriarchy. These power structures don’t operate in isolation; these entities support each other in an interdependent manner. A system can, therefore, seem to be adequate as per the Western standard, and at the same time, be embedded with oppression against the marginalised groups of India.²

5.2 Regulatory Recommendations

India needs to establish mandatory dataset audits before any algorithm is implemented in any critical sector.⁵ Theregulatory sandbox of the European Unionprovides a concrete example: test the algorithm against real Indian social inequality before adopting it.The government should disaggregate their data collection by gender, caste, and region, as this enables the creation of a representative dataset. Regular re-evaluation of the algorithm is essential as the changing conditions in society may shift the trajectory of their influence.⁴

5.3 Collaborative Governance

The best way to govern AI is through a multifaceted coalition of stakeholders: civil society organisations know the ground realities, academics have technical expertise that is necessary for scrutinising AI, and the affected population have lived experience to identify any shortcomings in practice.⁴

The objective isn’t to prohibit the algorithm system. However, it must uphold constitutional values and expand opportunities for marginalised groups instead of enhancing the exclusionary processes.²

Conclusion

Ifa computer system denies welfare, determines criminal risk, or blocks employment, then it’s the duty of the government to ensure that constitutional rights remain protected.⁴

India needs to develop its own approach that fits its unique challenges and circumstances,instead of copying the Western model.¹ It’s possible to achieve both technological progress and social justice if a decisive step is taken while the algorithms are still young and vulnerable.⁵Once an algorithm is applied in government and commerce, it’s difficult to change the course.

The path forward requires coordinated interventions rather than isolated reforms. Regular audits of AI systems must become mandatory across public institutions, with independent experts examining these systems throughout their lifecycle. Public institutions need to collect disaggregated data across gender, caste, and region to build representative datasets reflecting India's diversity.¹⁰ Transparency requirements should ensure citizens understand when algorithms affect their lives and can challenge harmful automated decisions. Meaningful representation of marginalised communities must extend beyond token inclusion to genuine participation in the design, development, and oversight of AI systems.¹¹ Enforcement mechanisms need real power, giving civil society organisations and affected individuals accessible avenues to challenge algorithmic discrimination through expanded public interest litigation addressing systemic harms.¹²

The stakes are high, and the window for action is closing. If India gets this right, it can show the world how AI becomes a tool for inclusive development, lifting all communities together.⁸ The explicit attention to caste-based discrimination, language diversity, and social hierarchies in governance principles sets India apart from generic international frameworks.¹ This contextual approach provides a template for other nations facing complex social inequalities. However, inadequate oversight risks automating and amplifying every existing social inequality.⁷

Therefore, theevolution of jurisprudence should be faster than technological progress, and it should not compromise the constitutional values. This requires continuous effort, rather than a one-time act of legislation. Furthermore, it also depends on the ability to create a powerful institution that can observe and adjust AI to the changing conditions of society.⁵

References

1.     Chander, Anupam and Singh, Ujjwal Kumar, "Re-imagining Algorithmic Fairness in India and Beyond", Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (2021), available at: https://arxiv.org/pdf/2101.09995 (last visited on March 10, 2026).

2.     Chander, Anupam and Singh, Ujjwal Kumar, "Re-imagining Algorithmic Fairness in India and Beyond", Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (2021), available at: https://arxiv.org/pdf/2101.09995 (last visited on March 10, 2026).

3.     The Digital Personal Data Protection Act, No. 22 of 2023, available at: https://www.meity.gov.in/writereaddata/files/Digital%20Personal%20Data%20Protection%20Act%202023.pdf (last visited on March 10, 2026).

4.     Ghosh, Debmalya, "India Needs an Algorithm Transparency Bill to Combat Bias", Observer Research Foundation (2020), available at: https://www.orfonline.org/expert-speak/india-needs-to-bring-an-algorithm-transparency-bill-to-combat-bias-55253 (last visited on March 10, 2026).

5.      Ray, Trisha, "Auditing AI: What Is It and Why Does It Matter for India", Observer Research Foundation (2022), available at: https://www.orfonline.org/expert-speak/auditing-ai-what-is-it-and-why-does-it-matter-for-india (last visited on March 10, 2026).

6.     The Constitution of India, 1950, Arts. 14, 15, 21.

7.     Vijayaraghavan, P. et al., "Caste Stereotypes in Generative AI", arXiv preprint (2025), available at: https://doi.org/10.48550/arXiv.2505.14971 (last visited on March 10, 2026).

8.     Ministry of Electronics and Information Technology, India AI Governance Guidelines (2025).

9.     European Parliament and Council, Regulation (EU) 2024/1689 on Artificial Intelligence (AI Act), 2024 O.J. (L 1689).

10.  NITI Aayog, National Strategy for Artificial Intelligence (2018), available at: https://niti.gov.in/sites/default/files/2023-03/National-Strategy-for-AI-2018.pdf (last visited on March 10, 2026).

11.  Santhosh, G.S., Akshay Govind, S., Krishnan, G.S., Ravindran, B. and Natarajan, S., "Caste Bias in Large Language Models", arXiv preprint (2025), available at: https://doi.org/10.48550/arXiv.2510.02742 (last visited on March 10, 2026).

12.  Barocas, S. and Selbst, A.D., "Big Data's Disparate Impact", 104 California Law Review 671 (2016).

Note - The information contained in this blog is for general information purposes only. We endeavour to keep all the information up to date and try our level best to avoid any misinformation or any kind of objectionable content. If you found any misinformation or objectionable contents in this website please report us at editors.ilw@gmail.com