GENERATIVE AI CHATBOTS burst into public consciousness with promises of revolutionizing services, including legal services. Yet in practice, their outputs (providing legal services) often mislead more than enlighten. Consider a case in New York: an attorney cited entirely fictional cases in a court brief - all generated by ChatGPT - and faced sanctions.
It has now become a very common understanding that AI “hallucinates;” meaning it invents fictitious precedents or law(s) that “sound good,” but are false. These incidents are not mere anecdotes. They illustrate a fundamental truth about current AI chatbots: they manipulate the statistical patterns of language rather than apply legal reasoning grounded in legislations and cases. As law schools teach, legal argument(s) requires rigorous issue-spotting, doctrinal analysis, and strict adherence to jurisdictional rules. These are qualities which AI chatbots do not inherently possess.
It is quite relevant to unpack why AI chatbots struggle in the legal domain. There exist technological limitations in large language models (‘LLMs’). The crucial difference between narrow AI tools and the “general AI” is notable. These limitations collide with the law’s demand for precise, consistent, jurisdiction‐specific direction. A report by the New York State Bar Association confirms that generic chatbots answer legal questions incorrectly, often by filling gaps with invented content. Unlike factual domains, the law does not tolerate such hallucinations. There lies a need to explore the ethical and institutional fallouts, namely, overreliance on AI can erode lawyers’ critical judgment and even infringe rights (e.g., through opaque predictive systems), while data privacy and bias concerns multiply.
AI’s proper place in law must be conceived as supplemental, not determinative. Practitioners of law must supervise AI tools closely and test them under varied scenarios. We must adopt a citizen-centric, policy-aware stance: avoiding hyperbole on either side, critically examining what kinds of regulation, professional norms, and institutional practices could preserve justice and efficacy in the age of legal AI. AI’s development must align with constitutional principles of due process and fairness.
