Cather­ine Ram­nar­ine

As pow­er­ful as gen­er­a­tive AI tools like Chat­G­PT are, they are not in­fal­li­ble. AI “hal­lu­ci­na­tions” can oc­cur when a type of AI, known as a large lan­guage mod­el, gen­er­ates false in­for­ma­tion. These hal­lu­ci­na­tions can range from sim­ple er­rors to dra­mat­ic in­ven­tions. They can pose re­al-world prob­lems, in­clud­ing in le­gal con­texts where stakes are high, and ac­cu­ra­cy is crit­i­cal. In this ar­ti­cle, we will look at a few sit­u­a­tions in which AI tools “hal­lu­ci­nat­ed” le­gal au­thor­i­ties that did not ac­tu­al­ly ex­ist, and how the Courts re­spond­ed.

One of the first cas­es to deal with AI hal­lu­ci­na­tions was the US case of Ma­ta v Avian­ca 22-cv-1461 (PKC). The claimant filed a law­suit against the de­fen­dant air­line, al­leg­ing that he was in­jured when a met­al serv­ing cart struck his left knee dur­ing a flight. Dur­ing the course of the law­suit, the claimant’s at­tor­neys filed le­gal sub­mis­sions that cit­ed and quot­ed from pur­port­ed le­gal au­thor­i­ties. In re­al­i­ty, the at­tor­neys had used Chat­G­PT, which had hal­lu­ci­nat­ed —ie com­plete­ly made up—the au­thor­i­ties. They did not ac­tu­al­ly ex­ist. Un­sur­pris­ing­ly, nei­ther the op­pos­ing at­tor­neys nor the Court were able to find them, and so the Court or­dered the Claimant’s at­tor­neys to pro­vide copies. Rather than come clean at this stage, the Claimant’s at­tor­neys dou­bled down. Af­ter all, they had asked Chat­G­PT to ver­i­fy that the au­thor­i­ties were re­al, and it had said yes! They pro­vid­ed copies of what they claimed were ex­tracts from the au­thor­i­ties, which had been gen­er­at­ed by AI.

Dis­trict judge P Kevin Cas­tel of the South­ern Dis­trict of New York dis­cussed the use of gen­er­a­tive AI in the le­gal sphere. He not­ed that tech­no­log­i­cal ad­vances are com­mon­place and there was noth­ing in­her­ent­ly im­prop­er about us­ing a re­li­able ar­ti­fi­cial in­tel­li­gence tool for as­sis­tance. How­ev­er, at­tor­neys had a re­spon­si­bil­i­ty to en­sure the ac­cu­ra­cy of their fil­ings. The claimant’s at­tor­neys had aban­doned their re­spon­si­bil­i­ties when they sub­mit­ted non-ex­is­tent ju­di­cial opin­ions with fake quotes and ci­ta­tions cre­at­ed by Chat­G­PT, then con­tin­ued to stand by the fake opin­ions af­ter the op­pos­ing at­tor­neys and the Court called their ex­is­tence in­to ques­tion.

The judge high­light­ed the risks of cit­ing non-ex­is­tent au­thor­i­ties. The op­pos­ing par­ty wastes time and mon­ey at­tempt­ing to con­firm their ex­is­tence. The Court’s time is tak­en from oth­er im­por­tant en­deav­ours. The client may be de­prived of ar­gu­ments based on au­then­tic ju­di­cial prece­dents. There is po­ten­tial harm to the rep­u­ta­tion of judges and courts whose names are false­ly in­voked as au­thors of the bo­gus au­thor­i­ties and to the rep­u­ta­tion of a per­sons at­trib­uted with fic­tion­al con­duct. It pro­motes cyn­i­cism about the le­gal pro­fes­sion and the ju­di­cial sys­tem. Fu­ture lit­i­gants may be tempt­ed to de­fy a ju­di­cial rul­ing by disin­gen­u­ous­ly claim­ing doubt about its au­then­tic­i­ty.

The Court found that the claimant’s at­tor­neys had act­ed with bad faith and or­dered them to pay a penal­ty of US$5000.

In the Cana­di­an case of Zhang v Chen 2004 BC­SC 285, an­oth­er at­tor­ney filed court doc­u­ments cit­ing two non-ex­is­tent cas­es that were hal­lu­ci­nat­ed by Chat­G­PT. Un­like the at­tor­neys in Avian­ca, this at­tor­ney dis­closed and ex­plained her er­ror, apol­o­gised to op­pos­ing coun­sel and the Court and in­di­cat­ed that she had no in­ten­tion of re­ly­ing on the fic­ti­tious au­thor­i­ties gen­er­at­ed by Chat­G­PT.

The Court not­ed that cit­ing fake cas­es in court fil­ings was tan­ta­mount to mak­ing a false state­ment to the Court, and amount­ed to an abuse process which, left unchecked, could lead to a mis­car­riage of jus­tice. How­ev­er, the sub­se­quent con­duct of the at­tor­ney showed no in­ten­tion to de­ceive. She was or­dered to pay cer­tain le­gal costs aris­ing out of the ci­ta­tion of the hal­lu­ci­nat­ed cas­es but was spared oth­er sanc­tions that had been sought by op­pos­ing coun­sel.

At­tor­neys are not the on­ly ones guilty of mis­us­ing AI in Court.

The UK case of Har­ber v Com­mis­sion­ers for HM­RC 2023 UK­FTT 1007, in­volved a self-rep­re­sent­ed lit­i­gant—ie a lit­i­gant act­ing for them­selves with­out the as­sis­tance of an at­tor­ney—who al­so cit­ed fab­ri­cat­ed au­thor­i­ties gen­er­at­ed by AI. The Court ac­cept­ed that the lit­i­gant had been un­aware that the AI au­thor­i­ties were not gen­uine and (as an un­rep­re­sent­ed lit­i­gant) had not known how to ver­i­fy their ex­is­tence us­ing oth­er le­gal tools and re­sources. How­ev­er, it re­ject­ed her at­tempt to down­play the sig­nif­i­cance of her er­ror, echo­ing the sen­ti­ments ex­pressed by the Court in the Avian­ca case about the dan­gers of sub­mit­ting non-ex­is­tent au­thor­i­ties. She ul­ti­mate­ly lost her case.

In the US case of We­ber 2024 NY Slip Op 24258, an ex­pert wit­ness re­lied on Copi­lot, a large lan­guage mod­el chat­bot, to cross check cer­tain cal­cu­la­tions. The ex­pert was un­able to re­call what in­put or prompt he used or to ex­plain how Copi­lot ar­rived at its out­put. The Court’s own tests of Copi­lot pro­duced vary­ing re­sults, rais­ing con­cerns. The Court ob­served that gen­er­a­tive AI tools had “in­her­ent re­li­a­bil­i­ty is­sues”. For these and oth­er rea­sons, the ex­pert’s ev­i­dence was held to be un­re­li­able and spec­u­la­tive.

Clos­er to home, the Caribbean Court of Jus­tice re­cent­ly is­sued a prac­tice di­rec­tion on the use of gen­er­a­tive AI tools in pro­ceed­ings be­fore it. This prac­tice di­rec­tion recog­nis­es the po­ten­tial val­ue of gen­er­a­tive AI tools, and al­lows for their use in draft­ing sub­mis­sions, sum­maris­ing le­gal ar­gu­ments and con­duct­ing ba­sic re­search (though not in prepar­ing wit­ness state­ments). How­ev­er, it al­so stress­es that in­for­ma­tion gen­er­at­ed by such tools must be thor­ough­ly fact-checked and re­viewed for ac­cu­ra­cy by those seek­ing to re­ly on it.

AI tools can be of gen­uine ben­e­fit in many spheres, in­clud­ing the le­gal sphere. How­ev­er, it is im­por­tant to recog­nise and re­mem­ber that they are just that—tools. They are not sub­sti­tutes or re­place­ments for care­ful hu­man over­sight and re­spon­si­bil­i­ty.

Cather­ine Ram­nar­ine is a Part­ner at M Hamel-Smith & Co. She can be reached at mhs@trinidad­law.com.