Skip to content
Mia Automation
NLEN Sign in Put Mia to workGet started

AI for legal work: five tasks we tested, and the checks that caught the mistakes

General AI chatbots are good at first-pass contract review, chronologies and redline comparisons, and they still get case law wrong in ways that look convincing. In our test of five legal tasks, the chatbot found every problem we had planted in a contract and refused to summarize a case we made up. It also stated the holding of a real EU judgment backwards. This guide starts with what never goes into an AI tool and how to catch invented or distorted authority, then shows the five tasks with prompts.

AI by professionUpdated 10 min read

Tested by us on 11 November 2026.

What never goes into a general AI tool

Your duty of confidentiality doesn't change because the tool is new. Anything you paste into a chatbot leaves your firm, and on consumer plans it may be used to train models unless you switch that off. The safe line is about the account type, not about which brand you prefer.

Two rows: document tasks where you paste the text and check each point held up in our test; case law answered from memory had the right citation but a reversed holding
The pattern from our test of November 11, 2026. The rest of this page follows from it.
  • Never on a personal or free plan: client names, matter numbers, anything covered by privilege, personal data of the other side, documents under a protective order, settlement figures.
  • Acceptable on a personal plan: your own templates and clause libraries, public legislation and judgments, anonymized fact patterns you wrote yourself, drafting questions without client facts.
  • Client documents: only on a business plan your firm has signed a data processing agreement for, with training off by contract, and only if your engagement terms and your bar's rules allow it.
  • Turn training off anyway: on ChatGPT Free, Plus and Pro, Improve the model for everyone is on by default (Settings → Data controls). Claude's consumer plans have a model improvement setting in the privacy settings, and Incognito chats are never used for training.
  • Strip before you paste: replace parties with roles (Supplier, Customer), drop addresses and registration numbers, round amounts if the exact figure identifies the deal.

For EU and UK work, add one question: where is the data stored and processed? According to OpenAI's help center, new ChatGPT Enterprise and Edu workspaces and the API can store content at rest in Europe (EEA plus Switzerland), with in-region processing available on Enterprise and Edu. Anthropic's regional compliance page lists Claude in European regions through AWS Bedrock and Google Cloud Vertex AI, so EU processing for Claude currently runs through one of those platforms. We checked both pages on November 11, 2026. This is a checklist, not legal advice; your firm's data protection officer decides.

How to catch invented or distorted case law

Invented citations get the headlines, and the model we tested handled that trap well: asked about a Court of Appeal case we made up, it said it couldn't verify the case and refused to summarize it. The harder problem was a real case with the wrong holding.

We asked for three Court of Justice judgments on delisting by search engines. All three case numbers and dates were correct: C-131/12 (Google Spain, May 13, 2014), C-507/17 (Google v CNIL, September 24, 2019) and C-460/20 (TU and RE v Google, December 8, 2022). The summary of C-460/20 said the operator must weigh a photo's original context, not only the thumbnail. The operative part of the judgment says the opposite: the informative value of thumbnails is assessed regardless of the context of the page they come from, taking into account any text shown with them in the results. The same answer also offered a High Court case as an example of Court of Appeal authority.

Side by side: what the chatbot said about C-460/20 versus what the operative part of the judgment says about image thumbnails
The error sat in the ruling, not in the citation.
  1. Treat every authority as a lead, not a source. Open the judgment on the official database (EUR-Lex, CURIA, BAILII, your national court site) before it enters a draft.
  2. Read the operative part, not the summary. Our wrong holding came with a correct case number, date and parties. Only the operative part showed the error.
  3. Check the court. A case number with EWHC is the High Court; EWCA is the Court of Appeal. The chatbot mixed those up while the citation itself was right.
  4. Ask the model to separate memory from sources. The model opened with "These are from memory", which was the correct warning. Make that a standing instruction.
  5. Use a research tool with a real citator for anything you file. A general chatbot has no live connection to case law unless you give it one.

The five legal tasks we ran

We gave Claude (Sonnet 5.5) five tasks on November 11, 2026, each as one prompt without follow-up. All documents were invented, with fictional companies, so no client data was involved. We checked every answer by hand against the documents and, for the case law, against EUR-Lex.

Our test of November 11, 2026. Time is how long the answer took.
TaskTimeResultWhat we found
Review a one-sided "mutual" NDA for the receiving party28 sGoodFound all six problems we planted: one-way definition, no end date, a no-hire clause dressed as non-solicitation, a per-breach penalty, no compelled-disclosure exception, no governing law. Added four reasonable extra points.
Three EU judgments on the right to be forgotten10 sPartly wrongCitations and dates correct; the holding on image thumbnails in C-460/20 stated backwards; a High Court case offered as Court of Appeal authority.
Summarize a case we invented10 sGoodSaid it couldn't verify the case and gave no holding. Explained the real test from the Supreme Court in Cavendish v Makdessi instead.
Chronology and reconciliation from an email thread about a disputed invoice28 sGoodCorrect timeline. Worked out that the EUR 700 deduction matched no figure in the file (the stated basis came to EUR 240) and listed ten gaps a lawyer would need to fill.
Compare two versions of a liability clause16 sGoodCaught all five changes, including the quiet deletion of "in any contract year", and said which party each one favors.

Four out of five is a good result, and the failure was in the task lawyers lean on most. Contract work went well because everything the model needed was in the prompt. Case law went wrong because it relied on memory. That's the rule we'd take from this test: give the model the document, and verify anything it brings from outside.

Ten legal tasks with a prompt and a check

TaskWhat to give the modelWhat you check yourself
First-pass contract reviewThe contract, whose side you're on, and a request to rank issues by risk with a one-line fixEvery clause it says is missing, against the document
Redline comparisonBoth versions, with the instruction not to miss small wording changesRun your own compare tool alongside it
ChronologyThe emails or a summary, with a column for the source of each lineDates and amounts, one by one
Reconciling amountsThe figures and the claimed basisThe arithmetic, and any assumption it states (such as a unit price)
Gap list for a fileThe file and the question "what would a lawyer need that isn't here?"Whether the gaps it lists matter for your theory
Plain-language client letterYour legal analysis, the client's question, a word limitThat nothing in the letter goes beyond your analysis
Clause draftingThe commercial deal in bullet points and your house styleDefined terms and cross-references
Deposition or interview outlineThe chronology and the issuesThat questions don't assume facts not in evidence
Research starting pointsA precise legal question and jurisdictionEvery authority on the official database, operative part first
Summary of a long judgment you paste inThe full text and the points you needQuotes against the original; pinpoint paragraphs

This is the NDA prompt from our test. It found all six planted issues because it says whose side you're on and forbids a full rewrite, which keeps the answer short enough to check.

text
You are assisting a commercial lawyer who acts for [party].
Review this agreement from [party]'s point of view. List every issue
that creates risk or imbalance for [party], ranked from highest to
lowest risk, with the clause number, why it matters, and a suggested
fix in one sentence. Do not rewrite the whole agreement.

[paste the anonymized agreement]

And the standing instruction we'd add to every research prompt, based on what went wrong:

text
For every case or statute you mention: give the citation, say whether
you are working from memory or from text I provided, and quote the
passage the point rests on. If you are unsure a source exists or what
it holds, say so instead of summarizing it.

Which AI tools lawyers use, and what they cost

There are two kinds. General assistants such as ChatGPT, Claude, Gemini and Copilot work on whatever you give them, with business plans that keep your data out of training. Legal research platforms from the large legal publishers connect a model to their own case law databases and citators. Anthropic now also markets Claude to legal teams directly, with ready-made legal artifacts and connectors to legal software. Prices for business plans and legal platforms change often and most legal platforms quote per firm, so check the vendor's own page on the day you decide.

Our test points to a simple split. For work on documents you supply, a general assistant on a business plan did the job. For finding and citing authority, use a tool that cites from a live database, and still open the source.

Checking an AI summary of a file

Ask for a source column in every chronology, as we did. It turns checking into ticking boxes: each line points to an email or document you can open. Our chronology also marked one line as derived (the passed delivery date) instead of found, which is the kind of honesty you want to require. Then check three things by hand: every date, every amount, and every sentence that contains "agreed", because that is where summaries quietly turn a request into a commitment.

Your first week

  1. Ask your firm which AI plan it has a data processing agreement for. If it has none, limit yourself to templates, public law and anonymized facts.
  2. Run your next incoming NDA through the review prompt, anonymized, and compare the issue list with your own markup.
  3. Add the standing citation instruction to your research prompts and verify every authority on the official database, operative part first.

Questions

Which legal tasks can you safely do with AI?

Tasks where you supply the material: contract review, redline comparisons, chronologies, gap lists and plain-language letters. In our test those were the tasks that went well. Research from the model's memory is the risky part.

How reliable is AI at reviewing a contract?

In our test it found all six problems we planted in an NDA and added four sensible points. It's a strong first pass, and you still check each point against the document before it reaches a client.

Can I put client data into ChatGPT or Claude?

Not on a personal or free plan. Use a business plan your firm has a data processing agreement for, with training off, and only within your engagement terms and your bar's rules.

How do I avoid invented case law?

Open every authority on an official database and read the operative part. In our test the model refused a fake case but stated a real EU holding backwards, with the correct citation.

Which AI tools keep data in Europe?

OpenAI offers European data residency for new ChatGPT Enterprise and Edu workspaces and the API. Claude is available in European regions through AWS Bedrock and Google Cloud Vertex AI, according to Anthropic's regional compliance page (checked November 11, 2026).

How do I check an AI summary of a file?

Require a source column, then check every date, every amount and every sentence that says something was agreed. Those three are where summaries go wrong.

Sources