AI-funny failure - Other - Weagree

AI-funny failure – Other

Artificial intelligence is all about adequate ‘prompting’. And so is AI-driven automated contract review. Sometimes, AI returns funny results. It is not only that AI may hallucinate (returning a so-called false positive).

While AI is incredibly effective and powerful when used in legal practice, it also gives other funny answers. Also a funny ‘false negative’ may show up.

AI and the option "Other"

We encountered a funny result regarding the option Other. When AI-reviewing an NDA, the AI model (LLM) must identify the ‘Permitted Purpose’. Weagree will by default ‘prompt’ ChatGPT with the Q&A-questions that are also used for creating a contract. In creating an NDA, an obvious Q&A-question is about the purpose of exchanging confidential information:

AI automated contract review Other option artificial intelligence

…and depending on the purpose, provisions are included or excluded (many of them without further asking). Think of provisions on intellectual property rights (which are included in discussing an R&D-project but excluded for M&A-negotiations).

No surprise that such Q&A-question for creating an NDA yourself also includes a catch-all option (“Other…”). If Other is selected, the questionnaire will ask about all optional provisions (the questionnaire will give more flexibility and is more elaborate).

The option "Other" in AI-reviewing an NDA

If you AI-review an NDA with a confidentiality purpose being “the settlement of a claim for alleged defects”, it clearly does not fit into any of the first 6 categories (see the above picture). Settlement of claims is just not R&D or M&A.

As a human being, we would bring claim-settlement under the rest-category 7: Other… But AI or ChatGPT does something different. We expected it to use subcategory Other for all those Purposes that ChatGPT could not classify as one of the six other categories. But this did not happen.

Fun false negative

Instead, in many NDA-reviews, ChatGPT returned a so-called false negative. A false negative is when the AI incorrectly concludes that something is not there. In reviewing NDA’s, ChatGPT did not always identify a ‘purpose’, as in: no purpose was defined at all. Needless to say that every NDA must define a ‘purpose’ (otherwise, under common law, NDA-restrictions would be unenforceable).

Here is the clue: AI does not interpret a contract or our NDA. It merely looks whether (small parts of) the text match patterns of words that it has been trained for. And ChatGPT is incredibly effective when it comes to reviewing contracts. If it finds such pattern, it will return the label that had been attached to it by the model-trainer. In our case: the NDA’s context or purpose.

So, as the purpose of an NDA is not likely to approximate the word “other…”, it will likely never be classified as such. While we human beings understand the idea, an AI will most likely fail. A more effective AI-prompt would be:

AI automated contract review prompt Other artificial intelligence

Did you get it? Human beings interpret a contact text. AI just seeks to recognise word patterns (and returns the label that the AI-model attaches to the prompted optional categories of patterns).

Why is Weagree AI so powerful?

Indeed: to AI-analyse a contract drafted by the counterparty, Weagree will use a customer’s own contract-creation-parameters. That makes Weagree AI so powerful for our customers and it explains why they can use AI almost out of the box.

While the Q&A-questions prove to be exceptionally effective for AI-reviewing the same type of contract when drafted by the counterparty, Weagree has also made it possible to tweak them for AI-reviewing. That makes adopting AI nice: Weagree users are shaping their own future of legal practice. A puzzle of why a Q&A-question is not always an AI-prompt.

Implementing Weagree AI does not require any efforts if either contract creation or CLM (post-signing contract management) have been set up for the relevant type of contract. The Q&A-questions an CLM metadata respectively are being used automatically for the AI-driven contract review.

Terms of Use

I hereby accept (or reconfirm my acceptance of) Weagree’ Terms of use, in which:

Terms of Use

I hereby accept (or reconfirm my acceptance of) Weagree’ Terms of use, in which: