Simplicity metrics for legal text

12 July 2020

Image: Romain Vignes / Unsplash

Sunday evening reflections on how a practical metric for the simplicity of legal text might work (possibly relevant to regulations, procedural rules, advice, contracts or whatever).

The nature of the problem

The main challenge is to weigh one good thing (simplicity) against a potentially good thing (detail and nuance). And to do so in a way that delivers meaningful distinctions, is automated so far as possible and isn’t too hard to apply where not automatable.

Rough idea

How about if the two factors above were weighed then combined as follows.

1. A meaningful metric for text simplicity can be automated

• Text volume alone is such an important aspect of the problem that I would consider starting quite simply with the number of words.
• A % adjustment could be made to reflect algorithmically-defined readability (there are various established algos for this).

2. ‘Benefits of detail’ (or ‘safety’) is harder to pin down

• There are so many considerations. And risk varies a lot with context. It involves both likelihood and impact. And it requires judgement as well as data.
• But let’s imagine that ‘safety’ can be approximated by scoring a text as a % against a gold standard of 100%.
• This score would be a human judgement — apply your experience and turn it into a number. There could be a checklist of things to consider but really you’re the jury — good faith and thought required, but not expressed reasoning.
• Blind voting is desirable (so it’s not just one person’s view).
• Also automation to capture and aggregate votes (and reusing them over time in the case of templates).
• Big differences in votes would be prompts for discussion — ‘who’s missing something?’ — they shouldn’t just be averaged.

Scaleability

To be practically useful, the metric has to be highly scaleable i.e. applicable at sentence level, clause/section level, part-of-document level or full document (or even multi-document) level. One way of doing this would be to use a negative exponent (root) of the text simplicity number.

Example formula

The ‘benefits of detail’ (let’s just call it ‘safety’) % figure could be divided by the quad root of the text simplicity figure to give a metric which compares the two factors in a potentially useful way and which is scaleable.

This gives the results in the spreadsheet image below for an entire agreement clause of 70 words (defined as 100% safe), 38 words (assumed to be 95% safe) and 10 words (with different assumptions as to safety). The image also shows examples for much longer texts without obviously absurd results.

Very rough thoughts indeed, and the formula is basic. Just thinking aloud in case of use to anyone. All sorts of detail require work but I’d be interested to hear any thoughts on the gist.

The rest of this article below the spreadsheet image comprises miscellaneous notes.

Rough hacky numbers

Notes and jottings

In no particular order

• The ‘benefits of detail’ figure inevitably becomes more subjective for large chunks of text. But could be supplanted by a suitably defined average of more granular assessments if and when more bottom-up scoring is done for documents which merit it.
• Such a metric is, of course, only an approximation and risks of misuse would exist (subjectivity, gaming, taking it too seriously, missing points not measured, self-delusion, bias…). But the idea is to create a discussion point and a more conscious approach to the problem of creeping complexity. Nothing more.
• Some benefits can only realistically be assessed from a particular party’s perspective i.e. the impact may not be symmetric. So, for scoring to be meaningful it may sometimes have to be from a specific user (or, indeed, other stakeholder) perspective — rather than done in the abstract.
• To some extent this is a professional agent issue e.g. the lawyers drafting something may become so close to it that it’s difficult for them to take into account the interests of non-lawyers or future readers (lawyers or not). Been there, done that.
• But there’s also a conflict of interest issue, e.g. the perspectives of a large corporation and a much smaller party are just different. This isn’t a weakness of the proposed formula, it’s part of the reality of life and recognising it helps open such issues to discussion and improvement over time. For a widely-used legal text it could be useful to develop one score from one perspective and another score from another perspective: bringing them closer together could be an aim, assuming good faith.
• Arguments could also develop between, say, a large corporation and a consumer group over the scoring of the former’s terms and conditions, given the inevitable element of judgement in the scoring. Legislation could be scored systematically. Targets could be set for reducing or managing complexity in a given domain.
• One of the only things I’m certain about here is that technically better text simplicity algorithm and a better exponential function can be developed than the rough hacks I’ve used in this paper!
• The safety % point needs particular reflection and challenge — is it meaningful, is it culturally acceptable? No doubt all sorts of holes can be found, but what are the better alternatives? Separate scoring of likelihood and impact is one possibility in certain contexts, though not always relevant or useful.
• Features and formats other than text may improve (or degrade) simplicity of interaction. This can be factored in, but I’ve started with text as that’s still how the overwhelming majority of legal material is exclusively expressed.
• If an approach can be settled, rules of thumb could be developed for the scores expected in certain contexts.
• Although the human scoring element involves applying some time and thought to the topic, that shouldn’t necessarily be a deal breaker if the issue is important. If it becomes a habit, it could be a quick way to test one’s own drafting as it develops — select the text, enter your % safety assessment and see what the result is. Then see it improve as you simplify the text without increasing risk.
• There will inevitably be an argument that the problem is adequately mitigated by the existence of tools for summarising lengthy text. My intuition is that this is only right up to a point.

Example clauses

The three example entire agreement clauses mentioned are below for completeness.

Long entire agreement clause (70 words)

This Agreement contains the entire agreement and understanding among the parties hereto with respect to the subject matter hereof, and supersedes all prior and contemporaneous agreements, understandings, inducements and conditions, express or implied, oral or written, of any nature whatsoever with respect to the subject matter hereof. The express terms hereof control and supersede any course of performance and/or usage of the trade inconsistent with any of the terms hereof.

Medium entire agreement clause (38 words)

This Agreement constitutes the entire agreement between the parties with respect to the subject matter hereof and supersedes all other prior agreements and understandings, both written and oral, between the parties with respect to the subject matter hereof.

Short entire agreement clause (10 words)

This document is the definitive record of the parties’ agreement.