IP portfolio management as a reinforcement learning environment

Our thesis is that an enterprise is a reinforcement learning system that runs on people, and that a model can only join it when feedback is tied to the work that earned it. This post applies that to one team.

In most teams that link is missing. In one team of a pharma company it is nearly there already. The IP portfolio team decides what the company patents, where, with which claims, and for how long it pays to keep each right alive.

Three things set this team apart.

1. 01Every action has a price. Each filing, each country and each year of protection comes with an official fee and an attorney's bill.
2. 02Every outcome is written down by someone else. Patent offices and courts publish their decisions with reasons, in registers anyone can read.
3. 03The stakes are the company's revenue. A medicine earns most of its return while its patents hold.

That is the profile of a trading desk, with one difference. A trade settles in days. A patent settles over twenty years.

A portfolio is a set of patent families. A family is one invention filed in many countries. It can protect a compound, a formulation, a medical use or a manufacturing process.

Over the life of a family the team makes the same kinds of decisions again and again.

File or keep secret

Whether an invention disclosure from research becomes a patent application, and when.

What to claim

How broad the claims are and which data supports them.

Where to file

One international application first, then a list of countries about thirty months after the first filing. [1] Every country on the list costs fees for up to twenty years.

How to answer

Argue with the examiner, amend the claims, split the application or appeal. Each office action sets a deadline.

Attack and defend

Oppose a competitor's patent, defend your own, and check each programme for freedom to operate.

Extend

Apply for up to five more years of protection once the medicine is approved. The window is six months in Europe and sixty days in the United States. [2]

Keep or drop

Pay the renewal fee for each patent in each country, or let it lapse. In most countries the question comes back every year.

None of this is done alone. One pharma company's published guideline describes the last step plainly. Before each renewal payment the IP, research and commercial teams review the scientific and commercial interest together and decide to maintain or abandon. [1] The work runs through a docketing system, patent databases, spreadsheets, drafts and a great deal of email with outside counsel.

Patent work leaves an unusually complete record, and the patent offices keep it.

The USPTO has released 4.4 million office actions sent to the applicants of 2.2 million applications between 2008 and mid-2017, with the ground of each rejection and the prior art cited. [3] Its examination dataset covers more than 13 million applications with their procedural history. [4] The European Patent Register opens the file of every published European application. Researchers have built on these sources. The Harvard USPTO Patent Dataset holds 4.5 million applications as they were first filed, each with the office's decision. [5]

Read as reinforcement learning data, an application is a trajectory. Each amendment is a step. Each office action is a review. A grant or a refusal is a reward.

Two things are missing from the public half.

It shows the moves and not the reasons. Nobody outside can see why a team entered nine countries and left out a tenth, which claim it was ready to give up, or which family it dropped to pay for another. Those choices are made in meetings, spreadsheets and emails.

It also scores the wrong unit. An office decides on one application. The team is judged on the portfolio, on how long each product stayed protected and what that cost.

The registers hold the outcomes. The desktops hold the decisions. An environment needs both.

Before states and actions, the model needs one number. What is this team for?

A patent earns nothing by existing. It earns when it keeps a competitor off a product that sells. So the team's return is the margin its patents protect, minus what the patents cost.

J(π) = E_π [ Σ_t γ^t ( Σ_{j, c} m_jct X_jct − k_t ) ]

Here m_jct is the margin that exclusivity protects for product j in country c in year t. X_jct is 1 when at least one valid patent in force covers that product there, and 0 otherwise. k_t is what the team spent that year. π is the team's policy and γ a discount.

Everything the team does moves X or k. Nothing else enters.

The structure of X matters. A product is covered when any one patent holds. That is an OR.

X = 1 − Π_i (1 − x_i)

Here x_i is 1 when patent i is in force, valid and covers the product. If each patent holds with probability p_i, independently of the others, the value that patent i adds is

Δ_i = m · p_i · Π_{h ≠ i} (1 − p_h)

A patent is worth its chance of holding, times the chance that every other patent fails.

Two things follow. Value is not additive across patents, so keep or drop cannot be decided one patent at a time. The second patent on a product is worth less than the first, and the fifth much less.

And the scarce quantity is independence. Suppose that with probability q the patents share one fate, because they rest on the same data or fall to the same prior art. Then n patents that each hold with probability p cover the product with probability

P(X = 1) = q p + (1 − q) (1 − (1 − p)ⁿ)

How far the patents share one weaknessnot at allfully

Chance the product stays coveredIf every patent failed for its own reason

Move the slider to see what a shared weakness does to a portfolio.

Two patents that fail for the same reason are one patent. A portfolio protects only as far as its patents can fail for different reasons, and that is a property no single file shows.

With the objective fixed, the team can be written down as a reinforcement learning problem.

Episode

One patent family, from the invention disclosure to the last expiry. That is up to twenty years, and in Europe up to five and a half more for an approved medicine. [2] A portfolio is many episodes running at once, coupled through the products they cover and the budget they share.

State

The legal status of the family in each country, the claims as they stand, the prior art cited so far and the open deadlines. Beside it the state of the business, with the stage of each programme, the launch plans and what competitors have filed. The team only ever sees part of it.

Action

A choice at a decision point (file, enter a country, amend, appeal, oppose, extend, keep, drop) and the document that carries it out.

Transition

Decided by others. Examiners, opponents, courts and clinical results move the state, slowly and outside the team's control.

Reward

The objective above, which arrives late. Until then a cost at every step, and signals that predict the rest.

Verifiers

Checks that hold at every step. The deadline is met. Each amendment has a basis in the application as filed. The claims agree across the family.

The environment has two time scales. Answering one office action takes a person several days and hundreds of steps across a patent database, the file, a draft and email. The decisions themselves lie months apart. So the natural shape is hierarchical, in the sense of the options framework. [6] One policy chooses among the decisions at each deadline. Another carries the chosen decision out at the desktop.

The team is paid in feedback long before it is paid in outcomes. The signals come from six senders. The fast ones come from people and stay on the desktop. The slow ones come from offices, courts and the market, and most of them are public.

Public registerCompany systemsOnly on the desktop

DaysMonthsYearsDecades

Reviewer in the team

Research and business

Outside counsel

Patent office

Opponents and courts

Market

Redline on a draftReviewer in the team, after days. Only on the desktop.

How far the filed text is from the draft. The densest signal the team produces, and no system keeps it.

Sorted by delay, the signals fall on four clocks.

Days

A senior attorney or outside counsel edits a draft before it is filed. The difference between the draft and the filed version is a dense signal. It exists only on the desktop.

Months

The office answers. A search report lists the prior art found against each claim. An office action accepts or rejects each claim and gives the ground.

Years

Others challenge the patent. At the European Patent Office 2.4% of the patents whose opposition period ended in 2022 were opposed. Across all fields 27% of the decisions revoked the patent. In pharmaceuticals 45% did, and a further 34% kept the patent only in amended form. [7]

Decades

The market answers. How long each product stayed exclusive in each country, what the portfolio cost, and whether the freedom-to-operate opinions held.

None of the signals on the map is the objective. Each one is information about it. That can be made exact.

Let G be the return the portfolio finally earns, the sum inside the objective. Let F_t be everything known at time t. The value at time t is the expected return given what is known.

V_t = E[ G | F_t ]

V only moves when something happens. Call the move at the n-th event δ_n.

δ_n = V_n − V_n−1G − V₀ = Σ_n δ_n

The final return is the first estimate plus the sum of the moves. Nothing is left over. This is what a reward signal is. It is the amount by which the expected outcome just moved.

Two kinds of events move V. The world answers, and that is news. News averages to zero, because anything predictable was already in V. Or the team acts, and that is a choice. The move at a choice is the advantage of that action over what the team would usually have done.

A(s, a) = Q(s, a) − V(s)

The advantage is exactly the credit an action deserves. The gap between two teams, or between a team and a model, is the sum of these advantages along the way. [8]

A choice by the teamNews from outside

1 Choice, year 0

Broad claims are filed.

2 News, year 1.5

The search report cites close prior art.

3 Choice, year 4

Claim 1 is amended with an added feature.

4 News, year 6

The patent is granted.

5 News, year 7

An opposition is filed.

6 News, year 9

Revoked. The amendment of year four added matter.

The same identity removes the twenty-year wait. A value function is not trained to predict the final outcome. It is trained so that each estimate predicts the next one. [9]

V(s_n−1) ← E[ r_n + γ V(s_n) ]

The estimate after a redline predicts the search report. The estimate after the search report predicts the grant. The estimate at grant predicts the opposition. No model has to wait for the market. It has to predict the next clock.

The two halves of the record divide this work. The slow clocks are public, so their predictors can be learned from every applicant's history. The fast clocks are private, so their predictors are learned from the team's own work. Each level only has to be good at predicting the level above it.

The simplest decision shows the whole structure. Every year, for every patent in every country, the team pays the fee or lets the right lapse. A European patent is kept for a median of ten years, and about 18% are kept to the end of the twentieth. [7] Most rights are given up early, and each lapse was a decision.

In 1986 Ariel Pakes modelled this as an option. Paying this year's fee buys this year's return and the right to decide again next year. [10]

V(a, r) = max { 0, r + β E[V(a + 1, r′)] − c_a }

Here a is the age of the patent, r its return this year and c_a the renewal fee. This is a Bellman equation, and V is a value function. Pakes had to estimate it from aggregate renewal counts, because he could not see what any one owner knew about any one patent. A record of the team's work holds exactly that.

Uncertainty about future returnsnonehigh

Renewal feeLowest return worth keepingKept at a loss, for the option

Move the slider to see how uncertainty changes the decision.

The gap between the two lines is the option. A patent in that gap returns less than it costs and is still worth keeping, because it may yet become valuable. The more uncertain the future, the wider the gap. It closes in the last year, when there is nothing left to wait for.

This model treats one patent alone. The coverage formula says the fee buys less when other patents already cover the same product. The team's real problem is both at once. It holds many options whose values depend on each other.

A patent revoked in its twelfth year can fail on a sentence written in its first. The outcome lands a decade later and on another desk.

Patent work has one property that makes this tractable. The reward comes with a written reason. An opposition decision names its ground. The claim was not new or not inventive, or the invention was not disclosed well enough to carry out, or an amendment added something the original application did not contain. Each ground points at one claim, one passage or one amendment. Each of those was produced at one step of someone's work.

In the terms above, the decision does more than move the value. It says which earlier estimate was wrong. In the figure, the amendment of year four looked like a gain when it was made. The decision of year nine takes that gain back and puts the loss on the step that caused it.

So the chain from outcome to step can be followed link by link. The decision names the amendment. The file shows when it was made. The record of the work shows how it was drafted, which alternatives were weighed and who signed it off. The reward is sparse and delayed, but it carries an address.

A log holds only what was done. When a team drops a patent, nobody observes what it would have been worth. When it leaves out a country, no examiner there ever answers.

There is a deeper problem, and it is the reason the public record alone cannot teach a model to act. The team chooses with knowledge the register never sees. Call it z. It is the attorney's private read of how strong the claim really is.

a ~ μ(a | s, z)

The public record supports one estimate, the average outcome after each action.

Q^pub(s, a) = E[ G | s, a ]

A model that acts needs another, the outcome if it took the action itself. [11]

Q(s, a) = E[ G | s, do(a) ] = Σ_z P(z | s) E[ G | s, z, a ]

The two differ whenever z moves both the choice and the outcome. In patent work it always does. Attorneys argue when they believe a claim is strong, and narrow it when they doubt it. Strong claims survive more often whatever is done. So arguing looks better in the register than it is.

How far private judgment drives the choicenot at allfully

True effect of arguing

Read from the public record

Read with the team's record

HarmfulHelpful

Read from the public record alone, arguing looks helpful. It is harmful.

The second formula needs z. That is the precise job of the record of the work. It does not only add examples. It records what each choice was conditioned on, and that turns a correlation in the public data into an effect.

It does so only as far as z shows up in the work. A judgment that leaves no trace stays a confounder.

For actions that were never taken at all, three things help.

1. 01The public record is wide. Other applicants have taken the actions this team did not take, in the same fields and before the same examiners.
2. 02Disagreement can be recorded. When a model runs in shadow beside the team, every case where it would have chosen differently is a labelled comparison, and time settles a share of them.
3. 03The past can be replayed. Freeze the world at a date, hide everything after it, ask for the decision and score it against what happened next. A quant firm calls this a backtest.

A backtest only works with a strict split in time. The Harvard group found that models predicting patent decisions lost accuracy when they were tested on later years and on other fields. [5] An eval for this team has to be renewed as the law and the examiners move.

We think of the environment in three stages. Each one is useful before the next exists.

Replay

Past decision points from the public registers and the team's own docket, with the state frozen at that date. The tasks are the team's real ones. Draft the reply to this office action. Propose the country list for this family. Propose this quarter's keep-or-drop list. Verifiers check each step and held-back outcomes score the result.

Shadow

The model proposes at live decision points and the team decides. Every difference is a label.

Supervised work

The model prepares the work and a patent attorney signs every filing. That signature stays. The measure of success is agreed in advance with the people who sign.

This is a model and not a result. These are the points we expect to be hard.

The horizon

No training run lasts twenty years. Everything rests on each clock predicting the next. How well it does so has to be measured field by field before it is trusted.

The wrong reward

A policy rewarded for grants learns to write narrow claims, which are granted quickly and protect nothing. The reward has to be the coverage in the objective and not the grant.

Shared weakness

The value of a portfolio turns on how far its patents fail together. That quantity is rarely observed, because most patents are never tested.

Moving rules

Case law shifts what counts as enough data. The Unified Patent Court opened in 2023. A value function learned on the last decade's outcomes partly describes the last decade's law.

Small numbers

Even a large portfolio sees few oppositions in a year. The private record has to be pooled with the public one. The private side supplies the reasons and the public side the volume.

Privilege

Exchanges with patent counsel are among the most protected material a company holds. This environment can only be built inside the company, on infrastructure the company controls, with each person deciding what of their own record is shared.

The patent team is the cleanest case and not the only one. Regulatory affairs answers an agency's questions and receives a written decision. Quality closes a deviation, and an inspector later reads the record. Wherever a third party writes the outcome down, the same construction applies.

The patent team is where we would start. It already has a price on each action and an outcome on the record.