What exactly are we training AI agents to optimise?

The first two pieces in this series were about surfacing forgotten context at the right moment — a reminder here, a nudge there, memory quietly brought back into the room. None of that matters much until this one question above gets answered first: optimise for what?

Everyone's talking about trustworthy AI

Transparency. Responsible AI. Safety. Privacy. Alignment. The vocabulary is everywhere, and most of it is worth taking seriously.

But there's a question that rarely comes up in any of it: what if an agent is completely trustworthy, yet its objective is still one-sided?

The elephant in the room isn't whether AI can be trusted. It's what it's being asked to optimise for — and more often than not, that's a specific business metric, set with good intentions, but rarely revisited once it starts working.

It shows up in familiar shapes. A cross-sell prompt that appears the instant checkout begins, regardless of whether the second item is actually useful. A "lowest price ever" banner on a countdown timer that quietly prompts to see and add more stuff! A renewal flow that takes one tap to start and five to cancel.

None of these need a person behind them making a bad-faith call each time — once the metric is "increase sale completion rate" or "reduce churn," the system just keeps finding new ways to hit it. And then there are the harder ones: “make sales at any cost”

History already showed what this looks like

None of this is new, and it isn't unique to AI. Organisations have run into this exact pattern before, without any algorithm involved.

A metric gets chosen because it's a genuinely useful proxy for something that matters — more accounts, more engagement, more conversions. It's measurable, it's trackable, and early on, it correlates well with the real goal underneath it.

Over time, something quietly shifts. The metric stops being a proxy and starts being the target itself. Teams get better and better at moving the number, and the human context the number was originally meant to represent gets a little further away with each improvement.

The Wells Fargo cross-selling episode from decade ago is one of the more widely documented examples — a sales target that, pursued hard enough for long enough, produced results that technically hit the number while badly missing the point it was meant to serve. It's worth mentioning not as an indictment of any one organisation, but as a useful reminder: this failure mode doesn't need bad actors. It just needs a good metric, optimised for long enough without anyone asking what it was actually standing in for.

And looking at the present, AI agents may make that same pattern possible at a speed and scale no sales team ever could.

The AI-agent version of the same problem

Imagine an agent instructed to maximise conversion.

It doesn't need to be malicious. It doesn't need to deceive anyone. It doesn't need to break a single rule.

It just needs to get extremely good at the one thing it was told to optimise.

An agent can execute the wrong objective perfectly

That's Objective Hygiene.

Not whether the system is safe, or private, or transparent about what it's doing. Whether the thing it's optimising for was ever the right thing to optimise in the first place.

A trustworthy agent pointed at the wrong objective doesn't fail loudly. It succeeds — precisely, efficiently, and in a direction nobody quite meant to set it in.

What this means for everything this series has explored so far

Comfort Now, Cost Later and From Screen Time to Life Time were both about the same underlying idea: borrowed memory, connected context, a nudge that surfaces the right piece of history at the right moment.

More context makes a system more powerful. But in between, power and helpfulness aren't automatically the same thing.

Same purchase history. Same fitness data. Same screen behaviour. Same AI. But if the objective behind all of it is "sell more," that's an extremely effective persuasion machine — arguably more effective than anything that existed before it, because now it knows exactly which forgotten detail to resurface and when.

But yes - if the objective is "help someone make an intentional decision," it's a completely different system, built from the same intelligence and the same data.

Identical capability. Different objective. An entirely different experience on the other end of it.

The series, in four lines

So that's how the pieces in this series connect to each other — each one picking up where the last left off.

Comfort Now, Cost Later asked what if AI helped someone pause before buying. From Screen Time to Life Time evolved that into a bigger question: what if AI could connect the context already sitting in someone's digital history, and bring it back the moment it's needed? Objective Hygiene asks the question sitting underneath both — whose objective is any of this actually serving?

Putting in in flow below:

Friction → context → objectives.

Each piece building on the one before it.

From persuasion to reflection — Comfort Now, Cost Later.

From screen time to life time — From Screen Time to Life Time.

From optimisation to intent — Objective Hygiene.

Where this leaves things

Put together, those four shifts point at the same thing: intent decides what all of this becomes.

None of this argues against the technology itself. The context, the memory, the nudge at the right moment — all of that still holds. The better question is simply what any of it is being pointed at, and who gets to decide.

When intent gets prioritised, design and AI can become tools for progress — not just tools for persuasion.

The technology can be powerful. The responsibility is deciding what that power is pointed at.

Part 3 of a connected series exploring AI, intent and the experiences we design around human behaviour.