The Cheapest Way to Be Wrong

I am considering A/B testing a single character.

Should the name be HAAM, or HAAM &?

The ampersand version suggests that something else is coming. Design and technology. Strategy and implementation. Serious work and sunshine. It might make HAAM feel more collaborative, more open-ended, or more like a company than one person's surname.

It might also look awkward and achieve absolutely nothing.

This is exactly the kind of decision designers normally make through taste. We look at both versions, develop an argument, and choose the one that feels right.

But taste is still a hypothesis.

The $100 million idea nobody wanted

In 2012, someone working on Bing proposed a small change to the way advertising headlines appeared in search results.

The change would only take a few days to build. Managers nevertheless considered it low priority. It joined hundreds of other ideas and sat untouched for more than six months.

Eventually, an engineer decided that it was cheap enough to try.

Within hours of launching the experiment, Bing's monitoring systems triggered an alert. The result looked too good to be true, which usually meant something had broken.

Nothing had broken.

The new version increased revenue by 12 percent without damaging Bing's important user-experience metrics. Microsoft estimated that the change was worth more than $100 million annually in the United States alone. It became the most valuable revenue experiment in Bing's history. (Harvard Business Review)

The obvious lesson is that small interface changes can be worth enormous amounts of money.

But the more interesting lesson is that the company had already evaluated the idea.

It had evaluated it as unimportant.

The experiment did not merely discover a better advertising headline. It revealed that Microsoft's internal system for judging ideas had placed one of its most valuable ideas near the bottom of the pile.

The idea's value was not hidden from the customer. It was hidden from the organization.

Experiments let low-status ideas compete

Most companies claim that the best idea should win.

In practice, ideas accumulate status before anyone encounters them.

An idea from a senior executive receives attention. An idea with a polished presentation feels more credible. A large redesign appears more strategic than a wording change. A video feels more ambitious than a photograph. A feature that took three months to build becomes emotionally difficult to abandon.

Ideas do not enter the world equally.

An experiment creates a temporary alternative political system. The intern's button and the director's redesign can be placed in front of comparable groups of people. For a brief moment, organizational status stops determining which idea is allowed to succeed.

This is what makes experimentation uncomfortable.

It is not only a method for improving conversion. It redistributes authority.

The campaign preferred the losing version

Before the 2008 United States presidential election, Barack Obama's campaign tested the splash page used to collect email addresses.

The team created 24 combinations from six pieces of media and four button labels. More than 310,000 visitors were randomly shown one of the combinations.

Campaign staff strongly preferred a video.

They were running a historic campaign, and video presumably felt energetic, persuasive, and presidential.

Every video lost to every still image.

The winning version combined a photograph of Obama's family with a modest button reading "Learn More." It produced an 11.6 percent signup rate, compared with 8.26 percent for the original page, a relative improvement of 40.6 percent.

Dan Siroker, who worked on the campaign's analytics, estimated that the increased signup rate eventually produced 2.88 million additional email addresses and approximately $60 million in donations. That figure is a campaign-side extrapolation rather than an independently audited amount, but the measured signup difference itself came from a large randomized experiment. (Optimizely)

The campaign wanted to persuade people.

The winning version asked them to learn more.

It is easy to invent a convincing explanation after seeing the result. Perhaps the family image made Obama feel trustworthy. Perhaps "Learn More" required less commitment. Perhaps visitors did not want to watch a campaign video.

But none of these explanations predicted the winner beforehand.

The experiment did.

Tiny changes do not create enormous value

The Obama button was not worth $60 million.

The button changed the probability of an email signup. The email address entered a system that could request donations, recruit volunteers, promote events, and mobilize voters for the rest of the campaign.

The small change affected the entrance to a much larger loop.

This is the pattern behind most legendary A/B tests.

A thumbnail changes whether someone watches a film. That changes the value of a content library.

A payment option changes whether someone completes checkout. That affects every transaction flowing through the system.

A support link changes whether a confused user becomes an activated customer. That changes subscription revenue.

The interface element contains almost none of the value. It redirects value already moving through the system.

This is why the same improvement can be irrelevant on one website and transformative on another. A one-percent lift multiplied by fifty visitors is noise. A one-percent lift multiplied by millions of transactions is an operating strategy.

The valuable question is not only, "How much did this change improve conversion?"

It is, "What larger system does this decision sit in front of?"

Fewer clicks can be better

YouTube once tested the icon for Super Thanks, a feature that lets viewers pay creators to show appreciation.

The original icon showed clapping hands. The alternative combined a heart with a dollar sign.

The heart-dollar icon received 20 percent fewer clicks.

A team monitoring click-through rate might have declared it the loser. But completed purchases increased by more than 50 percent.

The clapping hands generated more curiosity. The heart-dollar sign generated more understanding. People clicked it less often because fewer people clicked without knowing what it did. (YouTube)

The interface became less engaging and more effective.

This is a useful correction to the usual story about optimization. A/B testing does not necessarily teach us how to make people do more. Sometimes it teaches us that fewer, more intentional actions are worth more than a larger pile of ambiguous activity.

The winning icon did not maximize interaction. It clarified the transaction.

A metric can win while the product loses

Intercom ran an experiment in which half of its new self-service customers received the usual in-product help links, while the other half received none.

The test ran for 28 days.

Without the help links, users started fewer support conversations and performed fewer searches in the help center.

From one angle, this looked excellent. Removing support content appeared to reduce the need for support.

But after the trial ended, the group with access to help content converted to paid plans by a margin of nine percentage points.

The users without help were not more independent. They were more lost. They asked fewer questions because they did not know where to begin, and then they quietly left. (Intercom)

A similar problem appeared in a Bing experiment involving a bug that produced worse search results.

Distinct queries per user increased by more than 10 percent. Revenue per user increased by more than 30 percent.

By two important metrics, the broken product was an extraordinary success.

People were searching more because they could not find what they needed. They were clicking more advertising because the advertisements now looked more relevant than the damaged organic results. (Microsoft and Stanford researchers)

This is the darker value of experimentation.

A test can reveal that the company does not merely have the wrong design. It may have the wrong definition of success.

The experiment still requires judgment. Data does not decide which future is desirable. It only makes the consequences of a decision harder to ignore.

Measuring qualities that previously sounded soft

Product teams frequently say that speed, clarity, accessibility, trust, and support are important.

The problem is that these qualities compete against features with more visible business cases.

A new feature promises revenue. Performance work promises that the website will feel slightly faster.

Bing engineers created an experiment that artificially delayed pages by 100 milliseconds. That tiny delay reduced total revenue by 0.6 percent. The test gave latency an economic value and changed how the company could evaluate the trade-off between additional features and slower pages. (Harvard Data Science Review)

Performance stopped being an engineering preference. It became something the business could price.

A good experiment can make an invisible quality legible to an organization.

Sometimes the value is doing less

Wikipedia's fundraising campaigns used to run large personal appeals featuring founder Jimmy Wales.

In 2012, the Wikimedia Foundation tested a different format. Instead of asking readers to click through to read a personal message, a smaller banner placed the essential facts directly above the article.

In some tests, this "facts banner" produced approximately three times as many donations as the older personal appeal.

Testing and iteration helped Wikimedia reduce its main fundraising campaign from 46 days in 2011 to nine full days in 2012, while increasing the amount raised from roughly $24 million to $35 million. The campaign ended two weeks ahead of schedule. (Wikimedia Foundation)

The organization did not only gain more donations.

It needed fewer banners, fewer interruptions, and fewer days of asking.

Optimization is often presented as a machine for extracting more from people. But the best result can be reaching the necessary outcome with less pressure.

A more effective interface can create a quieter product.

The average customer does not exist

Stripe tested what happened when checkout pages dynamically displayed at least one relevant payment method in addition to cards.

Across its experiment, businesses saw an average revenue increase of 12 percent and a conversion increase of 7.4 percent.

The averages concealed much larger local effects. Offering Alipay to customers in China increased conversion by 91 percent in the tested conditions. BLIK increased conversion by 46 percent in Poland. iDEAL increased it by 39 percent in the Netherlands. (Stripe)

There was no universally superior checkout.

The better experience depended on where the customer lived, what they were buying, which device they were using, and how they were accustomed to paying.

The experiment did not locate the one correct design. It demonstrated that context was part of the design.

Most ideas do not work

The spectacular anecdotes can make A/B testing sound like a treasure hunt in which every button contains several million dollars.

It does not.

Evidence from Microsoft suggests that roughly two-thirds of product changes have a neutral or negative effect on the metric they were intended to improve. In heavily optimized products, the success rate can be lower still. (Harvard Data Science Review)

This may be the strongest argument for experimentation.

Without a test, many of those ideas would still have shipped.

Teams would have held meetings, written specifications, created designs, implemented code, announced releases, and added complexity to the product. Because the effect was small, nobody would have noticed that the new feature contributed nothing.

The visible value of experimentation is the winning variant.

The invisible value is all the losing variants that do not become permanent.

Building a company capable of surprise

I do not expect an ampersand to transform HAAM.

The site may not yet receive enough traffic to produce a statistically meaningful answer. Even with sufficient traffic, the test could show no difference. The result could depend on the headline beside it, the visitor's language, or what they already know about the company.

But the experiment changes the nature of the decision.

Instead of asking only which logo I prefer, I have to ask what I believe the ampersand will do.

Will it make people more likely to explore the work? Will they understand the company differently? Will more visitors reach the start page? Does the name promise collaboration, or merely create confusion?

A test forces an aesthetic instinct to become a claim about the world.

That claim can then encounter reality.

This does not mean surrendering design to whatever receives the most clicks. The YouTube experiment showed that clicks could fall while purchases rose. The Intercom test showed that reducing support activity could destroy conversion. The broken Bing results showed that revenue itself could temporarily reward a worse product.

Judgment remains necessary.

The better relationship is this:

Taste proposes.

Reality responds.

Judgment interprets.

The deepest value of A/B testing is therefore not optimization. It is institutional humility.

It gives organizations a way to admit that experience, confidence, seniority, and taste do not make customer behaviour predictable. It allows an ignored idea to become the best idea in Bing's history. It lets a family photograph defeat an expensive video. It reveals that fewer clicks can mean better understanding, and that fewer support questions can mean users have given up.

An experiment makes being wrong cheaper.

More importantly, it makes being surprised useful.

Maybe the ampersand wins. Maybe it loses. Maybe nobody cares.

The point is not to prove that one character contains hidden value.

The point is to build a company capable of discovering that it might.

Connected reading

• How Companies Actually Do Data-Driven Design
• Every Interface Contains a Business Model
• The Product Needs a Harness

AI authorship note

This essay grew from Kris Haamer's question about the value created by A/B testing and the possibility of testing HAAM with and without an ampersand. AI supported deep research, source comparison, synthesis, drafting, editing, and implementation. Kris directed the argument toward institutional humility, selected the final examples, and chose the text for publication.