What Happens When Software Starts Making Decisions

For a long time, I thought of software as something people used. You opened it, clicked around, entered some information, and got a result. Even when the system was complicated, the basic relationship was clear. The person decided what they wanted to do, and the software helped them do it.

That relationship is changing. More software now decides what to show, what to hide, what to recommend, what to flag, what to approve, and what should happen next. Sometimes the decision is small enough to ignore. A playlist continues with another song, an inbox moves a message into spam, or a map suggests a faster route. Sometimes the decision affects whether someone gets credit, reaches a doctor, qualifies for support, sees a job listing, or is judged to be a risk.

We still call these things products, which makes them sound like objects. But an object does not usually interpret a situation and take action on behalf of an organization. A website used to present the organization. Increasingly, the software is doing part of the organization’s work.

I started thinking about this while working on Green Filter, my master’s project. The original idea was fairly simple: help young people understand the environmental impact of what they buy, save some of the money they do not spend, and eventually invest it. Once I tried to make the idea work as a real system, it became much less simple.

What should the software consider sustainable? How confident should it be? Should it warn someone, suggest an alternative, or stay quiet? How should it balance environmental impact against health, price, convenience, and personal preference? What happens when the information is incomplete? Who is responsible when the recommendation is wrong?

These were no longer questions about screen layout. They were closer to policy. The software needed rules about how to behave, priorities for what mattered most, exceptions for unusual situations, and a way to handle uncertainty. It also needed someone to decide when the system should act and when it should defer.

That is usually what organizations do. They divide responsibility, set limits, decide what requires approval, create procedures for unusual cases, and develop ways to explain decisions to the people affected by them. When software begins doing this work, those organizational choices get buried inside the product.

A ranking system contains an opinion about what deserves attention. A recommendation system contains an opinion about what is relevant. A fraud detector contains an opinion about what looks suspicious. An automated approval process contains an opinion about which cases are normal enough to proceed without human review. These opinions may come from data, rules, models, or a mixture of all three, but they still have to come from somewhere.

The interface can make the decision look clean even when the process behind it is messy. A green checkmark appears. A request is declined. A candidate is moved down the list. A transaction is blocked. The person sees the result, but rarely sees the arguments, thresholds, missing information, or assumptions that produced it.

This is where product design starts to feel different from the work I learned to value earlier. I used to think the difficult part was making the system function: writing the code, fixing edge cases, getting the database right, improving performance, and making the interface usable. Those things still matter, and they can consume days and nights. But once the system starts making decisions, the technical implementation is only one part of the work.

Someone has to decide what the system is allowed to decide. That sounds obvious, yet many products seem to skip it. A team adds an AI feature because the model can classify documents, answer questions, generate text, or trigger actions. The feature works in a demonstration. Then it enters an organization where people have conflicting goals, unclear responsibilities, undocumented exceptions, and different ideas about what a correct answer looks like.

The software inherits all of that. When an organization does not know who owns a decision, AI does not solve the ambiguity. It conceals it behind a confident response. When a process depends on an experienced employee quietly noticing unusual cases, automation may remove the person who was holding the process together. When the rules are unfair, inconsistent, or outdated, software can apply them faster and more consistently.

This is why adding AI to a product often exposes problems that were already there. A human employee can notice that the official procedure makes no sense in one specific situation. They can ask a colleague, delay the decision, or make an exception. Software needs those possibilities to be designed into it.

Someone has to decide who can override the system, what gets logged, when the system should admit uncertainty, how a person can appeal, who investigates repeated mistakes, and whether the system can be switched off without stopping the whole operation. These are design questions, but they are also management questions. They shape authority, responsibility, and the way work moves through an organization.

The people affected by the software may never meet the people who made those choices. This creates a strange distance. A policy that once had a name and an office becomes a dropdown menu, a score, or an automated message. The decision feels technical even when it came from a business preference, a legal interpretation, a cost-saving target, or a rushed product meeting.

Software gives organizational decisions a surface. That surface can make them easier to use, but it can also make them harder to question. I have spent a lot of time thinking about interfaces as things people interact with. I am starting to think that this definition is too narrow.

An interface can also be where an institution exercises power. It can decide who gets through, what information counts, which option appears first, and what kind of explanation a person receives. The visual design may be friendly, while the underlying act is still consequential.

This does not mean every automated decision is dangerous. Many are useful, and some are better than the human processes they replace. Software can be more consistent, easier to audit, and less affected by fatigue or personal bias. But those benefits do not appear automatically. They depend on how the system is set up, what evidence it uses, who reviews it, and whether people can understand what happened.

The product therefore includes more than the interface and the code. It includes the rules, monitoring, escalation paths, staff who supervise it, documents that explain it, and people who have the authority to change it. It also includes what happens after launch.

That may be the biggest shift for me. I used to see launch as the point where the product entered the world. Now it looks more like the point where the product takes on a role inside an organization. From then on, it has to be watched, corrected, questioned, and sometimes restrained.

The product team is no longer only making a tool. It is deciding how part of the organization will behave.