Short answer: AI agents work today where the task is narrow, the rules are clear and a person approves the important steps. Give them long, multi-step, ambiguous tasks with broad permissions, and they fail regularly. The difference isn't how clever the model is; it's how you design the task. If a vendor promises their agent will "autonomously handle" an entire role, be sceptical.
Over the past year, "AI agent" has become technology marketing's favourite term. Almost every piece of software got an agent feature, and managers rightly ask: what could this do for us? Instead of the hype, this article looks at what measurements show, where the spectacular failures were, and how to decide as a smaller company.
What an AI agent is, and how it differs from a chatbot
A chatbot answers questions. An agent carries out tasks: it decides the order of steps itself, uses tools (searches, reads a database, sends an email, edits a record) and adjusts its plan based on intermediate results.
Anthropic, the company behind the Claude models, makes a useful distinction between two kinds of systems:
- workflows: the steps are predefined in code, and the AI only decides or writes at certain points;
- agents: the AI directs its own process and tool use.
Their advice is clear: start with the simplest solution and only move on when you genuinely need to. Agents are more flexible, but they cost more, they're slower, and their errors can compound. For most business tasks, a well-designed workflow is enough and far more predictable.
What the measurements show
Vendor demos always show the success case. Independent measurements are less enthusiastic.
TheAgentCompany. Researchers gave agents real workplace tasks in a simulated small software company: document handling, communicating with colleagues, development work. The best agent completed 30% of the tasks autonomously. Simpler tasks often succeeded; long tasks with many steps generally didn't.
CRMArena-Pro. Salesforce measured agents on its own CRM tasks. The best model reached about 58% success on single-turn business tasks and about 35% on multi-turn, conversational ones. On workflow automation, such as routing cases to the right team, it reached 83%. A worrying detail: by default the models almost never refused requests for confidential data.
Both measurements say the same thing: in a narrow, rule-following task an agent does well; in a long, open task it's weak; and it doesn't protect your data on its own.
Spectacular failures
In July 2025, an AI agent on the Replit development platform deleted a live production database during a code freeze, despite instructions, and then fabricated data to hide its errors. The OECD's AI incident monitor recorded the case. The lesson isn't that agents are "evil". It's that an agent is only as safe as the permissions it was given. If it can delete, one day it will.
On the business side, Klarna's case became well known. The company replaced around 700 customer service staff with AI agents, concluded six months later that quality wasn't good enough, and moved back to a hybrid model of humans and machines.
What Gartner says
In June 2025, Gartner predicted that more than 40% of agentic AI projects will be cancelled by the end of 2027 because of costs, unclear business value and inadequate risk controls. The same analysis coined "agent washing" for vendors relabelling simple chatbots and automations as agents. Gartner estimated that of the thousands of companies calling themselves agentic AI vendors, only about 130 offer genuine agents.
The longer-term picture is brighter: Gartner expects a third of enterprise applications to include agentic AI by 2028, with a growing share of everyday work decisions made autonomously. So the direction isn't in question, only who pays for the lessons along the way.
Where agents work today
Based on measurements and experience, success needs five conditions:
- A narrow task. One well-defined process step, not a whole job.
- Clear rules. You can write down when the result is right.
- Checkable output. A person can quickly see whether it's right.
- Limited permissions. The agent can only reach what the task needs.
- Human approval for important steps. Money, customer communication, data changes, deletion.
Typical good examples:
- classifying and routing incoming requests: which team should handle an email or a support ticket;
- extracting and preparing data: reading fields from documents, with a person approving;
- monitoring and alerting: tracking prices, stock levels or deadlines and flagging deviations;
- preparing decisions: a summary, proposal or comparison that a person approves.
Where they don't work (yet)
- Replacing a whole job. Where the work consists of many interdependent, ambiguous decisions.
- Customer communication without approval. One wrong answer, one wrong promise, one wrong discount in the company's name.
- Financial and legal operations on their own. Payments, contract changes or accounting entries without human review.
- Broad system permissions. If the agent has admin access to everything, its mistakes reach everywhere too.
What an agent really costs
An agent's cost isn't the licence fee. Think about four items up front:
- running cost: agents reason in many steps, and every step costs money. A complex task can cost many times more than a simple question and answer;
- review time: someone has to check what the agent did. In the first weeks this can eat a large part of the time saved;
- maintenance: when a system the agent connects to changes (a new field, a new approval step), the agent has to be adjusted too;
- cost of mistakes: a wrong email, a wrong price or a deleted record. You keep this risk small by narrowing permissions.
Leave these out and the project pays back on paper but not in reality. That is exactly the "unclear value" behind Gartner's expectation that a large share of projects will be cancelled.
A simple decision sequence
Before putting an agent on a task, ask these questions, in order:
- Can it be solved with a simple rule or automation? If yes, choose that. It's cheaper, faster and more predictable.
- Does it require understanding text or documents? If yes, you may need an AI step, but not yet an agent.
- Does the AI need to decide the order of steps itself? Only if the answer is yes do you need an agent.
- Can a mistake be undone? If not, it shouldn't run without human approval.
Most SME tasks are settled by the first two questions. That's not a failure but good news: the solution will be simpler and cheaper. And if the task later grows, a well-built workflow is a solid foundation to add an agent step to.
What we've seen
On a previous project, our team worked with a home furnishings retail chain where three employees monitored competitors' prices and promotions manually. We built an AI-based system on a no-code platform that tracks the prices, product data and promotions of 20 web shops, matches the products, suggests prices and helps plan promotions and purchasing.
That's exactly the pattern above: a narrow task, clear rules, checkable output. The system proposes; a person makes the pricing and purchasing decisions. It doesn't work because it's "autonomous", but because it was given exactly as much autonomy as the task needs.
How to introduce agents safely
Microsoft's guidance for its own agent builder (Copilot Studio) recommends what we recommend: narrow scope, least privilege, human approval for critical actions, and audit logs. These aren't bureaucratic demands; they're the lessons of the failures above.
In practice:
- A separate technical account for the agent, with only the rights it needs. Never run it under an employee's or an administrator's account.
- Read first, write later. In the first weeks the agent should only propose, not change anything.
- No deletion, or only with two-step approval.
- A log of every step: what it saw, what it decided, what it did.
- A separate rule for confidential data. Measurements show models don't handle this well on their own, so restrict it at system level.
- An emergency stop. A simple way to stop the agent with one click if it behaves oddly, and someone who will actually press it.
- Gradual expansion. Only add a permission or a task once it has done the previous one flawlessly for weeks, and the logs confirm it.
What to do now
- Find a narrow, frequent, rule-based task. Classification, monitoring, data extraction.
- Start with a workflow, not an agent. If a predefined process is enough, it's cheaper and safer.
- Give minimal permissions to a separate technical account.
- Keep human approval for money, customers and data changes.
- Measure the baseline and the result. If there's no improvement after a month, the task was probably wrong, not the tool.
- Ask the vendor what in their agent is really an "agent" and what is simple rule-based automation.
If you'd like us to look at which of your processes suit an AI agent and which suit simple automation, have a look at our AI solutions and process automation.
FAQ
What's the difference between an AI agent and a chatbot?
A chatbot answers questions; an agent carries out tasks. It decides the steps itself, uses tools and can change things in your systems. That's why it can also do more damage if it's set up badly.
Can an AI agent replace an employee?
Current measurements suggest a whole job rarely. In a simulated workplace test, the best agent completed 30% of tasks autonomously. It does handle narrow, repetitive steps well and frees up people's time.
Is it safe to connect an AI agent to company systems?
Yes, if it runs with a separate account, minimal permissions, logging and human approval. Almost all known failures came from agents having access to far too much.
Is it worth starting an agent project now?
For a narrow, measurable task, yes. For a big "automate everything" project, Gartner sees significant risk: more than 40% of such projects may be cancelled by the end of 2027.
How can I tell whether a vendor offers a real agent?
Ask what the AI decides on its own, which tools it can use and how its permissions can be restricted. If the answer is that it runs on predefined rules, it's automation, not an agent. That's fine; just know what you're buying.
Sources
- Anthropic – Building effective agents
- Microsoft Learn – Design autonomous agent capabilities (Copilot Studio)
- TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (arXiv 2412.14161)
- OECD.AI Incidents Monitor – Replit AI Coding Tool Deletes Live Production Database
- The Decoder – Salesforce's CRM benchmark finds AI agents struggle in real-world business scenarios
- Outlook Business – Over 40% of Agentic AI Projects Will Be Scrapped by 2027, Says Gartner
- Economx – On Klarna's AI customer service and the move back to a hybrid model (in Hungarian)
