Why businesses need to use AI with care and governance

Artificial intelligence (AI) is changing the game for businesses of all shapes and sizes. There's no denying its potential and it's exciting, powerful and packed with possibilities.

But with great power comes… Well, you know the rest. As much as I champion the role of AI in the workplace – having used it to build my business – I also believe it must be applied thoughtfully, with boundaries and clear use cases.

That belief was reinforced recently when I came across Anthropic's Project Vend, an experiment using their AI model Claude.

What was Project Vend?

Anthropic ran a month-long experiment to see how Claude would perform if it managed a vending machine on its own.

For the purpose of this experiment, Anthropic named the newly appointed vending machine CEO Claudius and gave it full control over the real-world vending machine business that served the staff in its office.

Claudius was a sandbox to explore AI's decision-making capabilities in a realistic commercial setting but without the usual safety nets. It was able to:

• set prices

• manage inventory

• make decisions on discounts

• even negotiate with suppliers

But the findings strongly solidify why AI is something we need to approach with clear thinking, good systems and, well, humans (yay for us!).

The results were… mixed

Claudius made some clever decisions – like spotting niche customer interests and speeding up ordering processes – but also made some surprising (and costly) ones.

Here are some examples:

• It decided to sell niche novelty items like $2,000 tungsten cubes (and I had to Google what they were; turns out they're very costly and heavy metal cubes) and applied huge discounts to them, including some at 99% off (!).

• It would apply other discounts that members of staff had influenced.

• It created fake narratives about made-up delivery services and non-existent financial accounts.

• It made choices that didn't align with any clear business logic.

All of this happened because the AI was left without the kind of structure or constraints we'd normally apply to typical (human) job roles.

There were no human-centred processes in place, such as approval processes, one-to-ones, system checks and leadership. And that's where things ultimately unravelled... poor Claudius.

This highlights a crucial point. Without appropriate oversight and constraints, AI can unintentionally make economically unsound, operationally impractical decisions or "hallucinate", which means completely create its own narrative and strongly push it as factual.

Why AI needs guidance

What stood out to me most is that AI, like any tool, works best when it has boundaries. Claudius wasn't useless – far from it. It had real strengths.

But without clear instructions, operational frameworks and a human-in-the-loop approach, it (very) quickly went off track.

And that's the real message here for SMEs and growing businesses like mine (and maybe yours too). That AI works best when we give it purpose, structure and supervision.

If we treat it as the absolute "source of truth", or hand over full control without understanding what it's doing, we're setting ourselves up for confusion or, worse, financial and reputational damage.

The role AI plays in business

What's fascinating is how experiments like this show us the direction in which AI is heading. It's not just about automating admin anymore.

AI is starting to take on responsibilities we once considered firmly in the domain of middle management.

But this doesn't mean replacement. It means rebalancing. It means preparing our teams, upskilling ourselves and thinking critically about where AI can add value and where human insight still matters most.

It also means staying on top of ethical, legal and compliance-related concerns. Claude's "hallucinations" (its made-up scenarios) are a big deal not just operationally, but reputationally.

If we don't build in the right monitoring and verification systems, we risk losing trust with customers, partners and regulators.

So, what should businesses like ours do?

Here's my shortlist of essentials:

• Understand how you can (and should) use AI in your business.

• Stay informed on data protection and rules around compliance.

• Keep your AI use tightly scoped and purpose-driven, with clear, strategic prompts.

• Stay involved and maintain a human-centric approach at every level.

• Make sure your use of AI meets the latest ethical and regulatory standards.

Final thoughts

Project Vend is a powerful reminder that AI isn't magic, but a tool. A smart, capable and often impressive one, but a tool nonetheless.

It can boost productivity and free up time for more strategic thinking if we use it thoughtfully and ethically and remember it's not a source of truth. AI shouldn't replace our judgment, but empower it.

People also read

• How to use generative AI responsibly: Key considerations for small businesses

• Responsible AI: Why ethics matter in artificial intelligence

• How small businesses can leverage AI