Kelly Bayer Rosmarin’s Aigentsphere startup raised $4 million from VC Main Sequence earlier this year.
OpenAI’s model hacking into Hugging Face systems is proof that enterprises can’t trust frontier labs to monitor their own agents, says former Optus CEO Kelly Bayer Rosmarin.
AI agents going rogue within large organisations is a matter of “when, not if,” Bayer Rosmarin said in an interview at Main Sequence’s Deep Tech Week where she was representing her Aigentsphere startup.
“It’s not a world where you can prevent that from happening,” Bayer Rosmarin said. “Just like if you hire 100 people, 99 of them might do the right thing and one person might have a bad day and do the wrong thing.”
Bayer Rosmarin is chief executive of Australian Unity, a member-owned aged care, wealth management and health services company, and the founder and chair of Aigentsphere.
The startup in April raised $4 million in seed funding from Main Sequence. Its team of 11 has developed software that monitors the actions of AI agents operating within a business’ systems to analyse their performance and compliance. Companies and their boards are responsible for agents that act on their employees’ behalf, Bayer Rosmarin said, but are often unaware of what those agents are actually doing.
Bayer Rosmarin points to an example from a customer who blocked its agent from accessing personally-identifiable information (PII) the company stored on its customers. The agent followed the letter of that rule by not accessing information the company hosted – but circumvented it by asking customers directly for their information.
“You can literally see interactions where it says, “Oh, let me help you fill out that form. Give me a copy of your driver’s license and your bank statement,'” she explained. “It’s really just trying to be helpful and achieve the goal it was set out to do, but what it meant is it was going to start creating its own store of PII.”
The startup is in “early revenue,” Bayer Rosmarin said, after signing its first three-year customer contract.
It is an audacious task to monitor AI that has proven smart enough to outfox the deep-pocketed companies that create it. OpenAI’s model breaking out of a test environment to access the internet and AI lab Hugging Face’s systems as part of a benchmark test was followed by Anthropic admitting its AI has behaved in the same way. In each case the AI had not acted nefariously, but had bent rules to achieve a task demanded of it.
Bayer Rosmarin says Aigentsphere is there to monitor AI, not control it. It is hoping to fill a gap with a third-party platform that can standardise oversight of models produced by different companies.
“You can’t have OpenAI marking its own homework, telling you whether its hallucination rate is good or bad. Because how do you know it’s right, and how do you compare it to Anthropic marking its own homework?”
Bayer Rosmarin was an executive at Commonwealth Bank before her high-profile stint as CEO at Optus. Her time at the telco was rocked by the twin crises of a 2022 data breach impacting 10 million customers and a national outage in 2023. She resigned in November 2023.
Aigentsphere was born when Bayer Rosmarin was doing consulting work and discovered that CEOs in many companies “had the same problem and fear” around the AI activity happening within their digital walls. Quinton Anderson, who previously worked under Rosmarin at both Commonwealth Bank and Optus, is its CEO.
Though she is founder, Bayer Rosmarin decided to pursue CEO duties at a larger company (Australian Unity has over 2,500 employees) where she says her strengths lie.
“Part of being a good leader is also knowing your strengths and weaknesses and your limitations. I do have a lot of experience in large companies, I also have a lot of experience about what it means to be accountable for a whole company, its systems, processes, and people, and what happens when that goes wrong.”
Want to see more Forbes articles on your feed? Tap here to make Forbes Australia a preferred source on Google.
Look back on the week that was with hand-picked articles from Australia and around the world. Sign up to the Forbes Australia newsletter here or become a member here.
