Author-it demo shows AI clashes in regulated sectors

Author-it has launched an interactive browser-based demonstration showing how enterprise AI systems can return conflicting answers from company documents. The tool covers six regulated industries.

The New Zealand software company says the demonstration compares answers drawn from an ungoverned folder of documents with those taken from governed, structured content. Users select one of six sectors, then put the same questions to each source to see how different content foundations affect the result.

The release focuses on a problem many companies have encountered as they try to use large language models with internal information. Even when an AI system is connected to a company's own documents, answers can still be inconsistent or difficult to verify.

Author-it's example industries are manufacturing, software, utilities, medical devices, financial services, and government. In each case, the demonstration asks users to build an AI knowledge base in two ways: first from common business files such as PDFs, Word documents, and Markdown exports; then from structured JSON output produced by the company's AION format.

One example focuses on manufacturing documentation. A question about a torque specification produces three different values when the AI is directed to a set of ordinary documents: one answer comes from a 2012 sales handout, another from a current service manual, and a third from a superseded specification.

When the same question is put to the governed source, the answer returns a single value alongside details including a topic identifier, author, and modification date. The demonstration also shows that not every answer from ordinary document sets is incorrect, with one question in each industry returning the right result from the ungoverned source.

The issue has become more pressing for companies in regulated sectors, where records, revision control, and approval status can determine whether information is usable in audits, service work, or product support. The tool is aimed at enterprise architects, AI platform leaders, and documentation teams assessing whether existing content is suitable for AI retrieval systems.

Research backdrop

Author-it linked the launch to published academic research on AI accuracy in legal information systems. It cited a study by Stanford researchers, published in the Journal of Empirical Legal Studies, that tested legal research tools sold by the two largest legal information providers.

According to Author-it, that study found hallucination rates of between 17 and 33 per cent. The company argues the finding matters because the tools examined were retrieval-augmented systems drawing on professionally curated legal databases rather than open internet sources.

Adrian Winks, Chief Executive Officer of Author-it, argued that the root problem is often the source material rather than the model itself. "I hear from a lot of people that wrong answers from AI are a model problem. They're not. It's a source problem," Winks said.

Winks described the weakness of ordinary document collections in settings where several files may conflict with each other. "A document doesn't carry a version, or an approval state, or usually even a reliable date. So when three files disagree, the system has nothing to go on. It picks one, and it doesn't tell you it guessed," he said.

Regulated sectors

The risks are sharpest where staff must rely on precise, approved information, Author-it says. In manufacturing, a wrong torque value could lead to a warranty claim or recall; in utilities, a superseded isolation procedure could create problems in front of a regulator.

Medical devices present another case where documentation errors can have operational consequences. An unvalidated cleaning agent in an instructions-for-use document could trigger corrective action, according to the company.

Winks said the cost lies not only in wrong answers, but in the need to check every answer when source quality is uncertain. "That's the part people miss, and it's why this gets expensive rather than just annoying," he said. "Nobody's claiming your documents are all wrong. A source that's right most of the time, with no way to tell which times, is a source you have to check every time. That's not automation. That's a second job."

The demonstration uses fictional product data and scripted responses. The AION output format shown in the tool is part of Author-it's live product release and is included with its cloud subscription offering.

Winks said accountability is central for teams trying to deploy AI in business processes. "Every one of those is an answer somebody has to put their name to," he said. "What teams are finding is that they can't, and it has nothing to do with which model they picked."