How do you stop AI from bluffing? HOUSEI's Haruto Koizumi on why company chatbots fail to take root

You feed an AI your company's documents, ask a question, and the answer that is sitting right there in the file never comes back. When I started using Claude and Codex, I lost a lot of time to exactly this: things I had written in Excel just would not surface.

The opposite happens too. The AI answers questions the documents never cover, and sounds perfectly sure of itself. Call it AI bluffing.

None of this has slowed companies down. According to Japan's 2026 White Paper on Information and Communications, 86.4% of Japanese companies now use generative AI somewhere in their work, and about half use it for internal help desks, the place employees go with questions. AI is moving into exactly the spots where a wrong answer hurts.

So nearly nine in ten companies use it, but fewer than seven in ten have decided, as a company, how to use it. There is still a gap between using AI and actually putting it to work.

How do you stop AI from bluffing? And how do you get the AI a company has bought to take root? One person works on both questions every day: Haruto Koizumi of HOUSEI Inc., a company that has spent decades building systems for Japanese newspapers and publishers.

Koizumi is product manager for "imprai ezKotae," a generative AI chatbot that answers questions from a company's own documents. A company uploads PDFs or Word files, and the chatbot answers questions from website visitors or employees while showing which document each answer is based on. Think of the chat window that pops up when you are looking up a part number or a spec on a supplier's website.

And this time, I got to sit down with Koizumi and hear, firsthand, what it takes to put AI at a company's front desk.

Here is the short answer. The keys to stopping AI from bluffing, and to making it stick, turned out to be surprisingly down-to-earth.

• Make the AI answer only from documents, and give grading to someone else
  One AI writes the answer, and a different AI checks it. An AI grading its own work tends to miss its own mistakes.
• The questions it can't answer show where your documents have holes
  Some companies have no documents at all, or never organized the ones they have. Collect the questions the AI couldn't answer, and the missing information comes into view.
• Handing over a tool is not enough
  Every company, and every person, uses AI differently. That is why HOUSEI set up a team of forward deployed engineers who work inside customers' operations.

WHO IS

Haruto KoizumiHaruto Koizumi

Product Manager, HOUSEI Inc.

“Being able to see the questions it couldn't answer. That's what we care about most.”── Haruto Koizumi, in this interview

1. Starting outJoins HOUSEI in sales, straight out of universityLearns generative AI on the job and starts overseeing development as well as selling
2. NextTakes on generative AI productsWorks on several products as the company's generative AI point person
3. NOWProduct Manager, imprai ezKotaeLeads planning and improvement of the generative AI chatbot, and still talks directly with customers as a salesperson

Thank you. I joined HOUSEI straight out of university, in sales. I still do sales, but I also oversee the development side as a whole.

Inside the company I'm the person for generative AI, and I've looked after several generative AI products. These days I'm the product manager, the person who shapes a product from planning through improvement, for a chatbot called ezKotae.

Honestly, I learned it all after joining the company. Generative AI moves incredibly fast, right? Riding that wave in my work was genuinely fun for me, and that's why I've stayed in this area ever since.

I talk a lot with customers and with the people doing the actual work, and generative AI is already widely used. Companies I visit are signing up for Gemini or Copilot and using them more and more.

But at many companies it's still at the stage where individual employees use it on their own. My impression is that rolling it out company-wide to make the whole organization more efficient isn't really happening yet.

There are three main parts to how we improve answer quality.

The first is the documents we take in. The answers are built from the PDFs and Word files the company has registered.

The second is the conversation log. It records what was asked, what the answer was, and even which documents the answer relied on. So if an answer looks off, you can go back and see where it went wrong.

The third is that you can see the questions it couldn't answer. That's what we care about most. For each of those questions, the system evaluates whether the documents were missing something or whether the answering approach was wrong, and suggests what to do. That improvement loop is built into the product.

It's less about learning and more about steadily improving the knowledge base, the body of information the AI answers from, and filling in the gaps.

Two ways, mainly. One is that the model that writes the answer and the model that evaluates it are kept separate. If the same model does the evaluating, it judges with the same habits it had when it wrote the answer, so it may well mark a wrong answer as correct. That's why we use a different model.

The other is that users can see, as a number, how closely each answer relates to the source documents behind it. That makes the logic behind what went wrong much easier to follow.

The speed of generative AI is a big one. Something a feature can't do today, it might be able to do a few months from now. So in ezKotae, you can choose which model to use, and you can always pick the latest one.

Most of them are manufacturers that sell to other businesses.

Product inquiries don't only come through the website form. They also arrive by email and phone. Someone forwards them to sales, sales forwards them to engineering, and only then does the customer finally get an answer. It takes a long time, and a great many companies have the same problem: in the meantime, the customer goes to a competitor.

Products sold to businesses come with part numbers and detailed conditions. But the answers are already written in the spec sheets. Since our chatbot answers only from documents, it's a very good fit.

Questions that can be answered right away go to the AI chatbot. Things a salesperson should explain in detail are handled by people. With that split, people can focus on the work they should really be spending their time on.

A lot of them. They want fewer inquiries, but they don't have the documents in the first place. The AI itself is very capable now, and the quality is high. But without documents, it struggles to give answers.

So before we even get to a sales meeting, we load everything on the company's website into the knowledge base and build a trial chatbot to show them. Even without proper documents, let's try it with what's on your web pages first. That also shows where it couldn't answer, and we fill those in one at a time.

Often the documents exist but aren't organized. In the end, I think handling the daily flood of inquiries eats up their time, so they never got around to it.

Before, an inquiry came in by email and someone typed a reply by email. Now the AI takes the first round and answers the simple questions, and people put their effort into the more complicated cases, like complaints. That's how they divide the work.

Many customers also realize how important their website is. Some have used it as a chance to rethink things: filling out their site, asking what visitors are actually looking for, and what information leads to sales.

I was surprised by how much inquiry work there actually is. I do feel we can make that better.

What's hard is the admin screen: how simple you can make the chatbot's management screen. Improving answer accuracy matters, of course. But if doing it creates more work for the administrator, it backfires. Talking with customers, I keep feeling how much they need a system that's easy to maintain.

We do hear that the maintenance steps are hard to follow. People say, we have roughly this many minutes a day for maintenance, so will it fit into that? Can we keep it maintained while balancing that against accuracy? We definitely hear that.

We're working on it practically every day.

At newspapers, for example, some reporters still want to write without AI.

But I don't really see a pattern by industry. It varies company by company. And even within a company, frontline employees and management see things differently. My sense is that it varies from person to person, too.

For the past year or two, our focus has been building generative AI products and providing them to customers.

But as we've done that, we've seen that if you just hand over a tool, it's hard to tell whether generative AI has really taken hold or how much efficiency it actually delivered. A lot of companies tell us the people on the ground just don't use it. We see that as a problem we need to take on, so just last month we set up a new forward deployed engineering team.

From hearing out the problems on the ground all the way to implementation, we support customers end to end. That's our main focus right now. We've always handled everything from design to development, so being able to do it end to end is the vision we've set for ourselves.

Companies that want to bring in AI but don't know how to introduce it or how to use it. Companies that can't tell how much impact it actually had on the business. I think there are a lot of customers like that.

Even with a DX team in place, the numbers are hard to see, so it's hard to push things forward. We go in alongside them and move forward together.

We're training people internally in various ways. Beyond that, what matters for forward deployed engineering is how we can support customers with technology only we have. These days, anyone can build a generative AI product fairly easily.

So we build AI using what we've learned from years of watching our customers' work. "ezNewsDesk," where generative AI helps reporters gather information, pick people to interview, and decide what to ask them, is a product I think only we could have made, because we've been providing systems to newspapers for so long. Support that draws on that long experience is our strength.

I want to do forward deployed work too, given where things are heading. But building products, making things, is really fun as well. So honestly, I want to do it all. Products and FDE, I'd like to try both.

HOUSEI's banner is "an IT partner that delivers AX," AI transformation. Before that, it was "an IT partner that delivers DX," and we supported customers as a systems integrator, a company that builds systems to order.

From here, we'll deliver useful tools, starting with the newspaper industry we've supported for so long. And through our forward deployed engineering work, we'll go from the customer's problem all the way to implementation. That's the vision.

So far, we've been talking about a chatbot that sits on a company's own website. Next, we're preparing a system that improves the answers when ordinary users ask ChatGPT or Gemini about that company.

People don't only ask about a company through the chatbot on its website. They often ask ChatGPT or Gemini, so we're planning to build, through ezKotae, a way to make those answers more accurate too.

When people talk about reducing AI mistakes, they usually mean switching to a smarter model. What Koizumi's team has built goes in a different direction. Answers come only from documents. A separate AI checks the AI that wrote them. The link between each answer and its source is shown as a number. And a person decides whether to apply each proposed fix, with a record kept.

None of this is flashy. But a company that lets AI answer its customers doesn't really need to know how smart the AI is. It needs to know whether, when the AI gets something wrong, it can trace where it went off track. If you're an engineer, think of code review by someone other than the author. Being able to explain it afterward is what decides whether you can trust AI at your front desk.

Line up the questions the AI couldn't answer, and you get a list of what customers want to know but the company hasn't written down yet. Build a demo from the website alone, then add documents wherever it falls short: Koizumi's approach starts by making that list.

That's why manufacturers using the chatbot have started rethinking their websites. Now that more than half of people search with AI, the record of unanswered questions is no longer just a way to cut inquiries. It is becoming something that shapes how AI describes your company.

Forward deployed engineering sounds like something from the cutting edge, the world of OpenAI and Anthropic. But what Koizumi talked about was far more grounded: maintenance steps that are hard to follow, and whether the work fits into a few minutes a day. Keep the effort within what people on the ground can sustain. Only then does AI go from "installed" to "rooted."

Koizumi's own path stayed with me too. He joined in sales straight out of university, listened to customers, and now leads a generative AI product. Understanding both the customer's words and the development work is exactly what forward deployed engineering asks for.

86.4% of Japanese companies now use generative AI at work. The next question is whether they can keep using it. If your company fields a lot of inquiries, start by sorting incoming questions into those your documents can answer and those only a person can. If you're an engineer, design the checking as carefully as the building. If you're a student, know that starting in sales is one way into making AI products. HOUSEI built its forward deployed engineering team to take on exactly this part: keeping AI in use.