The Interrogation Method: Using LLMs to Extract Human Expertise Through Dialogue

Overview

Large language models (LLMs) excel at generating text based on prompts, but their effectiveness often depends on the quality and completeness of the context we provide. When a task is complex—like designing a new feature, validating a specification, or capturing domain knowledge—the required context can run several pages. Traditionally, a human must craft this context manually. However, there is a more efficient approach: let the LLM interview you to gather the necessary information. This technique, sometimes called interrogatory LLM, flips the script: instead of you feeding the LLM a static document, the LLM asks you targeted questions to build or validate a context document on its own. This guide will walk you through implementing this method, whether you need to produce a comprehensive prompt for another LLM session or review an existing document through dialogue.

Prerequisites

Before you begin, ensure you have the following:

• Access to an LLM API or chat interface that can follow multi-turn instructions (e.g., OpenAI GPT-4, Anthropic Claude, or similar).
• Basic understanding of prompt engineering—knowing how to set system messages, use delimiters, and specify output formats.
• A clear goal for the context document you want to produce (e.g., a design spec, a set of business rules, or a review checklist).
• Patience for iterative refinement—the interrogatory process may require several back-and-forth exchanges.
• Optional but helpful: A code editor to test prompts programmatically via an API; however, the guide will use generic prompt examples.

Step-by-Step Instructions

1. Define the Scope of the Interrogation

Start by deciding what the LLM should produce or review. For example:

• Build a context document: You need a detailed prompt that includes user requirements, implementation guidelines, and external system references for a future AI task.
• Review an existing document: You have a software specification and want the LLM to interview a human expert to check its accuracy.

Write a brief description of the desired outcome. This will be part of the initial system prompt.

2. Craft the Initial Prompt

The first message to the LLM should instruct it to become an interrogator. Use a system message that clearly states its role. Here's an example for building context:

System: You are an expert interviewer. Your goal is to gather all information needed to write a comprehensive design document for a new feature. Ask me one question at a time. Do not ask multiple questions in a single turn. After you have enough information, produce the final document.

Include any constraints, such as:

• Ask only one question per response.
• Use follow-up questions to dig deeper.
• When you are ready, say "I have enough information" and then output the document.

3. Conduct the Interview

Once the LLM starts, respond to each question as if you were a subject matter expert. Keep answers concise but informative. If the LLM drifts and asks two questions at once, gently remind it to stick to one. For instance:

You: "Please ask only one question at a time." (in a new turn)
LLM: "Apologies. Let me start again: What is the primary user goal for this feature?"
You: "The user wants to automatically categorize incoming support tickets."
…

The dialogue will continue until the LLM signals it has gathered enough. This may take 10-20 exchanges depending on complexity.

4. Review and Refine the Output

After the LLM produces the final document (or a review critique), check it for completeness and accuracy. You may ask it to revise specific sections by starting a new interrogation with the same scope but asking it to focus on missing details. For example:

System: You previously collected information for a design document. Based on our conversation, you produced a draft. Now I want you to expand the section on error handling. Ask me follow-up questions about edge cases.

5. Using the Interrogatory LLM for Document Review

If your goal is to validate an existing document, provide the document as part of the context and instruct the LLM to interview you (or another expert) to verify each claim. Example prompt:

System: You have been given a software specification. Your task is to interview a domain expert to check if each requirement is accurate. Ask one question at a time, and after the interview, produce a list of discrepancies or confirmations.

Then attach the document in the same message. The LLM will begin questioning you about its contents. This is a great alternative to asking the expert to read the document from scratch, which many find tedious.

Common Mistakes

❌ Asking (or Allowing) Multiple Questions at Once

The most frequent issue is that the LLM often bundles several questions into one turn. This overwhelms the human and can cause confusion. Solution: Include a strong instruction in the system prompt: "Always ask exactly one question per turn." If it slips, correct it immediately.

❌ Vague Scope

Without a clear definition of what the final output should contain, the LLM may wander or collect irrelevant details. Solution: In the system prompt, specify the sections or topics the document must cover (e.g., user stories, data model, API endpoints).

❌ Forgetting to Remind the LLM

The LLM may need multiple reminders to stick to one question. Some models are stubborn. Solution: Interrupt the conversation with a short rephrasing of the rule if necessary. Alternatively, use a stricter persona, like "You are a methodical interviewer who never asks more than one question."

❌ Accepting the Output Without Human Verification

The final document will have an "AI-ish" tone, which some experts dislike. However, the factual content should be correct. Solution: Always have a human review the output for accuracy. If the tone matters, ask the LLM to rewrite the document in a more natural style, or perform a second interview focusing on style.

Summary

The interrogation method transforms the standard prompt engineering workflow by letting the LLM interview you (or another expert) to build or validate context. It is especially useful for people who find writing documents difficult but can articulate information in conversation. By insisting on one question at a time and providing clear scope, you can generate detailed, accurate context documents without manual drafting. The same technique can also be used to review existing documents through dialogue, potentially uncovering issues that a human might skim over. While the output may carry a synthetic flavor, it retains the essential facts—and that is often better than no documentation at all. Use this approach whenever you need to capture expert knowledge efficiently.