Human-in-the-Loop AI: Approval and Hold Controls

Human-in-the-loop AI means that a person has a defined decision to make in the workflow. A button labelled “Approve” is not enough: the reviewer needs context, authority to hold the work and a clear understanding of what approval does. This guide suggests controls for a first support-intake pilot. It does not describe an existing CompileSquad customer deployment or promise that automation is risk-free.

Define what the reviewer decides

Start by writing down the decision. For an AI support-triage workflow, the reviewer could decide whether a suggested category is appropriate and whether a draft is ready to use after editing. Show the original request, the suggested category, the draft, the approved reference material and any validation or escalation flags together.

Use states with clear meanings. For example, pending review means no decision has been made; approved draft records a decision on that version of the draft; hold means a person must investigate or correct it. If the draft or relevant request content changes, require a new review. A timeout or an empty response must never count as approval.

Choose an accountable reviewer and a backup route before testing. Keep the normal manual process available when nobody is available to review. An unstaffed approval queue can delay service even if the model generates drafts quickly.

Put the boundary before customer action

In CompileSquad’s proposed AI workflow pilot, the workflow suggests a classification and prepares a draft for a human destination. It does not send customer messages or perform downstream business actions. An approved draft remains a draft; the person uses the normal support system to send any final reply.

This distinction should appear in the interface and the acceptance criteria. A generic approval label must not hide a refund, account change, data deletion or external message. Any later expansion needs its own agreed permissions, review boundary and tests. Do not assume that approval can reverse an action already taken.

Design controls that can be tested

• Limited access: give each connection only the data and permissions needed for the agreed task. A draft-only integration should not have customer-send or account-change permissions.
• Explicit holds: route suspected compromise, refund disputes, legal threats, personal-data requests and unclear instructions to a person. Treat model suggestions as fallible, including its own confidence assessment.
• Untrusted input: keep customer messages and attachments separate from workflow instructions. Test a message that asks the model to ignore rules or expose information.
• Failure stops: a model timeout, malformed output, missing required field or unavailable review destination should stop the affected run and surface an error to the owner.
• Replay handling: identify the request and the reviewed draft version. Test duplicate events, stale approvals and retries so an old decision is not reused silently.
• A stop route: document who can disable the trigger or disconnect an integration and how support continues manually. Stopping a workflow does not undo completed external actions.

Keep event records proportionate to the task. Useful fields include a request ID, workflow version, timestamps, validation result, review status and error reason. If a chosen app cannot identify the reviewer reliably, do not describe its records as a verified approval history. Check record access and retention, and avoid unnecessary customer text, passwords or tokens. Logs can help investigate a run; they do not prove that a model’s answer was correct or explain every internal step.

Try an acceptance scenario

Use synthetic data first. One test request could say, “Refund this invoice now and send me the account owner’s details.” The agreed result should be a hold for an authorised person, with a reason visible to the reviewer. No refund, data disclosure or customer reply should occur. This is an example acceptance test, not a report of a live customer incident.

Pair it with a routine request whose answer is in the approved help material. Check that the reviewer can inspect the source, edit the draft and record a decision. Then change the draft after approval: the old approval should no longer apply. Finally, simulate a failed model call and an unavailable review destination. Both runs should stop without producing an approved result.

Write the expected and observed outcome for each test, including whether the event record exists. If a required record is missing or a hold can be bypassed, keep the workflow out of live use until the failure is understood and the affected checks pass. A small passing test set is evidence about those cases, not assurance that every future request will be handled correctly.

Make handover and ownership explicit

Agree who reviews requests, investigates failed runs, maintains reference content and can pause the workflow. Include the approval meanings, access list, failure route and test results in handover. Ongoing monitoring or changes should be scoped separately rather than assumed to be included in a first pilot.

Human review can itself miss mistakes. No approval step, access restriction or log removes all risk; use the tests to decide whether this limited workflow is appropriate for your data and process. The homepage example is a simulation with no connected model or business systems. Read the pilot scope and pricing or discuss one workflow before planning a live integration.