AI Support Triage: A Human-Reviewed Pilot

An AI support-triage pilot can explore whether a suggested category and a draft reply help a person handle incoming requests. It needs a clear boundary: preparing a draft is different from resolving a ticket or replying to a customer. This guide describes a proposed workflow and ways to test it, not results from a deployed CompileSquad customer system.

Define a narrow intake workflow

Choose one inbox or form, one destination for human review, and a small list of categories such as access, billing and general enquiry. Write down which request types are out of scope before connecting any apps. For a first pilot, keep the existing support process available so a failed run does not leave the customer without a route to help.

1. Receive: assign a request ID and check that the required fields are present. Start with synthetic or suitably redacted examples.
2. Suggest: ask the model for a category, a short reason and a draft based only on approved reference material. An unavailable answer should produce a hold reason, not an invented policy.
3. Validate: check that the output uses allowed categories and has the expected fields. A plausible-looking draft can still be wrong, so format checks do not replace review.
4. Review: show the original request, relevant reference material, suggestions and any flags to a named person. Let that person approve, edit or hold the draft.

In CompileSquad’s proposed AI workflow pilot, customer sending and downstream business actions are outside scope. The reviewer uses the normal support system to send any final reply after checking it. A pilot should not promise refunds, reset access, change account records or mark a ticket resolved automatically.

Test usefulness before measuring speed

Use examples you can score. For an agreed set of 20 tests, a useful starting mix is eight routine requests, four ambiguous or incomplete requests, four sensitive requests and four workflow failures. Agree the exact examples and expected outcomes with the person who owns support; this is a suggested test design, not a claim about a completed deployment.

For example, a synthetic request says, “I cannot sign in after changing my phone.” The expected result might be an access suggestion and a draft pointing to an approved recovery guide. It must not ask for a password or one-time code, claim the account is restored, or send a reply. A reviewer should be able to hold it if identity or policy checks are needed.

Record whether the category was useful, whether the draft was supported by the reference material, how much editing it needed and whether the expected review or hold state was reached. Measure review time against the same examples handled manually if time saving matters. Include retries and correction work in that comparison; a quicker draft does not necessarily mean a quicker or better response.

Review, escalation and failure controls

Decide which requests go straight to a person. Suggested examples include a refund dispute, suspected account compromise, a legal threat, a request for personal data or unclear instructions. Do not treat a model’s confidence score as proof that the answer is correct.

• Keep input separate from instructions. Treat messages and attachments as untrusted content, including text that asks the workflow to ignore its rules or reveal information.
• Hold on failure. Missing fields, invalid model output, a model timeout or an unavailable review destination should stop that run and leave an actionable error for the owner. Do not silently substitute approval.
• Limit access. Connect only the data and permissions the agreed test needs. A draft-only pilot should not need permission to send customer messages or change accounts.
• Avoid duplicate work. Use request IDs and test how retries are handled before allowing them against live data.

Agree a small event record: request ID, timestamps, workflow version, suggested category, review status and error reason. Record the reviewer’s decision and identity only where the chosen app can reliably provide them. Avoid copying passwords, tokens or unnecessary customer text into logs; agree who can see records and when they are deleted. Logs can be missing or incomplete and do not establish why a model produced an answer. If a required event cannot be recorded, hold the run for investigation.

These controls reduce specific risks but cannot eliminate all errors, data exposure or missed escalations. The approval and hold guide explains how to make review a testable part of the workflow.

Agree acceptance and next steps

Before implementation, agree what counts as accepted: every test has an expected outcome; no customer message is sent by the pilot; invalid or sensitive examples reach the agreed hold path; and a reviewer can inspect and decide on the draft. Record any failed test, fix it and rerun the affected checks before handover. Passing a small test set supports that scoped handover, not a claim that all future requests will work.

The homepage workflow example is simulated and has no connected model or business systems. It illustrates the review concept; a working integration depends on the selected tools, access and agreed scope. See the pilot scope and pricing, or describe one support-intake process to discuss feasibility with CompileSquad.