Agent File Understanding (Vision & Document Text)

Before this capability, an agent could see that a customer had attached a file, but not what was in it — it had to ask the customer to describe a screenshot or type out what a document said. agent_file_understanding closes that gap for web chat: when it’s on for a tenant, the agent can look at a customer’s photo directly and read the text pulled out of an attached document, and answer using that content in the same turn.

The flag ships off. It’s part of DEFAULT_DISABLED_FLAGS, so a tenant with no override row reads as disabled rather than silently on — nothing changes for a tenant until someone turns it on for them.

What the flag enables

With agent_file_understanding on for a tenant:

• Photo / image — A vision part pointing at a signed URL for the image — the model looks at the actual picture, the same way attachment-understanding-service already does for its own vision call.
• Document — The extracted text pulled from the file, not a summary of it.

Both only apply to web chat, and only to the attachment on the message the customer just sent — the turn currently being answered.

With the flag off, nothing changes: the agent’s instructions still tell it that it cannot see images, and that statement stays true until the flag is turned on for that tenant.

Images: what’s supported

An image is sent as a vision part, not raw bytes — no image data is copied into this process, only a reference to where it already lives.

Supported formats are jpeg, png, and webp. A GIF is dropped before it’s sent rather than forwarded and rejected downstream. If the model resolved for the conversation isn’t marked as vision-capable (or was reached through an explicit model pin, which carries no tier), the image is dropped the same way — the agent answers from text alone rather than risk failing the whole turn on a provider error.

If an image ends up dropped for any of these reasons and the customer’s message had no other text (a bare screenshot with nothing typed), the agent isn’t left with an empty message — it receives a short placeholder note in its place, so it knows to say it’s answering without having seen the picture rather than guessing or contradicting the customer.

Documents: extracted text, not a summary

A document is sent as the text already extracted from it during attachment processing — the same text that’s clamped to 40,000 characters and stored on the attachment once extraction succeeds. It is not the short AI-generated summary that’s shown elsewhere for that attachment.

This distinction is deliberate: a summary is a handful of bullets chosen before anyone knew what the customer would ask. Answering from the summary means the agent can only ever look as far as whatever the summarizer happened to pick, and it would still confidently answer even when the real question needed something the summary left out. The extracted text lets the agent answer a question it couldn’t have anticipated, which is why documents are read as text and not summaries.

Because the extracted text is stored going forward but nothing was backfilled, a document attached before this capability shipped for a tenant has no stored text and simply sends nothing — the agent behaves as it did before the flag existed for that one attachment.

Current limits

• Only the current turn’s attachment is reachable. A photo or document on an older message in the conversation is invisible to the agent — this capability only composes the file from the message being answered right now.
• Documents are read as extracted text, never a summary. If extraction hasn’t produced text for a document, the agent gets nothing for it, even if a summary exists.
• Web chat only. This does not apply to voice or any other channel.
• A specialist in passthrough mode does not receive the file. The file is composed onto the orchestrator’s turn; a passthrough specialist is not one of the paths that reads it, so a customer handed to passthrough should not assume the specialist can see what they attached.
• Unreadable or unsupported files remain invisible, and the agent is told as much when a file is present on the turn — it will say it may be missing something rather than claim to have seen a file it didn’t get.

Verifying or requesting the flag for a tenant

agent_file_understanding is a single flag for the whole capability, enabled per tenant through the existing tenant flag-override mechanism (the same mechanism used for other tenant-scoped flags) — there is no dedicated Settings toggle for it yet. If a customer reports the agent isn’t reading their attachment, or support needs to confirm whether a tenant has this on:

1. Confirm the conversation is web chat — this capability does nothing on other channels.
2. Confirm the attachment is on the message currently being answered, not an earlier one in the conversation.
3. If it’s a document, confirm the attachment finished extraction (text_extracted) rather than failing to extract — a document with no extracted text sends nothing regardless of the flag.
4. To check or change whether the flag is on for a tenant, escalate to engineering/support ops, who can look at or set the tenant’s override row. There is no customer-visible or admin-visible setting that reflects this today.

What the agent is told

When a turn carries a usable file, an additional block is appended to the agent’s prompt — only for that turn, and only when a file is actually present. It tells the model:

• An image may have been dropped and replaced by a short note (if the model isn’t vision-capable, or the format wasn’t supported).
• A document arrives as extracted text, so the agent should not claim to have “seen” the document itself.
• Files on earlier messages, and files that couldn’t be read, remain invisible to it.

With the flag off, none of this block is added, and the platform-level instruction that the agent cannot see files stays exactly as it was.