Agent File Understanding (Vision & Document Text)

Before this capability, an agent could see that a customer had attached a file, but not what was in it — it had to ask the customer to describe a screenshot or type out what a document said. File understanding closes that gap: when it’s on for a tenant and channel, the agent can look at a customer’s photo directly and read the text pulled out of an attached document, and answer using that content in the same turn.

There are two independent tenant flags:

• agent_file_understanding enables file understanding for web chat.
• agent_file_understanding_email enables file understanding for email.

Both flags ship off. They’re part of DEFAULT_DISABLED_FLAGS, so a tenant with no override row reads as disabled rather than silently on — nothing changes for a tenant until someone turns the relevant flag on for them. Turning on web chat file understanding does not turn on email attachment processing, and does not enable the extra vision or document spend that can come from reading email attachments.

What the flag enables

With the relevant flag on for a tenant:

• Photo / image — A vision part for the image. The platform first tries to send the 512px preview bytes inline. If there is no stored preview, it sends the original bytes when the recorded file size is at most 3 MB. If neither inline path works, it signs a link — for safe internal images only. If none of those work, the image is dropped.
• Document — The extracted text pulled from the file, not a summary of it.

agent_file_understanding applies to web chat. agent_file_understanding_email applies to email. They are separate switches: enabling one channel does not enable the other.

On both channels the platform searches the last 10 inbound messages, not only the one being answered. Web chat depends on that: the widget sends the file on its own message and the customer types the question on the next one, so the answered message rarely holds the file. Email works the same way.

With the relevant flag off, nothing changes for that channel: the agent’s instructions still tell it that it cannot see images, and that statement stays true until the matching flag is turned on for that tenant.

Images: what’s supported

An image is sent as a vision part. The platform resolves that vision part in this order:

1. Send the 512px preview bytes inline when a stored preview is available.
2. Otherwise, send the original image bytes inline when the recorded file size is at most 3 MB. A row with no size recorded, or a preview read that failed for any reason other than the file being absent, skips this step.
3. Otherwise, use a signed link — only when the image is a safe internal file.
4. Otherwise, send nothing for that image.

Supported formats are jpeg, png, and webp. GIFs are not inlined; a GIF is dropped before it’s sent rather than forwarded and rejected downstream. Migrated external images that cannot be inlined are also dropped rather than sent as unreadable raw URLs.

Model capability works the other way round from what you might expect. An image is withheld only when the resolved model is explicitly tagged as having no vision — and only one provider reports that capability at all, so absent capability data means send the image. If the provider then refuses it, the turn is retried once without the image rather than failing. A model reached through an explicit pin is treated exactly like any other: the model’s own record decides, and no plan or tier is consulted.

On those two paths — an explicitly non-vision model, or a provider refusal — every image part is replaced in place by a short placeholder note, whether or not the message also carried text, so the agent knows to say it is answering without having seen the picture. An image dropped earlier for an unsupported format or an unresolvable file produces no part at all, and therefore no note.

For email, inline images that the email body actually displays with a cid:... reference are treated as displayed email content rather than customer-provided attachments for the agent to read separately. Real customer attachments that happen to have a Content-ID are kept unless the email body actually references them through cid:.... That means Gmail or Apple Mail attachments with unreferenced Content-IDs can still be read by the agent when agent_file_understanding_email is on.

Documents: extracted text, not a summary

A document is sent as the text already extracted from it during attachment processing — the same text that’s clamped to 40,000 characters and stored on the attachment once extraction succeeds. It is not the short AI-generated summary that’s shown elsewhere for that attachment.

This distinction is deliberate: a summary is a handful of bullets chosen before anyone knew what the customer would ask. Answering from the summary means the agent can only ever look as far as whatever the summarizer happened to pick, and it would still confidently answer even when the real question needed something the summary left out. The extracted text lets the agent answer a question it couldn’t have anticipated, which is why documents are read as text and not summaries.

Because the extracted text is stored going forward but nothing was backfilled, a document attached before this capability shipped for a tenant has no stored text and simply sends nothing — the agent behaves as it did before the relevant flag existed for that one attachment.

Current limits

• Web chat and email are controlled by separate flags. agent_file_understanding enables web chat only. agent_file_understanding_email enables email only. Turning on web chat does not enable email attachment processing or the vision/document spend associated with email attachments.
• Both flags are off by default. A tenant needs an explicit override for each channel where file understanding should be enabled.
• The window is the last 10 inbound messages, on both channels. A file the customer sent a few messages ago is still reachable; one older than that window is not. The agent is explicitly told never to refuse a file just because it arrived earlier in the conversation.
• Documents are read as extracted text, never a summary. If extraction hasn’t produced text for a document, the agent gets nothing for it, even if a summary exists.
• This does not apply to voice or any other channel beyond the enabled web chat or email path.
• A specialist does receive the file, passthrough included. The same parts composed onto the orchestrator’s turn are handed to the specialist on every invocation — deliberately, because on the passthrough path the specialist’s reply is what the customer reads, and a file that reached only the orchestrator would be invisible on the reply that ships.
• Unreadable or unsupported files remain invisible, and the agent is told as much whenever at least one usable file part made it onto the turn — it will say it may be missing something rather than claim to have seen a file it didn’t get.
• Email inline cid:... content is filtered differently from real attachments. Attachments with a Content-ID are only removed when the email body actually displays them with a matching cid:... reference. Unreferenced Content-ID attachments from clients such as Gmail or Apple Mail can still be read when the email flag is on.

Verifying or requesting the flag for a tenant

File understanding is enabled per tenant through the existing tenant flag-override mechanism (the same mechanism used for other tenant-scoped flags) — there is no dedicated Settings toggle for it yet. If a customer reports the agent isn’t reading their attachment, or support needs to confirm whether a tenant has this on:

1. Confirm the channel:
   For web chat, check agent_file_understanding.
   For email, check agent_file_understanding_email.
2. Confirm the matching channel flag is enabled. Enabling agent_file_understanding for web chat does not enable email attachment processing.
3. Confirm the attachment is on one of the last 10 inbound messages in the conversation. Older than that and it is out of the window on either channel.
4. For email, if the attachment has a Content-ID, confirm whether the body actually references it with cid:...; unreferenced Content-ID attachments can still be read when the email flag is on.
5. If it’s a document, confirm the attachment finished extraction (text_extracted) rather than failing to extract — a document with no extracted text sends nothing regardless of the flag.
6. To check or change whether either flag is on for a tenant, escalate to engineering/support ops. The flags are visible and overridable on the superadmin Feature Flags page; there is no tenant-admin setting that reflects them.

What the agent is told

When a turn carries at least one usable file part, an additional block is appended to the agent’s prompt — only for that turn. It tells the model:

• An image may have been replaced by a short note saying it could not be shown.
• A document arrives as extracted text, so the agent should not claim to have “seen” the document itself.
• A file often arrives earlier in the conversation than the words being answered, so it must never refuse a file just because it came earlier — but a file nothing could be read from, and one older than the recent-message window, are not there at all.

With the relevant flag off, none of this block is added for that channel, and the platform-level instruction that the agent cannot see files stays exactly as it was.