AI content checks in Runtime
Nimi runs content checks directly in the Runtime on the user's device. Every app that calls text.generate gets the same inspection for prompts, generated output, and complete tool-call arguments—whether the request runs on a local model or a cloud provider.
Suppose a research app reads a file containing a private key and includes it in the next prompt, or a tool returns a token that gets fed back into the conversation. If that content matches one of the rules below, Runtime rejects the request before it reaches the model. Model output is checked too. Apps keep their own workflows without each maintaining a separate scanner.
Developer preview
Content checks are available in current Runtime source builds. For prebuilt downloads, check the release notes for your version. Apps do not need new request fields, registration steps, or configuration to use them.
How local inspection works
Runtime already resolves caller identity, holds provider credentials, and dispatches model requests. Running deterministic format checks in the same place means protection does not depend on whether a model obeys safety instructions in a prompt.
Inspection runs entirely on the user's machine with no network dependency and no external scanning service. When an app uses a cloud route, requests that pass inspection are still sent to the selected model provider.
Responsibilities stay distinct across the stack:
| Responsibility | Owner |
|---|---|
| Inspect business text in model inputs, outputs, and complete tool arguments | Runtime AI |
| Verify identity, permissions, credential custody, and resource access | Existing Runtime resource services |
| Execute tools, manage multi-turn state, render output, and obtain business approval | Your App host |
Passing a content check does not grant permission, and model output such as approved: true never establishes authorization. Integration business results and Delegation approval policies work as before.
What gets checked and when
Current inspection covers text.generate in SYNC and STREAM execution. Candidate generation and RuntimeAgent calls are included whenever they use that path. Other AI capabilities, raw image/audio/video bytes, opaque continuity payloads, and calls outside text.generate are not covered.
| Stage | Checked content | Result of a match |
|---|---|---|
| Model input | Normalized business text and JSON string values: system text, selected history and summaries, tool descriptions and schemas, tool results, and visible media-reference strings | Refused before model dispatch, media materialization, request-body capture, or new Job creation |
| Streaming output | Generated text and visible reasoning summaries, checked incrementally | Stops undelivered output and reports failure through the existing stream and Job path |
| Tool call | Complete arguments after decoding JSON escapes into actual values | Refused before forwarding the tool call to the App |
| Final result | Naturally complete text, visible summaries, and tool arguments | Refused before flushing final buffered output, completing normally, or storing a successful result |
Normal output keeps its existing chunking and buffering behavior. Inspection adds no timers, full-step buffering, or new output-length limits. Because a transport chunk boundary is not always a value boundary, Runtime can confirm some matches only when a delimiter or the end of a logical item arrives. Normal text continues to stream in the meantime.
In streaming mode, text already delivered before a match is confirmed cannot be recalled. Content checks help catch accidental credential exposure, but they do not guarantee zero leakage, act as an operating-system sandbox, or prevent all prompt injection, malicious model behavior, or data exfiltration.
Current rules
The initial rules target specific, well-defined credential formats rather than blocking every sensitive word or high-entropy string.
| Rule | Match condition | Inspection mode |
|---|---|---|
pem-private-key | Complete, unencrypted PEM private keys that parse as PKCS8 (PRIVATE KEY), PKCS1 (RSA PRIVATE KEY), or SEC1 (EC PRIVATE KEY) | Complete values only |
github-classic-pat | ghp_ followed by exactly 36 ASCII letters or digits, with identifier boundaries on both sides | Complete values and incremental text |
For github-classic-pat, ASCII letters, digits, underscores (_), and non-ASCII bytes continue an identifier, so a match only triggers when the token stands on its own. Matching is purely structural—Runtime does not verify whether a credential was actually issued or remains active.
Other token formats, encrypted or OpenSSH private keys, and arbitrary encodings are not covered. Placeholders and ordinary technical text—such as token: string, password: <redacted>, empty values, hashes, standalone PEM headers, public keys, certificates, generic Bearer wording, sensitive field names, high-entropy strings, or prompt-injection phrases—do not trigger refusal on their own.
Local and cloud routes apply the same rules. A local request containing a private key in one of the covered formats is refused even for legitimate analysis. You can supply a redacted copy through your App’s existing input controls; Runtime never silently rewrites your input.
Handle a refusal in your App
When Runtime blocks content, it returns gRPC FailedPrecondition with reason AI_CONTENT_FILTER_BLOCKED (and exposes ai-content-filter-blocked on the protected App carrier). Branch on the typed error or event from your SDK rather than parsing user-facing error messages.
- Input refusal: The call fails before Runtime creates a Job or opens an output stream. Handle it as a call error; there is no stream
failedevent or successful result. - Output refusal: An active stream reports failure through the existing
failedpath (and records Job failure) when the connection can still deliver it. Treat the entire model turn as failed, and do not execute any tool calls received earlier in that failed batch. - User guidance: Tell the user that content filtering blocked the request or response, and ask them to review and redact the input or selected context before submitting again. Do not automatically retry, replay tools, or switch models, providers, or local/cloud routes.
A clear user-facing message is:
This request or response did not pass the content check. Review the input and selected context, then remove or redact restricted content before submitting again.
Cancellation, timeouts, permission errors, Runtime shutdown, and transport failures keep their own error codes and priority. Content-check diagnostics record the rule, stage, and related information without storing the matched secret. Known usage may remain in the existing Job record; missing usage does not mean a failed call was free.
Matching source builds of Home and the standard Kit Shell add one nonmodal content-check notice while preserving the App's original failure result. Safe diagnostics contain only the reason and available fixed rule/stage values. A format match means the input may contain a credential; it does not establish validity, malicious model behavior, or a leak. Users can edit their input and continue other tasks without submitting a report or disabling checks. A RuntimeAgent rejection in a later model round does not mean the entire user operation stayed on the device.
History and conversation recovery
Runtime never globally rewrites or scrubs stored App history, and a diagnostic event does not mean earlier records were cleaned. Old turns remain readable, but any history or summary selected into a new request is checked again. If matched material stays in the selected context, even a harmless follow-up question will be refused.
Apps that own their context use their existing input and context editors. Reopening a RuntimeAgent conversation reuses its anchor and does not clear it.
Current source builds add a Conversation context control in Agent Center's State & memory section. After reviewing its scope, the user can select where future history starts and optionally stop referencing supplemental Memory and retrieval in that Conversation. Existing messages stay readable, and new messages use the same anchor. This choice affects the same Agent in every App and remains in effect after reopening or restarting until the user changes it again. It does not delete Memory, turn Memory off, or send the current draft.
The operation requires existing agent.configure access. A chat-only App can hand the known anchor to Home with open-agent-center; Home resolves the target under its own current management access. A busy Agent or stale revision prevents the change. Restoring the full range also restores normal checking of earlier material, so it can cause refusal again. Required character or App instructions must be corrected by their source owner; changing history cannot remove them.
The new notice and context controls require matching source-built SDK, Kit/native, Home and Runtime. The Windows Home → Desktop-supervised Lab journey has been verified with DeepSeek deepseek-v4-pro, including normal streaming, a tool round trip, stopping, input refusal, and continuing the same RuntimeAgent conversation after an explicit context change and a full restart. This is an unreleased developer preview; it does not establish coverage for older packages, other operating systems, or arbitrary App hosts. Check the release notes for a prebuilt installation.
Memory keeps its existing rules for accepting stored content. If it rejects a proposed correction, the original memory stays unchanged and the proposed text remains editable. Choosing not to reference supplemental context does not disable those rules or grant an exception to content checks.