Google Cloud describes the Envoy agentic stack as a networking layer for AI-agent traffic: it can inspect requests, apply token-aware limits, route model and MCP calls, and connect policy checks. Its Google Distributed Cloud air-gapped reference is a more specific design, with distinct component roles and clear limits on the traffic it handles. Google Cloud’s technical description and GDC reference architecture describe the two scopes.
What Google Cloud says Envoy adds for AI-agent traffic
Google Cloud’s proposal focuses on information that can sit inside an AI request body, rather than only in the URL, method or headers that conventional Layer 7 proxies typically use for routing. Its account groups the approach around request parsing, token-aware controls, model and MCP routing, and security-policy callouts.
| Traffic concern | Conventional proxy approach, as Google Cloud describes it | Agent-oriented Envoy capability, as Google Cloud describes it |
| Choosing a destination | Route by URL, method and headers | Inspect request bodies to identify model or MCP details |
| Managing usage | Count requests | Apply token limits and session-level budgets |
| Keeping multi-turn requests together | Balance requests without session context | Use session or cache-prefix affinity when selecting model endpoints |
| Checking tool calls | Route traffic without inline tool-policy checks | Send requests to external policy processors |
The practical shift is from treating each request as a generic network transaction to using details about its model, token use or tool call. Google Cloud presents these as capabilities of its Envoy approach, rather than as results of an independent performance evaluation.
How the proposed traffic controls work
For model requests, Google Cloud says Envoy can parse a request body, identify its target model and translate among OpenAI-, Anthropic- and Gemini-format requests. It also describes extracting token-use information from server-sent event (SSE) response streams, which lets the proxy work with usage reported after inference as well as estimate input tokens beforehand.
The described quotas can cap tokens per minute or per day, with cumulative budgets for a session. Google Cloud says requests that exceed a quota can be throttled before inference, with HTTP 429 as the response. That makes the limit about token consumption, not just how many requests arrive.
For model selection, the proposal uses an Endpoint Picker and can favor a worker with a matching key-value cache prefix, which may help retain reusable context. Google Cloud also describes retrying or falling back to a secondary model pool after a primary-model failure or an HTTP 429 response.
MCP, or the Model Context Protocol, lets an agent communicate with tools through a defined protocol. Google Cloud describes routing MCP requests using JSON-RPC 2.0 request bodies and headers such as Mcp-Method and Mcp-Name. Its account says a mismatch between the headers and request body triggers HTTP 400 and error code -32020.
For policy checks, Google Cloud describes Envoy ext_proc callouts to external processors such as Model Armor, ShieldGemma or local classifiers. It also describes pairing Envoy with Agent Substrate to steer traffic to suspended agent runtimes and control outbound traffic.
For agent traffic, the key design question is whether the gateway can act on the request’s model, token use and tool call—not just its destination address.
Inside the GDC air-gapped reference architecture
Google’s GDC design gives the components separate jobs. Envoy Gateway implements the Kubernetes Gateway API; Envoy Agent Router, formerly Envoy AI Gateway, handles AI-specific request processing. An Envoy proxy works with an external processor, and the architecture presents a unified OpenAI-compatible endpoint. The guide describes staging artifacts in Harbor and running the gateway components on CPU nodes.
The reference has specific prerequisites: Google Distributed Cloud air-gapped 1.16.2-hf1 or later and Kubernetes v1.32.13-gke.400 or later. Its scope excludes MCP agent-to-tool traffic through MCPRoute, managed gateway products, and model deployment or tuning. That means the broader MCP-routing description and this particular GDC design cover different traffic boundaries.
Redis is optional in this architecture. It supplies shared counters when global token-based rate limiting is configured; deployments without that global limit do not require it. Endpoint selection also depends on the backend type: the Endpoint Picker selects model-serving endpoints for InferencePool backends, while AIServiceBackend uses proxy balancing across Service endpoints.
The guide also describes TLS termination, backend credential injection and network policies for this GDC setup. It identifies InferenceObjective as an alpha API and characterizes support for the open-source components as best-effort; model throughput and latency remain bounded by the model servers.