Google announced on November 9, 2026, that MaxText v0.2.4 supports long-context training on Cloud TPU through context parallelism: the sequence is split across chips, which exchange or stream the key/value (KV) data needed for attention. That keeps one chip from having to hold the full sequence’s activations, but the four approaches trade memory against communication and model-head constraints. Google’s announcement describes the feature as available in v0.2.4.

Context parallelism matters when a training sequence is too long for one chip to handle efficiently on its own. Each chip processes part of the sequence; the system then coordinates the keys and values that attention needs. The practical question is how those KV blocks move, and how much of them each chip must keep in high-bandwidth memory (HBM).

Start by asking whether the gathered KV data fits in HBM. That points you toward the simplest approach or toward a ring-based strategy.

How the four strategies move KV data

An animation tracing Q and K/V blocks across four devices through A2A and P2P exchanges.

Google describes four options. All-gather collects KV data across chips; A2A/Ulysses redistributes token and attention-head shards; P2P/Ring Attention streams KV chunks around neighboring chips; and Unified Sequence Parallelism (USP) combines A2A and P2P.

StrategyHow KV data movesStated advantageConstraint
All-gatherEach chip gathers keys and values from the other chips at every layer.The simplest option when the gathered KV data fits in HBM; Google says it was fastest or near-fastest in its runs when it fit.Memory scales with the full sequence. Causal masking gives later tokens larger prefixes, so the workload needs balancing.
A2A/UlyssesRedistributes token and attention-head shards before attention, then rotates the output back.Reduces per-chip KV memory as the context-parallel factor increases and can use ordinary single-chip attention kernels after redistribution.The factor is limited by the available attention or KV heads. The all-to-all waits for completion rather than overlapping with attention computation, and Google says this approach does not support multi-query attention.
P2P/Ring AttentionSends KV chunks between neighboring chips around a ring while attention computation proceeds.Overlaps communication with computation, without any chip holding the full sequence.Requires load balancing; small chunks can reduce local kernel efficiency, and a longer ring adds communication hops.
USPCombines all-to-all and ring communication across two axes of the TPU chip mesh.Uses longer chunks with fewer ring hops; the ring can extend beyond the all-to-all factor.The all-to-all factor remains limited by the model’s KV-head count, and the two communication patterns must be mapped onto the TPU mesh.

Google’s selection guidance follows those trade-offs: use All-gather when the gathered KV data fits in HBM; consider A2A for a smaller chip count, up to the model’s KV-head limit; and use a ring-based approach when gathered KV data no longer fits. Google recommends USP when the model’s KV heads allow an A2A factor of at least 2. It identifies P2P alone for multi-head latent attention models such as DeepSeek, or when the head count does not divide evenly.

The four strategies use Tokamax Splash Attention. Google says its kernel skips fully masked blocks and handles packed sequences without padding.

The animation traces Q and K/V blocks across four devices as A2A and P2P paths redistribute data and produce outputs.

What Google reports from its TPU runs

Google reports training sequences up to 14 million tokens across TPU 5p and TPU 7. It also reports 70% Model FLOPs Utilization (MFU) for Llama 3.1 8B on TPU 5p at sequence lengths up to 4 million tokens, and 50% MFU on TPU 7 at sequence lengths below 6 million tokens.

For a separate 14-million-token comparison, Google reports 48.7% MFU on TPU 7 versus 37.1% on TPU 5p in 64-chip runs. Google says TPU 7 has twice the HBM per chip of TPU 5p. These figures describe the configurations and sequence lengths in Google’s reported runs.

Configuring context parallelism and managing memory

Google says MaxText v0.2.4 is available through PyPI, Docker images, and GitHub main. Its sample recipes cover Llama 3.1 8B on TPU 5p and TPU 7. To configure a run, choose a value for context_parallel_strategy and set the context-parallel size. Context parallelism can also be combined with data and tensor parallelism.

When memory pressure remains, Google recommends offloading saved tensors to host memory in this order: attention outputs, optimizer states, layer inputs, then parameters. Rematerialization—recomputing values rather than keeping them in memory—comes last.