Cisco’s approach to event reordering sorts buffered records by event time within each session; Apache Flink watermarks signal progress in event time. They address related timing problems, but a watermark is not the same mechanism as Cisco’s explicit per-session sort.
Why event order matters in Cisco’s pipeline
Cisco’s network-connection pipeline reads events from Kafka and evaluates steps such as auth, assoc, dhcp and dns. Those records can arrive in a different order from the order in which the connection events occurred. Cisco says out-of-order arrivals could lead its original application to report successful connections when the sequence did not support that result.
Cisco also says the original path accessed RocksDB state and registered a timer for every incoming event, adding work to the processing hot path. Its revised design changes how the application collects and orders events before parsing a connection sequence.
Event time, processing time and watermarks
Event time is when an event occurred. Processing time is the system time on the machine processing it. A watermark signals progress in event time, allowing a Flink operator to advance a time-dependent computation. Apache Flink’s time concepts describe watermarks and explain that handling out-of-order events in event time involves waiting for a finite period.
Cisco describes a different, more specific operation: it gathers records for a session and sorts them by their event timestamps before parsing the sequence. A watermark marks progress for time-based processing; it does not, by itself, perform that per-session sort. The distinction is useful when choosing a mental model for the design: one concerns event-time progress, the other explicitly arranges buffered session records.
How Cisco buffers and recovers session events
Cisco says its revised implementation uses a KeyedProcessFunction with a HashMap for each session. A custom comparator orders the buffered records by event time, rather than by arrival or processing time, before the application parses them.
The buffer sits on the JVM heap outside Flink-managed state, so Cisco added controls to limit its growth. The company describes a custom three-minute idle timeout to evict inactive sessions, timer debouncing, compact POJOs, and parsing fields once when events enter the pipeline.
At checkpoints, Cisco copies the buffer into Flink ListState and restores it after a restart. Flink’s state documentation describes keyed state as scoped to the current key. Cisco also notes that recovering application state alone does not ensure end-to-end exactly-once output: the sink must handle replayed output appropriately.
What Cisco reports about production operation
Cisco says it ran the revised version alongside the original on production data for five months. Feature flags controlled which results users saw, while the original version remained the default during that comparison. For the cutover, Cisco changed the default topics and removed branching logic from the UI, dashboard server and ClickHouse query layer.
Cisco reports that this network-connection pipeline handled 200,000 messages per second and ran on Kubernetes on Amazon EKS. The rate is Cisco’s reported result for its own pipeline.
