Vals’ CyberOffense benchmark reported that DeepSeek V4.1 Flash 1017 completed an average of 16.3 of 21 objectives in Nova and 21 of 31 in Lighthouse. GPT-6 Sol Daybreak averaged 15.7 of 21 in Nova and completed 27 of 31 in Lighthouse. Both tests used isolated, simulated corporate networks, not live company systems. Vals’ results and methodology explain how the scores were produced.
Scores across the two environments
| Model | Nova objectives | Lighthouse objectives |
| DeepSeek V4.1 Flash 1017 | 16.3/21, average of three runs | 21/31 |
| GPT-6 Sol Daybreak | 15.7/21, average of three runs | 27/31 |
| GLM 5.3 Flash | 5/21 | 5/31 |
Vals gives Nova and Lighthouse equal weight in its overall score, averaging each environment’s share of completed objectives. Applying that method to the reported figures gives about 72.7% for DeepSeek and 80.9% for Daybreak. The range-level results show where the comparison shifts: DeepSeek’s reported average was slightly higher in Nova, while Daybreak completed more Lighthouse objectives.
Vals lists the GLM result specifically for GLM 5.3 Flash. Its scores are 5 of 21 objectives in Nova and 5 of 31 in Lighthouse.
To understand a benchmark percentage, check the objectives, environments and scoring rule behind it.
DeepSeek V4.1 Flash 1017 is an open-weight model. For broader context on Chinese open-weight models and closed systems, see our earlier analysis.
How Vals ran and scored the test
Nova is a simulated, multi-domain Windows network with 21 objectives. Lighthouse spans two simulated corporate networks and has 31 objectives. Vals ran the Pi agent on an isolated Kali Linux machine. The agent could reach the model provider and the test range, but not the open internet; each run received a fresh copy of its range.
Vals checked recorded command output for hidden markers tied to the objectives. An agent’s own statement that it had succeeded did not count as verification. Each run had a limit of 500 million cumulative tokens or 48 hours. Vals says the runs in this set ended before reaching either limit.
The scores describe progress through those simulations. Vals says no model completed Lighthouse’s final objectives, and only one run reached Nova’s final stage.
What the scores cover
CyberOffense evaluates multi-step campaigns in simulated enterprise networks. It does not establish that these models compromised real company networks. The controlled setup and hidden-marker scoring make the results a measure of performance on Vals’s defined tasks, under those conditions.
Vals also runs CyberBench, a separate evaluation that tests whether models can reproduce OSS-Fuzz-style crashes with a proof-of-concept input and patch source code. Those tasks and outcomes differ from CyberOffense’s simulated network campaigns, so the two benchmarks’ scores measure different things. Vals’ CyberBench page describes that evaluation.
