Built at Fireworks: Deploy with Microsoft & NVIDIA

CyberNemo

CyberNemo: making open LLMs secure by default THE PROBLEM More and more production code is written by AI, and that code runs inside company infrastructure. Every vulnerability a model writes (path traversal, header injection, cross-site scripting, a C buffer overflow) is an opening for an attacker. Today's coding models are very good at writing code that works, and much worse at writing code that is safe, because nobody tells them to think like an attacker. We measured this on CWEval, a benchmark from Columbia (LLM4Code @ ICSE 2025) whose tasks come with executable attack tests: - Base NVIDIA Nemotron 3.5 Super 120B writes broken or exploitable code in 42% of its answers. - Even the best frontier model we tested, Reflection Beam, does so in 24%. WHO BENEFITS Companies that want an AI coding assistant they can host on their own infrastructure, so their source code never leaves the company, without accepting a less secure model as the price: security-sensitive teams in finance, healthcare, government and enterprise software. Open models give them control and privacy but trail frontier models on security. CyberNemo closes that gap. WHAT THE SYSTEM DOES Given a function stub and a docstring, CyberNemo writes the body so that it works and cannot be exploited, in C, C++, JavaScript or Python. The docstring never mentions security; the model has to spot the risk on its own. - Log injection: a function that formats a log line must strip newlines from the user's message, or an attacker can forge a fake log entry. Every system we tested failed this task, even though all of them produced working code. - Tar slip: a function that unpacks an archive must refuse file names containing "../", or files land outside the target folder. Base Nemotron was 0% secure on this task; CyberNemo with the review harness was 100% secure. APPROACH: TWO PARTS Part 1, secure-by-default fine-tuning (changes the model): 1. Practice tasks: Nemotron 3 Ultra 550B wrote 1,813 synthetic coding tasks (96 vulnerability-type x language combinations), each with a spec, a secure solution, a deliberately vulnerable solution, and functionality and attack tests. 2. Quality check: a task was kept only if, in an offline sandbox, the secure solution passed everything and the attack tests caught the vulnerable one. 3. Leakage filter: removes anything resembling a benchmark task. 4. Self-distillation: Nemotron 120B solved each task with reasoning; we kept only answers that passed every test and removed the reasoning. No frontier-model outputs were used. 5. LoRA fine-tuning on Fireworks: rank 16, one epoch, about 1,000-1,400 examples, roughly $10 per run. Result: +4.9 points of working and secure code, with no loss in functionality. Part 2, security-review harness (wraps the model at answer time): After CyberNemo writes code, the same model reviews it against a generic security checklist (escape HTML, reject CR/LF in headers, validate file paths and URLs, parameterise queries, bounds-check C buffers) and returns the fixed version. The harness never sees the benchmark's tests. Result: another +8.9 points. HOW SUCCESS IS MEASURED - Benchmark: CWEval, 100 human-written tasks (C 31, C++ 21, JavaScript 23, Python 25), held out from training; every training task passed the leakage filter. - Strict scoring: an answer counts only if every functionality test and every attack test passes. Missing code, crashes and timeouts score zero. - Same conditions for every system: same prompts, temperature 0.8, 5-10 answers per task, frontier models in their default reasoning mode with up to 32k tokens. - Statistics: 95% confidence intervals from a paired bootstrap over the 100 tasks. - Latency: the same 30 prompts and 4 concurrent requests for every system. RESULTS (works + secure / works, %) - Reflection Beam 1.0 (501B): 76.2 / 86.9 - Gemini 3.8 Flash: 74.3 / 88.9 - CyberNemo + review harness: 72.4 / 81.1 - Kimi K2.7 Code: 67.0 / 87.8 - CyberNemo (fine-tune only): 63.3 / 86.2 - MiniMax M3.1 Flash: 62.0 / 82.4 - Nemotron 120B base: 58.4 / 86.4 Headline: working and secure code goes from 58.4 to 72.4, +14.0 points (95% CI +5.2 to +23.1). Third of seven systems, statistically on par with Reflection Beam and Gemini 3.8 Flash (neither gap is significant). The share of code that works but is exploitable drops from 28% to 9%, the lowest of any system tested. Fast mode (no reasoning), for IDE completion and agents: 1.1-1.8 seconds per answer at 59.3 works + secure vs 53.2 for the base model (+6.1, significant), 9-17x faster than the leading frontier models. BUSINESS VALUE - Data control: an open model you deploy yourself, so code stays inside your network. - Security close to the frontier: statistically level with the two leading frontier models, with the least exploitable code of any system tested. - Low adoption cost: about $10 per fine-tuning run; the review harness is one extra call that wraps any model. - A reusable recipe: synthetic tasks with attack tests, then verified self-distillation, then a review harness. It transfers to other languages, vulnerability types or in-house coding standards. TRADEOFFS AND LIMITATIONS - Functionality: with the review step, code works 81% of the time, 6-8 points below the top frontier models; the reviewer sometimes over-restricts valid input. - Peak security: Beam and Gemini 3.8 Flash still lead by 3.8 and 1.9 points (not significant). - Latency: the harness doubles inference calls (about 12 s per answer); the big speed advantage applies to fast mode only. - Scope: single functions, four languages, one benchmark. "Secure" means it passed the benchmark's attack tests, not that no exploit exists. - Serving: a LoRA on Fireworks needs a dedicated deployment (4xB300). - Disclosures: only CyberNemo gets the review pass; the harness row covers 869 of 1,000 answers (a billing cap stopped the run; every task is still covered); v3's choice of vulnerability families was informed by error analysis on the benchmark, with no benchmark content used. BUILT BEFORE VS DURING THE EVENT Before: the task generator, verifier, benchmark runners and scorer; early 30B experiments; CyberNemo v1. During: CyberNemo v2 and v3, the security-review harness, all frontier baselines, the speed benchmark, the results tooling, charts and documentation. AI assistance: Claude Code was used as a pair programmer; the team designed the experiments and owns the results. TECHNOLOGY - Fireworks AI: LoRA fine-tuning and dedicated evaluation deployments. - NVIDIA Nemotron: 3.5 Super 120B (the model we tuned) and 3 Ultra 550B (which wrote the practice tasks). - CWEval: the benchmark and scorer (Apache-2.0).