SoTA Feed — Every open-weights release from the labs that matter

Ad: Read SoTA Feed without this slot — ad-free site plus a personal ad-free feed URL $3/month

Nemotron-Labs-Sigma-distill-spec-3B

Nov 9, 2026 · NVIDIA · license: other · view on Hugging Face ↗
8.5 GB · 4.2B dense

Nemotron-Labs-Sigma-distill-spec-3B

Model Overview

Description

Nemotron-Labs-Sigma-distill-spec-3B is a 3 billion-parameter continuous diffusion language model trained for self-speculative decoding. It was produced by Phased Distribution Matching Distillation (Phased DMD) from a multiscale SUSReg-regularized 3B teacher of the same Nemotron-Labs-Sigma architecture family (MS-SUSReg-3B), then continued with an LK acceptance objective ("X25") so that its one-call diffusion draft is accepted more often by its own AR path.

Unlike the -distill-* siblings, which are deployed as pure few-step blockwise diffusion samplers, this checkpoint is trained so that its diffusion head drafts a block of 32 tokens that its own autoregressive head then verifies. Each drafting round costs 1 diffusion call, and accepted tokens are emitted without a further forward pass. Both heads are therefore required and both are shipped in this repository.

Self-conditioning is off: the underlying distilled student was trained SC-off, and the one-call draft never self-conditions, as in serving.

This model is ready for commercial or non-commercial use.

License/Terms of Use

Use of this model is governed by the NVIDIA Nemotron Open Model License.

Release Date

References

Model Architecture

Input

Output

Software Integration

Model Version(s)

Self-speculative decoding

The diffusion head drafts a block, the model's own AR head verifies it, and the returned text is exactly the greedy autoregressive output -- acceptance changes only how many tokens each round commits, never which tokens. Both heads ship in this repository and both are required.

out_ids, stats = model.generate_speculative(
    input_ids,
    max_new_tokens=256,
    return_stats=True,        # rounds, accepted, mean_accepted_length
)
print(tokenizer.decode(out_ids[0, input_ids.shape[1]:], skip_special_tokens=True))
print(f"[mean accepted length = {stats['mean_accepted_length']:.2f}]")

draft_width and diffusion_calls_per_round default to this checkpoint's trained contract (see generation_config.json); drafts never use self-conditioning or guidance, matching how the model was trained and served.

Evaluation Results

Self-speculative decoding is measured by **mean accepted length (AL) per verification round, excluding the bonus token** (higher is better; it is a throughput measure, not an accuracy measure): a round commits AL+1 tokens. The AR bootstrap round is excluded, and the test sets are restricted to the prompts on which neither model loops nor hits the generation cap, so the prompt counts below are smaller than the full benchmark sizes. Note that the distill-spec-8B card reports a bonus-inclusive AL on the full prompt sets; the two numbers are not directly comparable. Single evaluation snapshot from the release handoff.

BenchmarkPromptsMean accepted lengthSpeedup over AR
GSM8K (CoT)130310.493.28x
HumanEval1587.902.61x
MBPP4999.212.61x
SPEED-Bench7035.251.98x

Accepted lengths are paired against the pre-acceptance-training Phased-DMD4 student over the prompts that neither model loops or caps on; acceptance training raises them by 14.9%, 14.1% and 15.4% on GSM8K, HumanEval and MBPP. Speedups are throughput relative to this model's own autoregressive decoding, measured with SGLang in BF16 on one NVIDIA GB200, one request at a time. AR-decoding accuracy is unchanged by acceptance training.

Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

For more detailed information on ethical considerations for this model, please see the Model Card++ Bias, Explainability, Safety & Security, and Privacy Subcards.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns here.

Citations

@article{nemotronlabssigma2026,
  title       = {Large Language Continuous Diffusion Models},
  author      = {Zhihan Yang and Wei Guo and Jean-Marie Lemercier and Simon Welker and
                 Mohammad Mahdi Kamani and Yonggan Fu and Sajad Norouzi and Julius Berner and
                 Tomas Geffner and Karsten Kreis and Yongxin Chen and Molei Tao and
                 John Thickstun and Pavlo Molchanov and Ante Juki{\'c} and Arash Vahdat and
                 Morteza Mardani},
  journal     = {arXiv preprint arXiv:2610.02665},
  year        = {2026},
  url         = {https://arxiv.org/abs/2610.02665}
}

← all releases