Architecture Diagram · Model Card

Stream — Continuous Byte-Level State Space Language Model

A token-free, position-free, O(n) language model. No tokenizer. No positional encoding. No quadratic attention. The recurrence is position.

Vocabulary
256
raw bytes — the only "tokenizer" is UTF-8
Position
SSM scan
order encoded by the recurrence itself
Complexity
O(n)
linear compute & memory, flat 228K tok/s
Best val loss
1.69
Stream 6L/256D · 4.43M params (TinyStories bytes)
Stream training pipeline — raw bytes in, next bytes out token-free · vocab = 256 bytes position-free · recurrence encodes order O(n) scan · no quadratic attention Raw bytes b₁ b₂ … bₔ ∈ [0,255] no tokenizer byte_embed xₚ = Embed(bₚ) · 256 → D SSM blocks × N_layers · selective scan recurrence = position LN ln_f Multi-byte head D → n_predict × 256 Loss L = Σₖ CE(predₖ, bₚ+ₖ) (B, T) · int8 · 256 values (B, T, D) · float state hₚ (B, T, D) per layer (B, T, D) logits (B, T, n_predict·256) single objective · next-byte cross-entropy summed over the n_predict horizons
Figure 1 — The whole model is four stages. There is no vocabulary table, no positional embedding, and no attention: order comes from the sequential scan, and the only "vocabulary" is the 256-entry byte embedding. source: VECTOR/model.py — byte_embed, blocks, ln_f/head
SSM block — a Mamba-style selective state-space layer, repeated N_layers times xₚ (D) from previous block in_proj D → 2H · split into main + gate gate branch SiLU Conv1d k=4 · depthwise · causal SiLU dt_proj · x_proj softplus → Δₚ · [Bₚ | Cₚ] Δₚ = softplus(WΔ·xₚ) SSM scan · O(T) · SSMScanFn hₚ = hₚ-₁ · exp(Δₚ·A) + Δₚ·Bₚ·xₚ yₚ = Cₚ · hₚ + D · xₚ A = −exp(A_log) → always stable y = y · gate gating (sigmoid) gate sigmoid out_proj H → D LN(x + out) residual connection residual output xₚ₁ → next SSM block (or ln_f after the final one)
Figure 2 — One selective SSM block. The input is split into a main path (SiLU → depthwise conv → SiLU → projections) and a gating path; the projections produce an input-dependent step size Δₚ and state projections Bₚ, Cₚ; the scan integrates the recurrence in O(T); the output is gated, projected back to D, and added residually. source: VECTOR/model.py:123 — SSMBlock
The scan primitive — three ways to run hₚ = hₚ-₁·aₚ + bₚ (a) Forward · sequential O(T) aₚ = exp(ΔₚA) bₚ = ΔₚBₚxₚ h₀=0 ·a₁+b₁ h₁ ·a₂+b₂ h₂ … JIT-fused loop (model.py) — one CUDA kernel, no per-step Python overhead. O(T) time, O(T) state. _ssm_fwd @torch.jit.script sequential — optimal on CPU (b) Backward · reverse scan O(T) SSMScanFn · custom autograd Function dhₔ ·aₔ dhₔ-₁ ·aₔ-₁ dh₁ grad_bₚ = dhₚ (b-input grad) grad_aₚ = dhₚ · hₚ-₁ (decay grad) manual reverse pass replaces the O(T) autograd graph — no graph memory, bit-exact grads. (c) Parallel scan · O(log T) · Triton (a₂,b₂)∘(a₁,b₁) = (a₂a₁, a₂b₁+b₂) fused kernel · grid (B,H,N) · log-depth tree in SRAM — no HBM round-trips
Figure 3 — The core primitive. (a) The JIT-fused sequential recurrence for CPU. (b) SSMScanFn, the custom torch.autograd.Function whose manual reverse scan avoids building the O(T) autograd graph — roughly 10× faster backward, bit-exact. (c) The Triton parallel associative scan: since the combination operator is associative, prefix states can be merged in a binary tree, giving O(log T) parallel depth on GPU. source: model.py#L31 (_ssm_fwd/_ssm_bwd), model.py#L54 (SSMScanFn), VECTOR/triton_scan.py
Multi-byte prediction head — n_predict = 4 future bytes per position xₚ (D) at position t head D → 4 × 256 one forward pass logits k=0 · (256) logits k=1 · (256) logits k=2 · (256) logits k=3 · (256) bₚ+₁ bₚ+₂ bₚ+₂₁ bₚ+₄ L = Σₖ CE(predₖ, bₚ+ₖ) k = 0 … n_predict−1 single objective 4× training signal per position 4× faster autoregressive decoding (4 bytes per model step) also forces semantic learning: cannot copy byte t→t+1 only
Figure 4 — From each position the head emits n_predict independent 256-way byte distributions and samples the next n_predict bytes in one forward pass. This multiplies the per-step training signal by 4 and shortens autoregressive decoding by 4×. source: model.py#L245 (_compute_loss), model.py#L276 (generate)
The Stream family — what gets added on top of the pure SSM backbone Stream pure byte-level SSM · single loss · O(n) VECTOR SSM + learned pruning + attention + MoE • saliency gate — STE position pruning • AtomAttention — GQA, masked to kept atoms • MoE top-2 + five-term dual loss status: gate collapse (active_ratio=1.0) — needs retune MoE-Stream SSM + sparse mixture-of-experts • MoE FFN after each SSM block • expert lifecycle — value = impact / freq • replacement via best-expert clone + noise status: replacement hurts (2.35 vs 2.19) — keep OFF
Figure 5 — Variants built on the Stream backbone. VECTOR adds learned position pruning, GQA attention, and MoE under a five-term loss (gate collapsed in practice). MoE-Stream adds MoE FFNs with gradient-aware expert lifecycle management; its controlled ablation showed replacement hurts validation loss, so it is disabled by default. source: VECTOR/model_vector.py, VECTOR/moe_stream.py
Honest context. Stream's value proposition is O(n) — flat ~228K tok/s across all sequence lengths (forward-only benchmark), O(n) memory (6.77 GB at T=65,536 on a T4), and a projected crossover vs. quadratic attention at T≈9k (1.73× at T=16k, forward-only). The naive JIT scan measures Python-loop overhead, not asymptotics; the current GPU implementation is the fused Triton scan, plus a chunked two-level scan for low-B·H, occupancy-starved shapes and a shape-conditional auto-dispatch rule (chunked iff B·H≤128, else fused; no T threshold) — all benchmarked on T4 (see §5.9 of the report). The kernel body is still a serial loop over T (grid B·H), not a true O(log T) parallel tree scan. End-to-end training throughput is being re-measured after fixing a missing torch.cuda.synchronize(); earlier "12 ms/iter / 73×" figures were CPU-scheduling artifacts. The transformer still wins on loss at matched small-scale parameter counts — this diagram is the architecture, not the final verdict.