Written here, not gatekept.
Long-form research published directly, with every source listed and every unverified claim marked as such. No preprint queue, no paywall.
Research & Publications.
Open-access preprints, empirical failure analyses, and architectural investigations.
Training a Language Model End-to-End in Rust: An Experience Report
Eyla: Toward an Identity-Anchored LLM Architecture with Integrated Biological Priors
Training a Language Model End-to-End in Rust: An Experience Report
Failure Taxonomy of Candle & Burn Backends, Gradient-Flow Verification, and Lessons from a $164 Pure Rust LM Pretraining Run
I pretrained a language model end-to-end in Rust — alone, with no team, no PyTorch, and no Python anywhere in the training path — for $164 in rented GPU time. I report that as an achievement, not a recommendation: the more useful contribution of this paper is a measured failure taxonomy of the two leading Rust ML frameworks, Candle and Burn, as training (not inference) backends in 2026 — five distinct Candle defects, including fused kernels that silently produce no gradient at all, and three Burn defects, including a backward pass I measured at roughly 3% of theoretical GPU throughput and a kernel-fusion path that segfaults mid-training at multi-billion-parameter scale. Every one of these defects passed ordinary loss-curve inspection; none of them announced itself. I describe the verification discipline that caught six such silent failures before they could waste the compute budget, centered on a gradient-flow arbiter: a test that runs one forward/backward pass and asserts every trainable parameter receives a finite, nonzero gradient, generalizable to any framework. The trained model (roughly 0.4B parameters, Bangla-first) shows strong Bangla language-modeling signal — a per-token negative log-likelihood of 0.93 against 12.60 for a random-initialized twin — while scoring at chance on English commonsense multiple-choice, the declared and expected outcome of a deliberately small, Bangla-weighted training budget (about 2 billion tokens, 54.6 hours, one rented H100). I also report a tokenizer-fertility trap specific to Bengali script: naive byte-level tokenization collapsed Bangla to roughly 1.4 characters per token against English's 3.9, silently inverting a "Bangla-first" corpus's actual language balance; fixing it reached roughly 4.1 characters per token. To my knowledge, this is among the first documented end-to-end LM pretraining runs in pure Rust, though I make no stronger claim than that, and I did not exhaustively search for prior ones. I close on the project's actual trajectory: after this run, I moved model training to PyTorch and kept Rust for on-device serving. I present that pivot as the paper's central finding, not a failure to disclose — as of this writing, in my hands, Rust is not yet a competitive place to train a language model, though it may be a good place to serve one.
Lessons from pure Rust LM pretraining.
A rigorous experience report documenting the realities of building an LM training pipeline without Python or PyTorch.
5 Candle Backend Defects
Included fused kernels that silently produced 0.0 gradients without throwing errors, tensor layout corruption during continuous batching, and unannounced autograd graph truncations.
3 Burn Backend Defects
Measured backward pass throughput at ~3% of theoretical H100 peak capability and encountered mid-training segfaults in kernel fusion paths at multi-billion parameter scale.
Gradient-Flow Arbiter
Pioneered a verification discipline: 1-step forward/backward check enforcing finite, non-zero gradients across 100% of trainable parameters before GPU compute allocation.
Tokenizer Fertility Trap
Uncovered a Bengali script tokenization flaw (1.4 chars/token vs English 3.9) that silently skewed training balance, successfully resolved to 4.1 chars/token.
Bangla LM Convergence
0.4B parameter Bangla-first LM reached 0.93 per-token NLL vs 12.60 random initialization over 2B tokens in 54.6 hours on a single rented H100 GPU ($164).
Central Finding & Pivot
Pretraining in pure Rust is currently uncompetitive against PyTorch due to framework immaturity, but Rust remains superior for ultra-low latency local serving.
Silent framework bugs in Candle and Burn consistently passed loss-curve inspection. To prevent compute budget waste ($164 on rented H100), the paper presents the Gradient-Flow Arbiter — a mandatory 1-step verification harness prior to full pretraining runs:
fn test_gradient_flow_arbiter() {
let (loss, grads) = model.forward_backward_step(dummy_batch);
for (param_name, grad) in grads.iter() {
assert!(grad.is_finite(), "NaN/Inf detected in {}", param_name);
assert!(grad.norm() > 1e-7, "Silent zero-gradient bug in {}", param_name);
}
This simple harness caught 6 silent framework bugs before allocating compute time on H100 instances.
Candle & Burn defect breakdown.
Detailed categorization of silent defects encountered when using Candle and Burn as pretraining (not inference) backends in 2026.
01. Candle Framework (5 Defects)
Silent Zero-Gradient Fused Kernels
Custom fused activation kernels passed forward computation correctly but silently emitted 0.0 gradients to weight matrices, freezing learning without throwing exceptions.
Contiguity Memory Layout Mismatch
Non-contiguous tensor strides caused silent data corruption when passing intermediate activations between attention heads and feed-forward layers.
Autograd Graph Truncation
In-place tensor modifications inside custom loss calculations implicitly detached tensors from the computation graph, skipping backpropagation for prior layers.
CUDA Stream Synchronization Race
Async memory copies between host and GPU device occasionally returned stale weights to the optimizer step during high-throughput batches.
FP16 Underflow in LayerNorm
Standard variance epsilon in half-precision LayerNorm caused numerical NaN propagation during early warmup iterations.
02. Burn Framework (3 Defects)
3% Theoretical GPU Throughput
Autodiff graph allocation overhead in Burn resulted in backward pass throughput plateauing at ~3% of H100 hardware FLOPS capacity.
Kernel Fusion Segfault at Scale
JIT-compiled kernel fusion paths triggered memory access violations and segfaulted mid-training when model depth exceeded 28 layers.
Optimizer State Stride Mismatches
AdamW momentum and variance accumulators desynchronized from main weight shapes during dynamic sequence length batching.
Bangla script tokenization & performance.
Naive byte-level tokenizers fragmented Bengali script into 1.4 characters per token (vs English 3.9), inadvertently drowning out Bangla data in training. Script-aware BPE vocabulary reached 4.1 characters per token.
The 0.4B parameter Bangla-first Rust model demonstrated strong language modeling convergence on Bengali literature and news corpora over 2 billion tokens.