Overview

Possible Procedure

1. 1
   Initialize: pre-LN or RMSNorm, gated residuals (alpha_l ~ 0.05-0.1), DeepNet/muParam scaling, light spectral constraints.
2. 2
   Calibrate: stream 1-5% of tokens (more if necessary) to estimate K-FAC/Shampoo factors per block, collect residual RMS and logit scale stats. Dropout off.
3. 3
   Solve: compute blockwise Gauss-Newton/natural-gradient delta W_l with damping and a small proximal term. Allocate per-block KL budgets, scale each delta W_l to fit.
4. 4
   Compose delta theta: enforce a global KL cap via delta theta^T F delta theta <= epsilon, then apply alpha_l gates (start 0.05-0.2, larger for top blocks).
5. 5
   Guard: run linearization-error checks per block, downscale any block that fails. Run spectral checks for MHA/MLP.
6. 6
   Line search: Armijo on held-out loss plus KL cap, adjust alpha_l (not just a single alpha). Prefer to increase alpha_l for blocks passing both KL and linearization checks.
7. 7
   Refresh: optionally relinearize top K blocks, refresh their factors, and take a second (smaller) trust-region step.
8. 8
   Fine-tune: second-order optimizer (K-FAC/Shampoo grafted onto AdamW) with big batches for a few thousand steps. Re-enable dropout. Unfreeze MoE router here if we decide to tackle that. Maintain per-step KL regularization to keep the student initially near the trust region.

Possible Procedure

1. 1
   Initialize: pre-LN + gated residuals, add planner head, write head, and light fusion blocks. Storage: concept nodes with hyperbolic codes, relation vocab R, induced table schemas, ANN for nodes/leaves.
2. 2
   Calibrate: cluster 1-5% of corpus to seed nodes/relations, label nodes by summarizing prototypes, estimate top-k recall/coverage, set per-source budgets and alpha_ext gates.
3. 3
   Write: train create/link policy (straight-through Gumbel). Attach support spans and timestamps, regularize node/edge growth and degree, merge near-duplicates.
4. 4
   Plan: teach a tiny DSL (select, hop, filter, join, aggregate) with weak traces (BM25/DPR/SQL/paths). Router picks graph/text/table. Penalize long programs.
5. 5
   Retrieve/Fuse: execute plans, return compact subgraphs/rows/snippets. Fuse with cross-attn, gating evidence vs param prediction, require citations of nodes/edges used.
6. 6
   Externalize: anti-memorization drops and KL penalties so factual answers rely on the store. Increase alpha_ext and budgets where evidence consistently helps.
7. 7
   Compose: alternate phases: (A) train writer/planner/retriever with LM frozen, (B) train LM-on-evidence with store frozen, (C) brief joint pass.
8. 8
   Guard: structural checks (type purity, degree/entropy caps, cycle detection), drift alarms, and timeouts with fallback to text-only retrieval.
9. 9
   Line search: tune program depth/width and alpha_ext on held-out loss + latency + citation quality, promote sources that pass support/consistency checks.
10. 10
    Refresh & Fine-tune: incremental writes and nightly re-embed/re-index, merge/retire nodes, maintain hot shards in memory, version edges for freshness. Preference training that rewards supported, cited answers, keep base LM small, adapt planner, writer, fusion. Maintain offloading penalties early.

Possible Procedure

1. 1
   Initialize: pre-LN/RMSNorm Transformer with gated residuals, add two heads: a utility u(x,t) predicting per-token residual bits, and a template planner that emits short programs over a tiny DSL (steps, slots, control).
2. 2
   Calibrate: stream 1-2% of tokens. Fit u using loss/entropy/grad-sketch proxies. Seed a macro dictionary with frequent spans and a small set of reasoning schemas from mined CoT traces. No dropout.
3. 3
   Solve (encode): train a compression model with an MDL objective:
   • Direct compressibility (semantic LZ): segment text, learn macros that maximize code-length reduction across paraphrases and formats.
   • Hierarchical compressibility (Template VM): induce typed templates (e.g. 1) compare 2) sort 3) argmax, 1) retrieve 2) filter 3) aggregate) with slots, compile to executable programs.
   • Choose between macro, template, or raw tokens per segment via straight-through gating, penalize long codes and deep programs.
4. 4
   Compose (dataset): re-encode the corpus as codes:
   • Keep only (a) template calls + slot fills and (b) residual tokens above a u-threshold, drop predictable tokens.
   • Importance-weight residuals to match the original distribution, reserve a small shadow random batch each step to measure gradient mismatch.
5. 5
   Guard: track gradient-match vs full sampling on shadow batches, bound NLL drift per domain, auto-bail to denser sampling if mismatch or rare-phenomena error rises. Enforce diversity caps so the dictionary doesn't collapse.
6. 6
   Line search (budget): on held-out loss + code length, adapt thresholds: raise macro/template use until mismatch grows, lower when residual error spikes. Learn per-domain quotas and max program depth.
7. 7
   Refresh: every K steps, re-minimize MDL on a fresh stream: merge/split macros, specialize/generalize templates, retire low-utility entries. Maintain a novelty buffer that bypasses compression for surprising spans until absorbed.
8. 8
   Fine-tune: short full corpus sweep at low rate to debias, then large-batch training dominated by residuals. Keep off-policy checks with the shadow batch. At inference-time, optional on-the-fly template execution for chain-of-thought.

Possible Procedure

1. 1
   Initialize: pre-LN transformer with a planner head for plan tokens/templates, per-block low-rank adapters in early layers, verifier API and prefix extractor, global KL and latency budgets.
2. 2
   Calibrate: collect pass@N logs on a training pool, label winners with the verifier, record prefixes, partial checks, and problem type features. Train a light prefix-success predictor to guide sampling and hard negative mining.
3. 3
   Solve: build a preference set of (winner_prefix, loser_prefix, context). Train with DPO or IPO on prefixes. Add step-wise advantage-weighted regression using partial checks as rewards with a control variate. Apply per-step KL penalties and EWC on core capabilities, constrain updates to planner head and early adapters.
4. 4
   Compose: at inference, lightly bias decoding toward learned plan tokens and templates (logit offsets, small temperature on planner head). Keep standard decoding otherwise. Cache priors per problem type.
5. 5
   Guard (no-cheat constraints):
   • Train only on training pool logs, never eval logs.
   • Use verifier and partial checks that exist at inference (unit tests, proofs, execution), not hidden labels.
   • Restrict supervision to prefixes and plan tokens, not full final strings.
   • Enforce per-batch and cumulative KL caps, run regression canaries and length monitors.
6. 6
   Line search: tune prefix length K, preference temperature, advantage weights, and KL caps on held-out pass@1 uplift vs pass@N, verbosity, and latency. Prefer shorter prefixes with large uplift.
7. 7
   Refresh: periodically mine fresh pass@N logs, add hard negatives near the decision boundary, merge redundant templates via MDL, retire low-utility plan tokens.
8. 8
   Fine-tune: brief consolidation pass to stabilize planner and adapters under tight KL. Verify gains persist with normal decoding and no reliance on best-of sampling.

Possible Procedure

1. 1
   Initialize: base model with planner head and early-layer adapters, add a prefix critic (small transformer or MLP on hidden prefix states), verifier API and partial-check probes, set KL and latency budgets.
2. 2
   Calibrate: collect pass@N trees by saving top-k beams and sampled branches per problem. Annotate each node with partial checks passed and leaf success. Estimate off-policy correction weights for branches.
3. 3
   Solve: compute soft Q at each node via backward propagation of verifier success with entropy regularization. Train the prefix critic to predict Q from hidden states and plan tokens. Improve the planner with advantage-weighted cross-entropy or PPO-style updates under KL to the current policy, apply EWC to protect core skills.
4. 4
   Compose: at inference, combine planner logits with critic advantages for the first K steps (small additive bias). Keep decoding otherwise unchanged. Cache advantages by problem type for fast reuse.
5. 5
   Guard (no-cheat constraints):
   • Construct trees only on training pools, never use eval trees.
   • Rely on verifiers and partial checks available at inference.
   • Bias only early plan decisions, do not regress on final answers.
   • Cap per-update and cumulative KL, enforce length and latency bounds, run regression canaries.
6. 6
   Line search: tune K, critic regularization, advantage temperature, and KL caps on held-out pass@1 vs pass@N gap, stability, and verbosity. Prune nodes with low contribution to avoid overfitting.
7. 7
   Refresh: rebuild trees on new snapshots, update the critic with fresh logs, remove stale branches, merge redundant plan tokens, maintain a buffer of hard problems for continual improvement.
8. 8
   Fine-tune: periodic short run to co-train planner and critic with small KL, freeze late layers, verify gains on unseen tasks and that pass@N remains stable.

Approach

• Scope: Identify candidate latent programs by probing for low-entropy, low-rank, highly repeatable behaviors at specific layers/heads like 'angle to sin(angle)', 'string to match(pattern)'.
• Specification mining: Generate test benches using counterfactual inputs, invariance checks (e.g. sin(x+2pi)=sin(x)), and smoothness/periodicity detectors, then fit candidates from a library (trig, polynomials, finite-state transducers, arithmetic, set ops).
• Synthesis: Use hybrid methods—sparse identification of nonlinear dynamics (SINDy-style), symbolic regression, and enumerative search over a compact IR (typed SSA with vector ops, conditionals, and bounded loops).
• Verification: Run equivalence tests against the original subnetwork over adversarial and randomized inputs, then certify error bounds and input domain. If verified, freeze/retire the circuit and insert a differentiable "call-module" stub with gradients routed to arguments, not the replaced weights.
• Maintenance: Version modules, track provenance and input domains, and maintain a regression suite. Allow hot-swaps (e.g. better trig approximations) without touching base weights.

Neural circuits often realize simple, reusable functions. Explicit modules are easier to verify, cache, optimize, and upgrade. Synthesis with invariants curbs overfitting, then calls avoid recomputation and shrink parameter/activation footprints.

Approach

• ABI design: Typed arguments (scalars, vectors, strings, sets), effect annotations (pure/impure), shape contracts, and gradient rules (exact, straight-through, or stop-grad).
• Router: A lightweight planner head predicts when to call, which module, and with what arguments. Provide partial credit via differentiable surrogates (e.g. relaxed arg parsing, soft alignment of spans to args).
• Costs and rewards: Penalize param-only solutions when a verified module exists, then reward correct calls with small KL bonuses and latency/energy credits, then enforce per-batch "offload budgets" to shift usage gradually.
• Pruning: After stable adoption, gradually L0-prune circuits shadowed by calls, then keep a safety adapter to catch drift and trigger re-training.
• Continual learning: When no module fits, log traces and auto-propose new candidates for Project 1 to compile. Close the loop: discovered modules immediately become callable via the dABI.

Models already plan tool usage, then making library calls first-class and cheap creates selection pressure to externalize. A typed ABI contains complexity, reduces integration bugs, and allows acceleration (e.g. vectorized trig kernels).

Suggested Approach

• Representations:
  • Static capability vector z_h per head/block with a canonical docstring (learned once, versioned).
  • Dynamic call-site docstring d_h,t per token/time: a short sequence in a constrained DSL (e.g. COPY_FROM(span), MATCH(pattern), COREF_ANTE(span), AGG(NUM, window), DATE_ADD, ARGMAX(key), ROUTE_TO(node), ASSERT(invariant)).
  • Citations c_h,t: a sparse set of input spans (start, end, source_id) with confidence.
• Losses and Constraints:
  • Reconstruction: small decoders reconstruct the head's output from (d_h,t, c_h,t, selected source embeddings), the docstring must be sufficient to predict the unit's effect.
  • Causal faithfulness: ablate or perturb cited spans, require the predicted effect to change accordingly (contrastive consistency). Enforce sparsity on citations and docstring length (MDL).
  • Alignment: cluster behaviors offline and map clusters to codebook tokens, with human-curated seeds for a few canonical operations (coref, copy, local-n-gram, delimiter detection, number aggregation).
• Runtime: A trace aggregator collects (d_h,t, c_h,t, z_h) across layers and compiles a readable stack trace with timestamps and evidence links.

Why: 'Documentation' used to predict and constrain the forward computation is far harder to fake than post-hoc gloss. Minimal codebook tokens with explicit citations make trace outputs short, comparable, and auditable.

Possible Procedure

1. 1
   Initialize: Choose base model (e.g. 7B pre-LN Transformer), instrument attention heads/MLPs with documentation head (linear to small LM over fixed codebook of 64-128 tokens), citation head (pointer network over input tokens/spans, k=1-3 pointers), and lightweight reconstructor. Define codebook with 12-20 primitive tokens (COPY, COREF_ANTE, LOCAL_NGRAM, OPEN_QUOTE, CLOSE_QUOTE, MATCH_DIGITS, SUM_NUMS, MAX_BY_KEY, TABLE_HEADER_ALIGN, DATE_PARSE, DATE_ADD, FORMAT). Leave "OTHER_x" slots for emergent clusters.
2. 2
   Calibrate: Collect behavior sketches: run base model on QA and math corpora, log attention patterns and interventions (activation patching, heads ablation). Cluster head behaviors to propose initial head-to-codebook mappings, hand-label only 10-20 examples to seed.
3. 3
   Train (Phase A: doc-only): Freeze base LM. Train documentation and citation heads and reconstructors: cross-entropy loss on doc tokens with entropy regularization, length penalty (hard cap 3-5 tokens), sparse pointer loss for citations (top-k), MSE/cosine reconstruction loss, and causal imitation on counterfactual subsets.
4. 4
   Guard (doc quality): Disallow trivial docs: if doc length > 1 but adds sub-epsilon reconstruction gain vs "OTHER", increase MDL penalty. Merge near-duplicate codebook entries, bound per-head doc entropy over time.
5. 5
   Train (Phase B: explain-to-execute coupling): Unfreeze small gates around each unit: when a docstring declares COPY_FROM(span), soft-mask attention to non-cited spans; when AGG(NUM), encourage number-feature heads to turn on. Performance loss KL-regularized to base, plus penalty for off-trace token usage.
6. 6
   Runtime trace: Implement a trace aggregator producing per-token JSON trace (step_id, layer, head, doc_tokens, citations, confidence, before/after norms). UI: render collapsible stack trace with clickable spans showing source context.
7. 7
   Evaluate: Fidelity via "mask-the-citation" causal tests (drop cited spans, rerun, record delta; report per-domain causal F1). Measure compactness (average doc length, citation count) and stability (doc overlap on new corpora, drift alarms).
8. 8
   Refresh: Periodically recluster behaviors, reassign "OTHER" tokens to concrete roles, prune unused codebook entries. Small preference-tuning round to favor shorter, more faithful docs without hurting task metrics.

Literature & Weaknesses Addressed

• Chain-of-thought: Boosts performance, but often unfaithful and verbose. We avoid free-form text and enforce causal use via masking.
• Attention != explanation: We add reconstruction plus interventional tests to ensure attention/citations matter causally.
• Self-Explaining Neural Networks / rationale extraction: Many produce proxies not used by the model. We couple docs to execution.
• Sparse autoencoders on residual streams: Promising for features but not tied to live traces. We integrate with citations and reconstruction.

Suggested Approach

• DSL (first pass, 8-12 atoms):
  • QA: FETCH(span_id), HOP(via_anchor), COMPARE(span_a, span_b, key), AGG(list, op), SELECT(condition), ASSERT(supports(answer)), CITE(span_id).
  • Math: PARSE_NUM(span), APPLY(op, args), KEEP(track_id), CHECK(invariant), FORMAT(result).
• Execution Coupling:
  • When an atom fires, apply compile-time masks: limit attention to nominated spans, route to dedicated submodules (e.g. arithmetic/date kernels), and activate heads matching the atom (HeadDocstoDSL mapping).
  • At each step, trace channel proposes 0-2 atoms with confidences. If confidence < tau, default to unconstrained decoding but flag "no-trace" for transparency.
• Supervision: Weak signals: distant supervision from retrieval citations, gold answer spans, program-of-thought datasets (math), and tool returns. No reliance on ground-truth explanations at scale. Consistency: require emitted atoms and HeadDocs to agree.
• Guardrails: Budget per step: at most one FETCH and one APPLY per token budget. Abstain when inputs are ambiguous, do not fabricate arguments.

Why: A small, typed set of atoms is enough to make traces legible and enforce faithfulness. Gating makes cheating costly—if the atom says FETCH(span), only that span is accessible, so the model can't secretly use other facts.

Possible Procedure

1. 1
   Initialize: Add a parallel trace head (small decoder) conditioned on main model's hidden states, share early layers. Define DSL schema and static mapping from HeadDocs codebook tokens to eligible atoms. Implement attention masks for FETCH/HOP and router for APPLY.
2. 2
   Calibrate: Build small training pools (Hotpot-like QA with support spans; GSM8K math with synthetic step checks). Measure baseline unsupported answer rate and spontaneous citations.
3. 3
   Warm-start (supervised): Train trace head to imitate weak labels (support spans, tool steps) where available, apply MDL length penalty and abstain option. Enforce atom-to-mask in a fraction of steps, keep base LM frozen.
4. 4
   Couple (explain-to-execute): Gradually increase fraction of steps where masks are enforced, using a KL trust region. Penalize off-trace usage when gradients/attention mass flow outside allowed spans.
5. 5
   Consistency with HeadDocs: Jointly train with Project 1 so active heads' dynamic docs match atom class; add small cross-entropy alignment loss. Break ties by preferring HeadDocs if atom confidence is low.
6. 6
   Guard: Atom budget per K tokens; high cost for atoms that don't change downstream behavior (measured by ablation). Drift alarms if unsupported answers creep up.
7. 7
   Evaluate: Fidelity via mask-the-citation tests at step level. Measure fraction of runs where removing fetched spans or preventing APPLY flips final answer. Aim for <= 15% latency overhead.
8. 8
   Refresh & Distill: Add 2-3 atoms based on observed gaps (e.g. MATCH_DATE, JOIN_TABLE) only if clearing MDL/utility threshold. Occasionally fine-tune to reduce mask reliance while preserving stable traces.