Research topic: Faster learners

Thesis: training power usage should be many orders of magnitude lower. No single change will get us there, but a plausible research program is to:

1) make most tokens unnecessary,
2) quickly get to a good solution analytically or with amortized learners,
3) learn to use external structure rather than training every fact into the weights, and
4) combine second order methods with low precision training to converge in fewer, larger steps.

Project: Faster training via approximate analytic solutions

Motivation: training big networks via SGD is hideously inefficient. Can approximate analytic solutions provide a 100x reduction in training time?

Idea: use curvature information from a tiny fraction of the corpus, solve the resulting quadratic problem once per block, then de-linearize by injecting the update through small gates so that the network stays in its local linear regime.

Suggested approach: estimate a good solution by solving a sketched natural-gradient step in closed form. Stream the corpus once to estimate a block-diagonal or Kronecker-factored Fisher (or Gauss-Newton) in each layer using random projections, compute the ridge solution for linearized parameters, de-linearize by composing layers near identity.

Why: layerwise K-FAC/Shampoo already works, randomized second-order solvers are mature, neural nets are close to linear early in training. A high-quality one-shot quasi-Newton init could remove 90-99% of steps.

Risks: linearization error, imperfect Fisher, stability in deep stacks. (+ still needs a brief fine-tune at the end). LLMs also benefit from feature learning outside the lazy/NTK regime, where analytic linearized steps help least.

Possible procedure:

1. Initialize: pre-LN or RMSNorm, gated residuals (alpha_l ~ 0.05-0.1), DeepNet/muParam scaling, light spectral constraints.
2. Calibrate: stream 1-5% of tokens (more if necessary) to estimate K-FAC/Shampoo factors per block, collect residual RMS and logit scale stats. Dropout off.
3. Solve: compute blockwise Gauss-Newton/natural-gradient delta W_l with damping and a small proximal term. Allocate per-block KL budgets, scale each delta W_l to fit.
4. Compose delta theta: enforce a global KL cap via delta theta^T F delta theta \<= epsilon, then apply alpha_l gates (start 0.05-0.2, larger for top blocks).
5. Guard: run linearization-error checks per block, downscale any block that fails. Run spectral checks for MHA/MLP.
6. Line search: Armijo on held-out loss plus KL cap, adjust alpha_l (not just a single alpha). Prefer to increase alpha_l for blocks passing both KL and linearization checks.
7. Refresh: optionally relinearize top K blocks, refresh their factors, and take a second (smaller) trust-region step.
8. Fine-tune: second-order optimizer (K-FAC/Shampoo grafted onto AdamW) with big batches for a few thousand steps. Re-enable dropout. Unfreeze MoE router here if we decide to tackle that. Maintain per-step KL regularization to keep the student initially near the trust region.

Literature:

Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures

SKFAC: Training Neural Networks with Faster Kronecker-Factored Approximate Curvature

Near-optimal Sketchy Natural Gradients for Physics-Informed Neural Networks

Project: Learn-to-database (explicit hierarchical memory)

Motivation: storing facts in weights is wasteful and gets stale. Can we externalize knowledge as an explicit, updatable hierarchy the model learns to write to and query?

Idea: make the model 1) induce its own ontology of concept nodes, typed edges, and table schemas, and 2) plan short programs over it. The LM becomes a planner/fuser, facts live in a compact, interpretable store.

Suggested approach: build a small, learned hierarchy: nodes with hyperbolic/tree codes and prototypes, sparse typed edges with evidence, leaves as text passages and table rows. Train write (create/link) and plan (DSL) heads, retrieve tiny typed subgraphs/rows, fuse with gated cross-attention, penalize param-only answers.

Why: transformers already form latent hierarchies. Making them explicit improves sample efficiency, updatability, and interpretability. RAG/FiD, Poincare embeddings, schema induction, and programmatic retrieval provide working parts.

Risks: ontology drift/fragmentation, noisy links, planner brittleness, latency/complexity. Model may bypass the store. Some fast/robust knowledge must remain parametric.

Possible procedure:

1. Initialize: pre-LN + gated residuals, add planner head, write head, and light fusion blocks. Storage: concept nodes with hyperbolic codes, relation vocab R, induced table schemas, ANN for nodes/leaves.
2. Calibrate: cluster 1-5% of corpus to seed nodes/relations, label nodes by summarizing prototypes, estimate top-k recall/coverage, set per-source budgets and alpha_ext gates.
3. Write: train create/link policy (straight-through Gumbel). Attach support spans and timestamps, regularize node/edge growth and degree, merge near-duplicates.
4. Plan: teach a tiny DSL (select, hop, filter, join, aggregate) with weak traces (BM25/DPR/SQL/paths). Router picks graph/text/table. Penalize long programs.
5. Retrieve/Fuse: execute plans, return compact subgraphs/rows/snippets. Fuse with cross-attn, gating evidence vs param prediction, require citations of nodes/edges used.
6. Externalize: anti-memorization drops and KL penalties so factual answers rely on the store. Increase alpha_ext and budgets where evidence consistently helps.
7. Compose: alternate phases: (A) train writer/planner/retriever with LM frozen, (B) train LM-on-evidence with store frozen, (C) brief joint pass.
8. Guard: structural checks (type purity, degree/entropy caps, cycle detection), drift alarms, and timeouts with fallback to text-only retrieval.
9. Line search: tune program depth/width and alpha_ext on held-out loss + latency + citation quality, promote sources that pass support/consistency checks.
10. Refresh: incremental writes and nightly re-embed/re-index, merge/retire nodes, maintain hot shards in memory, version edges for freshness.
11. Fine-tune: preference training that rewards supported, cited answers, keep base LM small, adapt planner, writer, fusion. Maintain offloading penalties early.

Literature:

Memory^3: Language Modeling with Explicit Memory

Ontology Generation using Large Language Models

KB-Plugin: A Plug-and-play Framework for Large Language Models to Induce Programs over Low-resourced Knowledge Bases

Sub-project: Write-head sparsity curriculum (explicit, updatable hierarchies)

Motivation: unconstrained writes bloat the store and kill interpretability. Can we grow a compact, legible hierarchy by starting ultra-sparse and relaxing only when utility is proven?

Idea: treat create node/edge/schema as gated actions with an explicit cost. Start with near-zero write capacity, force reuse/links, relax L0/MDL penalties as evidence accumulates that new structure reduces loss.

Suggested approach: hard-concrete gates per write op (create/link/split), group-lasso over relation types, and an MDL-style budget. A staged schedule: 1) link-only, 2) controlled splitting, 3) schema induction, 4) relaxed growth. Each write gets a justification (support spans, proto summary), enabling audit and rollback.

Why: early noise is what makes ontologies sprawl. A sparsity curriculum preserves tree-like structure, encourages reuse, and keeps concepts interpretable while still allowing growth when it pays off.

Risks: over-sparsity (missed concepts), late discovery tax (hard to recover if you never split), planner gaming the penalties, merge instability.

Possible procedure:

1. Initialize: seed a tiny tree (hyperbolic codes) and relation vocab. Enable retrieval/fusion, disable creates (writes off). Set very high L0/MDL penalties and tiny per-batch budgets (0-1 creates, 1-5 links).
2. Calibrate: measure coverage gaps, residual/error hotspots, node heterogeneity (prototype entropy), and average degree. Set a target growth curve (sublinear nodes vs tokens).
3. Link-only: allow edges to existing nodes with top-k sparsity, dedup/merge aggressively. New edges must carry citations and type, enforce degree/branching caps.
4. Controlled splitting: permit K new nodes per N tokens when triggers fire (high residual on a node, multi-modal embeddings, planner dead-ends). Child nodes inherit parent, get proto summaries, and must improve held-out loss to persist.
5. Schema induction: when repeated attribute patterns appear, propose columns/typed relations with group-lasso over relation types. Writes to schema have higher cost, require multi-example support.
6. Relaxation: slowly decay L0 and MDL penalties, increase budgets in domains where new structure consistently helps. Keep periodic prune sweeps (drop low-utility nodes/edges, auto-merge siblings).
7. Guard: pre-commit checks (type purity, cycle limits, citation density), post-commit A/B (with/without the write) to confirm utility, rollback on regressions. Maintain bounded hyperbolic radius to keep tree-like geometry.
8. Line search (budget control): adjust penalties to hit the target growth slope while minimizing held-out loss + write cost + latency. Prefer granting budget to sources/relations with highest utility per write.
9. Refresh: regular scheduled merge/consolidate, re-embed nodes, retire stale edges, version writes with timestamps.
10. Fine-tune: alternate freeze phases: 1) train planner/retriever under strict budgets, 2) briefly unfreeze write-heads to apply a small number of high-confidence writes, 3) joint stabilization pass. Keep anti-memorization so factual wins come from the store.

Signals it's working: sublinear graph growth, rising citation rate, stable small branching factors, and large accuracy drops when the store is ablated in the domains it covers.

Literature:

Crisp Attention: Regularizing Transformers via Structured Sparsity, https://arxiv.org/abs/2508.06016
The Role of Sparsity for Length Generalization in Transformers, https://arxiv.org/pdf/2502.16792
White-Box Transformers via Sparse Rate Reduction: Compression Is All There Is?, https://arxiv.org/abs/2311.13110

Growing neural networks: dynamic evolution through gradient descent, https://arxiv.org/abs/2501.18012
Identifying hubs in directed networks, https://arxiv.org/pdf/2312.03347
GKG-LLM: A Unified Framework for Generalized Knowledge Graph Construction, https://arxiv.org/html/2503.11227v2

Project: Train on the residual (semantic LZ + template VM)

Motivation: pretraining rereads the same patterns and reasoning chains. If models learn reusable templates, we should only pay for residual bits, not repetitions. Maybe direct + hierarchical compressibility can cut token exposure.

Idea: turn pretraining into compression. Learn a hierarchical dictionary of semantic spans and reasoning templates. Encode the corpus into 1) template calls with slot fills and 2) residual tokens the dictionary can't predict. Train primarily on the residual, keep the dictionary executable so a few examples teach many instances.

Suggested approach: 1) a neural compressor that induces macros (semantic LZ) and templates (reasoning VM) with MDL pressure, and 2) a residual trainer that samples only novel bits with importance weighting. Refresh the dictionary online so the residual steadily shrinks.

Why: language is highly compressible, gradient contributions are heavy-tailed. Grammar induction, dataset distillation, and macro tokenization already show large sample cuts. If chain-of-thought compresses into a small set of schemas, we need far fewer demonstrations.

Risks: over-compression (miss rarities), drift as the model/dictionary co-evolve, template brittleness, compressor overhead. Requires tight guards to avoid bias and loss spikes.

Possible procedure:

1. Initialize: pre-LN/RMSNorm Transformer with gated residuals, add two heads: a utility u(x,t) predicting per-token residual bits, and a template planner that emits short programs over a tiny DSL (steps, slots, control).
2. Calibrate: stream 1-2% of tokens. Fit u using loss/entropy/grad-sketch proxies. Seed a macro dictionary with frequent spans and a small set of reasoning schemas from mined CoT traces. No dropout.
3. Solve (encode): train a compression model with an MDL objective:
   - direct compressibility (semantic LZ): segment text, learn macros that maximize code-length reduction across paraphrases and formats.
   - hierarchical compressibility (Template VM): induce typed templates (e.g. 1) compare 2) sort 3) argmax, 1) retrieve 2) filter 3) aggregate) with slots, compile to executable programs.
   - choose between macro, template, or raw tokens per segment via straight-through gating, penalize long codes and deep programs.
4. Compose (dataset): re-encode the corpus as codes:
   - keep only (a) template calls + slot fills and (b) residual tokens above a u-threshold, drop predictable tokens.
   - importance-weight residuals to match the original distribution, reserve a small shadow random batch each step to measure gradient mismatch.
5. Guard: track gradient-match vs full sampling on shadow batches, bound NLL drift per domain, auto-bail to denser sampling if mismatch or rare-phenomena error rises. Enforce diversity caps so the dictionary doesn't collapse.
6. Line search (budget): on held-out loss + code length, adapt thresholds: raise macro/template use until mismatch grows, lower when residual error spikes. Learn per-domain quotas and max program depth.
7. Refresh: every K steps, re-minimize MDL on a fresh stream: merge/split macros, specialize/generalize templates, retire low-utility entries. Maintain a novelty buffer that bypasses compression for surprising spans until absorbed.
8. Fine-tune: short full corpus sweep at low rate to debias, then large-batch training dominated by residuals. Keep off-policy checks with the shadow batch. At inference-time, optional on-the-fly template execution for chain-of-thought.

Expected outcome: train on 3-10% of original tokens at steady state for compressible domains. Large wall-clock/power savings in early training, better factual and reasoning generalization per token via explicit reuse of templates, fast adaptation by updating the dictionary.

Literature:

Language Modeling is Compression, https://arxiv.org/abs/2309.10668
Compression Represents Intelligence Linearly, https://arxiv.org/pdf/2404.09937
Entropy Law: The Story Behind Data Compression and LLM Performance, https://arxiv.org/html/2407.06645v1

Research topic: Learning from failure

Thesis: people learn from our mistakes. Models don't, but pass@512 works better than @1, indicating the answer is often reachable already. Let's make max(pass@N) \== pass@1:

1) turn search into supervision,
2) convert in-context fixes into weights,
3) compile failures into repairs, and
4) keep gains without regressions/forgetting.

rough ideas: 1) bound updates with KL/Fisher caps, run canaries, use selective counterfactual replay, periodically distill validated patches into main. 2) align fail vs win traces to learn detectors and conditional rewrites, store them as routed patches and planner rules. 3) trigger on surprise from tool returns, crystallize traces into reusable templates and low-rank adapters with trust-region commits. 4) harvest pass@N trees with verifiers and partial checks, learn prefix value functions and distill winner prefixes into the policy.

Project: Prefix advantage distillation

Motivation: many problems succeed with pass@64-512 but miss on pass@1 because early plan choices diverge. Can we shift policy to prefer winners without leaking finals or eval labels?

Idea: distill the advantage of early prefixes from winning traces over near-misses. Learn to emit better plan tokens, decompositions, and invariants in the first few steps, under a KL trust region and without training on final answers.

Suggested approach: mine pass@N logs with a verifier. extract prefixes up to the first tool call or K tokens. run preference learning on winner vs loser prefixes plus step-wise advantage-weighted updates from partial checks. update the planner head and small early-layer adapters, freeze late layers.

Why: early decisions carry most of the causal weight on success, pass@N pools expose reliable winner patterns, DPO/IPO and advantage-weighted regression work with preferences and partial rewards, avoiding ground-truth leakage.

Risks: spurious correlations in prefixes, verbosity inflation, domain shift from synthetic near-misses, regressions if KL is loose.

Possible procedure:

1. Initialize: pre-LN transformer with a planner head for plan tokens/templates, per-block low-rank adapters in early layers, verifier API and prefix extractor, global KL and latency budgets.
2. Calibrate: collect pass@N logs on a training pool, label winners with the verifier, record prefixes, partial checks, and problem type features. train a light prefix-success predictor to guide sampling and hard negative mining.
3. Solve: build a preference set of (winner_prefix, loser_prefix, context). train with DPO or IPO on prefixes. add step-wise advantage-weighted regression using partial checks as rewards with a control variate. apply per-step KL penalties and EWC on core capabilities, constrain updates to planner head and early adapters.
4. Compose: at inference, lightly bias decoding toward learned plan tokens and templates (logit offsets, small temperature on planner head). keep standard decoding otherwise. cache priors per problem type.
5. Guard: no-cheat constraints
   - train only on training pool logs, never eval logs
   - use verifier and partial checks that exist at inference (unit tests, proofs, execution), not hidden labels
   - restrict supervision to prefixes and plan tokens, not full final strings
   - enforce per-batch and cumulative KL caps, run regression canaries and length monitors
6. Line search: tune prefix length K, preference temperature, advantage weights, and KL caps on held-out pass@1 uplift vs pass@N, verbosity, and latency. prefer shorter prefixes with large uplift.
7. Refresh: periodically mine fresh pass@N logs, add hard negatives near the decision boundary, merge redundant templates via MDL, retire low-utility plan tokens.
8. Fine-tune: brief consolidation pass to stabilize planner and adapters under tight KL. verify gains persist with normal decoding and no reliance on best-of sampling.

Expected outcome: pass@1 gains from the same pass@N budget, localized updates that do not overfit finals, minimal compute overhead beyond log mining and short preference training.

Literature:

Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning, https://arxiv.org/pdf/2410.08146
RECURSIVE INTROSPECTION: Teaching Foundation Model Agents How to Self-Improve, https://openreview.net/pdf?id=UPoQqreegH
Counterspeech the ultimate shield! Multi-Conditioned Counterspeech Generation through Attributed Prefix Learning, https://arxiv.org/pdf/2505.11958

Project: verifier-guided tree policy iteration

Motivation: best-of-N search induces an implicit decision tree where early nodes determine success. Can we turn those trees into a value signal over plan tokens and perform policy iteration to improve first-shot behavior?

Idea: build a prefix value model that predicts expected verifier success given a partial plan. estimate Q over plan decisions from search trees and partial checks, then perform KL-regularized policy improvement on the planner head.

Suggested approach: construct shallow trees from pass@N rollouts, annotate nodes with verifier outcomes and partial rewards, compute soft Q via backward value propagation. train a small critic over prefixes, improve the planner policy with advantage-weighted cross-entropy while freezing late layers.

Why: tree policy iteration is a principled way to learn from search, verifier and partial checks provide dense rewards, KL-regularized policy updates are stable and sample-efficient.

Risks: value leakage if trees are overfit, instability from off-policy bias, verbosity creep, and interference with non-searched domains.

Possible procedure:

1. Initialize: base model with planner head and early-layer adapters, add a prefix critic (small transformer or MLP on hidden prefix states), verifier API and partial-check probes, set KL and latency budgets.
2. Calibrate: collect pass@N trees by saving top-k beams and sampled branches per problem. annotate each node with partial checks passed and leaf success. estimate off-policy correction weights for branches.
3. Solve: compute soft Q at each node via backward propagation of verifier success with entropy regularization. train the prefix critic to predict Q from hidden states and plan tokens. improve the planner with advantage-weighted cross-entropy or PPO-style updates under KL to the current policy, apply EWC to protect core skills.
4. Compose: at inference, combine planner logits with critic advantages for the first K steps (small additive bias). keep decoding otherwise unchanged. cache advantages by problem type for fast reuse.
5. Guard: no-cheat constraints
   - construct trees only on training pools, never use eval trees
   - rely on verifiers and partial checks available at inference
   - bias only early plan decisions, do not regress on final answers
   - cap per-update and cumulative KL, enforce length and latency bounds, run regression canaries
6. Line search: tune K, critic regularization, advantage temperature, and KL caps on held-out pass@1 vs pass@N gap, stability, and verbosity. prune nodes with low contribution to avoid overfitting.
7. Refresh: rebuild trees on new snapshots, update the critic with fresh logs, remove stale branches, merge redundant plan tokens, maintain a buffer of hard problems for continual improvement.
8. Fine-tune: periodic short run to co-train planner and critic with small KL, freeze late layers, verify gains on unseen tasks and that pass@N remains stable.

Expected outcome: first-shot accuracy approaches best-of-N without extra sampling at inference, stable, interpretable improvements via early plan value shaping, modest compute overhead with strong sample efficiency.