I research applied AI: knowledge transfer into small models, long-context efficiency, neural codecs, and quantization. The work is published openly: paper, weights, and code.
I care about systems that run on ordinary hardware: small models with the capabilities of large ones, cheap inference, and control over data. The main works are below.
Research · arXivarXiv:2502.08213MITTrained for under $10
01LLM Modules: Knowledge Transfer from a Large to a Small Model using Enhanced Cross-Attention
Knowledge transfer from a large model to a small one
A way to give a small model the knowledge of a large one, without distillation and without full retraining. The large model is frozen as a knowledge source, the small one trains as the reasoner, and cross-attention carries the knowledge between them.
Method
A frozen Qwen2-1.5B teacher and a trained GPT-Neo-125M student on the Bespoke-Stratos-17k reasoning set. The teacher's embeddings and LM-head are stripped, its output flows through a cross-attention layer, and the student gets its own LM-head under the teacher's tokenizer.
Results
15 epochs, about 5 hours on a single NVIDIA L4, under $10 in training cost; loss fell from 13.8 to about 1.1. The small model reproduces the teacher's structured reasoning.
Research · Paper + DOIDOI 10.57967/hf/8973Apache-2.0~5.04M parameters
02MPT-VC: Matryoshka Phase-on-Torus Video Codec
A neural video codec built on deterministic operators
A neural video codec and tokenizer built entirely from deterministic operators. Video compresses into a single rate-ordered stream: the start of the file gives a preview, more bytes give higher quality, the whole file gives full quality. One file covers the entire quality ladder without re-encoding.
Method
A Matryoshka Laplacian pyramid over three levels, windowed axial 3D RoPE attention at linear cost, a scene-aware phase-jump across cuts, FSQ with no codebook, and a bit-exact arithmetic entropy coder.
Results
On UVG it beats Nvidia's Cosmos Tokenizer on average Y-PSNR at lower time and memory. The checkpoint is 20.36 MB, trained on ~290 hours of video.
An open model that cleans up raw text: restores punctuation, capitalization, and paragraphs, and fixes typos and spelling. Kazakh, Russian, English. It keeps your words and meaning: it corrects, it does not rewrite.
Method
A finetune of Qwen3.5-0.8B plus our own quantization: 4/8/16-bit quantization-aware training with bit-exact train==deploy parity. Runs in llama.cpp, LM Studio, and Ollama on an ordinary CPU.
A way to fit long context into modest memory: neighboring tokens are grouped into entities before the main stack of layers, so attention works over groups instead of raw tokens. Attention memory scales as G² instead of N².
Method
A custom layer 0 clusters neighboring tokens at prefill and averages them into groups; a small adapter processes the grouped embeddings. Only that layer and the adapter train; the rest stays frozen. Built on Qwen3-0.6B.
Honest status: no published numbers yet. This is an active direction being taken to a showable state.
