Tokenization → embeddings → causal Transformer → LM head → softmax → loss → backpropagation. No TensorFlow, no PyTorch, and no hidden autograd.
Repository: tiny-language-model-neuro-js.
Most explanations of language models present correct formulas but hide the path between them inside a framework. I wanted the opposite: one small scenario where every scalar is visible and where the terminal clearly shows incorrect answers before learning and correct answers after it.
The project now has one command:
node src/train.js --generalize --adaptive-teach
It requires Node.js 18.19+ and has no dependencies.
The result first
The model is queried immediately after random initialization:
BEFORE TRAINING — random, usually wrong answers
> can human read ?
model: ? <unk> ...
expected: human can read. [WRONG]
> can fish swim ?
model: ? <unk> ...
expected: fish can swim. [WRONG]
> can cat read ?
model: ? <unk>
Discussion
Be the first to comment
Add your perspective to get the discussion started.