It predicts. It doesn't look up.
The transformer chooses one character token at a time from its learned weights. The demo runs actual inference, without a hosted AI provider or a canned answer lookup.
Autoregressive generationPEBBLE ECOSYSTEM / 02
Learning, at
a human scale.
A real transformer. Exactly two million parameters. Built and trained in a language you can explore.
Meet PebbleLM: a small educational model that makes the journey from code to learned behavior visible.
NO HIDDEN BIG MODEL
Actual trained weights. No external AI API. Code, data and weights are MIT.
2,000,000trainable parameters
4 × 4layers × attention heads
40authored training topics
MITsource, data & weights
01 / MEET THE MODEL
From characters
to learned answers.
A compact experiment with the whole stack open: the network, tokenizer, training loop, dataset and checkpoint.
A small, specific curriculum.
Forty topics cover Pebble syntax, programming concepts and machine learning. Short questions work best; unfamiliar wording can produce a wrong answer.
Explore the actual outputs →Open all the way down.
Inspect every parameter, run the checkpoint on your CPU, or change the Pebble source and retrain. Code, original data and released weights carry the MIT license.
Run it on your machine →BUILT FOR EXPLORATION
Learn how transformers work
Trace embeddings, causal attention, training loss and sampling in readable source.
Experiment with tiny models
Compare prompts and temperatures, then inspect where memorization stops generalizing.
Build a focused demo
Use the JSON endpoint or local CLI in educational prototypes. It isn't trained for general chat or reliable factual advice.
02 / EXPERIMENT
A question. A small experiment.
Start with a familiar topic, then change its wording. Watch what a tiny model remembers—and where it struggles.
PEBBLELM / OUTPUTReady to experiment
A small question can teach you a lot about a small model.
Your output appears here after you generate.
Generated from the released trained
weights
An educational model, not a general assistant. It learns a narrow set of programming and ML topics. It can repeat an answer, mix learned phrases or produce incorrect text. Changing wording is a test of generalization, not a request for new knowledge.
Your recent experiments
Last three results, held only in this page’s memory.
03 / RECORDED OUTPUTS
Same topic.
Different wording.
Explore recorded greedy generations from the released checkpoint. Compare a seen training phrasing with a reserved test phrasing.
SEEN WORDINGExact match
What is Pebble?
Pebble is a small programming language.
Greedy generation · published canonical evaluation
RESERVED WORDINGExact match
How does Pebble work?
Pebble is a small programming language.
Training target:
Greedy generation · published reserved evaluation
Topics and target answers are shared between splits. Seen phrasing: 38/40 exact answers. Reserved phrasing: 4/40. These examples are recorded results; use the playground for a fresh run.
04 / EVIDENCE
Small model. Open notebook.
Exact matches on the first training phrasing for each topic.
Read every familiar-prompt output ↗Exact matches when those same topics are asked with held-out wording.
Read every reserved-prompt output ↗Remembering is easier than generalizing.
Both sets use the same 40 topics and target answers. This measures sensitivity to phrasing. It does not measure new knowledge or general reasoning.
Released checkpoint measurements
PRETRAINING2,000 updates
Packed 299-position sequences
REFINEMENT1,000 updates
Individual question/answer prompts
SELECTED VALIDATION LOSS0.0551 nats
Validation selects the checkpoint
05 / UNDERSTAND
From a character to a prediction.
01 / ENCODE CHARACTERS
A fixed vocabulary. Learned vectors.
The tokenizer maps printable ASCII, newline, tab and four special tokens into 101 IDs. Each ID gets a learned 200-dimensional vector, plus a learned position embedding.
101 tokens × 200 dimensions + 299 positions × 200
dimensionsInspect the tokenizer ↗
Read the characters
A fixed vocabulary turns your question into character tokens. The model sees at most 299 characters of context.
Connect the context
Four decoder blocks use causal attention. Each position learns from the characters that came before it.
Predict one more
The tied output projection assigns probabilities to the next character. Temperature controls sampling.
Repeat and inspect
Generation continues until the end token or the limit. Compare the output with the published evaluation.
DEFINED AND TRAINED IN PEBBLE
The whole learning loop is open.
The network, tokenizer, training loop and sampling are Pebble programs. TensorFlow.js/WASM provides numerical kernels and automatic differentiation.
Read the actual network// Exact trainable parameter count
(101 + 299) × 200
+ 4 × (4 × 200² + 2 × 200 × 800)
= 2,000,00006 / MADE TO BE OPENED
Take it beyond the page.
Run inference locally, call the demo endpoint, inspect the source or reproduce training. No model API key or database setup.
01 / RUN LOCALLY
Your CPU. The released weights.
Node 24, one install, then a prompt.
↗02 / CALL THE ENDPOINTA small JSON interface.
POST a question with bounded sampling options; receive the answer and elapsed time.
↗03 / EXPERIMENTRead it. Change it. Retrain it.
Tokenizer, training data, optimizer loop and evaluation are open.
↗curl https://pebble-llm.vercel.app/api/generate \
-H 'Content-Type: application/json' \
--data '{"prompt":"What is Pebble?","temperature":0,"seed":2026,"maxNewTokens":96}'Public demo endpoint · 160-character ASCII prompt · 96-character output limit
07 / KEEP EXPLORING
One ecosystem. Three ways in.
LANGUAGE / BUILD
Learn the language.
Run Pebble, inspect how programs behave and explore the machine-learning guide.
Open the machine-learning guideMODEL / REPRODUCERun your own experiment.
Reproduce training, inspect the real weights and compare memorization with new phrasing.
Read the reproduction stepsSENTINEL / APPLYPut behavior under a lens.
Explore a separate Pebble-trained model that scores typed agent actions before they run.
Open Sentinel