Blog

NCA Pre-Pre-Training

Can language models learn useful priors without ever seeing language? We explore this question by pre-pre-training LLMs on neural cellular automata, a fully synthetic and computational source of training data.