LLM Zoo 🦓

Transformer: 24 layersOutput Projection2880 → 196KRMSNorm2880Mixture of Experts4 / 32 experts2880 → 2088 → 2880RMSNorm2880Grouped Query Attention64 heads, 8 KV (8:1)2880 → 2880RMSNorm2880ConcatenateEmbedding196K → 2880Vision EncoderViT, 14px patchesimage → 2880Text (input)Image (input)