Abu Dhabi publishes a training mixture, including approximately 10 trillion synthetic tokens
· L4 Models · ~10T synthetic · Confirmed
Most laboratories keep secret what their models were taught on. A research institute in Abu Dhabi has published its recipe for six models, from 0.9 billion to 375 billion parameters. Each saw approximately 20 trillion tokens, and approximately 10 trillion synthetic tokens were used across the whole set, with no breakdown per model.