synthetic training data

Definitions

1. Training data generated by AI models rather than collected from humans, used to train or fine-tune other models. While dramatically cheaper to produce, synthetic data risks model collapse if used exclusively across generations, as each generation loses fidelity to real-world distributions.

   In plain English: Fake data made by one AI to teach another AI, kind of like a teacher writing practice problems instead of using real test scores.

   Example: They bootstrapped their specialized medical model using synthetic training data from GPT-4, then fine-tuned on a smaller set of expert-annotated cases.

Related Terms

• Training Data
• Synthetic Data
• Model Collapse
• Neural Scaling Laws
• Compute Optimal Training
• Data Curation
• Data Scaling
• Self-Instruct
• Unstructured Data AI