Needle in a Haystack

Definitions

1. A type of evaluation that tests whether a model can retrieve or use one small crucial detail hidden inside a very large context. The phrase is often used to discuss long-context performance and whether large windows are actually useful.

   In plain English: A test of whether a model can find one important detail hidden in a lot of context.

   Example: "Their long-context benchmark included a needle in a haystack test to see whether the model could find one key policy exception buried in hundreds of pages."

Related Terms

• Needle in a Haystack Test
• AI Evaluation
• Context Extension
• Context Length
• Elicitation
• Evaluation Harness
• Faithfulness
• Large Context Model
• LLM Judge
• Proxy Task
• RAG Evaluation
• Self-Evaluation