Models are compared on standard tests, which is useful but easy to over-read. A model can score well by having seen similar questions, or because the test rewards a narrow skill that does not reflect the messy tasks of real use.
Treating a single number as the whole truth hides where a model is weak. Sensible evaluation uses several tests, checks for overlap with training data, and ultimately judges a model on the specific job it will actually do.