Once upon a time there were two small towns, Saint Augustine and Saint Bernardus. The children of both towns went to the same school.
The citizens of these towns decided to get together and figure out how well their students were doing in school, since this seemed to be all the rage in other towns at the time. So the people adopted a test that their state government had created to see how well kids were learning. They administered the test and were surprised to find out that Saint Augustine kids did significantly worse than Saint Bernardus kids. When averaged together the two town's kids did worse than the average in the state, with Saint Bernardus doing better than the state average and Saint Augustine doing much worse.
Word of the overall result being below state average got back to state officials who soon showed up and had a meeting with the mayors of both towns.
'The school', said the state officials, 'would have to be closed down.' It was state policy that schools that scored below average were disbanded and the students sent to other schools, he further clarified.
The mayor of Saint Bernardus spoke up, 'but our town's children are doing better than the state average', he objected.
'Dont play games with numbers with me, it's the school we rate, not towns', was the response.
Next the mayor of Saint Augustine spoke up. 'I do some statistics as a hobby on the side and I took a closer look at our test results and noticed that many of our kids missed all the same questions. I got a copy of the test and noticed that these were questions that required identifying a color as part of the solution.'
'So?' asked a state representative with a hint of disdainful boredom.
'Well, half of the kids in our town are color-blind', responded the Saint Augustine mayor. 'None in Saint Bernardus are.'
'I'm afraid I can't help that', stumbled the state official, suddenly nervously pulling at his collar. 'The policy is to use the overall numbers. People are not interested in studying every school in detail to decide whether you are making excuses or whether there may in fact not be a reason our tests don't accurately reflect the quality of your learning environment. That would cost money you know!'
With that the state officials hurriedly stood up, left the room and drove out of town.
The school was closed the following month.
The state official gave a toast at the state-sponsored annual school reform extravaganza, praising the virtues of his own department, using as an example how he and his colleagues brought improved education to the children of Saint Augustine and Saint Bernardus.