Lying numbers, the people who use them, the people who count

The trouble with reporting numbers in the news is that the people who provide the numbers have an agenda and the people who write the news don’t pay enough attention to what they’re being fed.  We have a great example here in Texas—great in the sense of being a good example, because in fact children and taxpayers are being hurt.

Last week I ran across a piece from Jack Shafer in Slate praising a new book that reminds us to be skeptical of the numbers.   “No debate lasts very long without a reference to data, and as the numbers boil their way into the argument, you must challenge them or be burned blind by them,” warns Shafer, and he highlights some of the authors’ examples of cases in which funny figures are used for moral and political suasion.  He’s reminding us all to be vigilant; he reminded me of a case that makes the point.

This came to my attention earlier this month in a column by Rick Casey in the Houston Chronicle.  He wrote about  Houston state representative  Scott Hochberg, regarded across Texas as the man with the best understanding of our school finance system, and his discovery that the Texas Education Agency was manipulating statistics to show dramatic improvement in the number of schools with better student performance on the Texas Assessment of Knowledge and Skills, the standardized test of primary and secondary students’ mastery of basic skills.

How could they do that?  Through use of a statistical tool called the Texas Projection Measure, which was developed by a national testing company.

Hochberg asked…how many correct answers a fourth-grader with barely passing math and reading scores at Benavidez Elementary in Houston needed to be counted as “passing” the writing test.

The unbelievable answer Hochberg had reached himself was confirmed…The child needed zero correct answers for his or her teachers and administrators to get credit for his or her “improvement.”

That’s right: zero.

Under questioning by someone they couldn’t bullshit, the TPM developers had to admit the tool considers inappropriate data to come up with this remarkable result.  That made news (here and here and here), and not just because the state’s education commissioner failed to attend the hearing.  The governor’s appointee sent staff to defend the use of a statistical tool which, we now know, translates the reality of continuing poor performance by students on standardized tests into the appearance of improved performance by schools and the bureaucrats who run them.

And all just months before the governor stands for re-election.

So there’s no real surprise that the education chief now opens his mouth (via e-mail), resorting to one of the favorite tactics of the governor himself: accusing anyone found not to be in lockstep with the powers that be of trying to harm the state and the little skoolchirrun of Texas.  He also says he’s considering new ways to use TPM: maybe use it less, or leave it up to the independent school districts to decide if they want to use it.

What?!  The TPM is flawed; it’s designed to let school administrators pad the TAKS results to make it appear that their schools—and they themselves—are doing a better job.  Casey sums it up well:

That the commissioner would allow a system such as the TPM to be put into place without serious evaluation, and then defend it as “reliable and accurate,” tells me that neither he nor the governor who appointed him take seriously the most daunting and vital challenge facing Texas: building up Texas by educating our children.