1,000 Words (give or take) on the Nashville Public Education Foundation’s Latest “Citywide Survey on Education”
After reading the Nashville Public Education Foundation (NPEF) press release about a survey of “500 randomly sampled likely voters in Nashville,” I was left with so many questions – about the methodology, about the respondents, about the survey questions themselves, about the motivation of the foundation in conducting the survey (especially since it was not conducted in collaboration – or even in cooperation – with Metro Nashville Public Schools).
The press release states that the “margin of error for the sample is +/- 4.4 percentage points at the 95% level of confidence.” Hearkening back to what I learned in high school statistics class (in a Tennessee public school, no less), I recall that a 95% confidence level means there’s only a 5% chance that the data is wrong, and a margin of error tells you how much your results will differ from the real population value.
There were 484,814 registered voters in Davidson County as of August 4, 2022. I looked up the formula to calculate sample size to achieve a certain margin of error, and according to that formula, the margin of error reported by NPEF can be achieved by polling 496 members of a population of 484,814. Good job NPEF; they polled 500 Davidson Co. voters. However, I still had questions, because so much of what they’re reporting is specific to MNPS.
I reached out to Katie Cour, President and CEO of NPEF, to ask what percentage of respondents were MNPS parents. I also asked what they meant by “likely voter,” and was that the only requirement for a respondent’s answers to be included?
According to Cour, “33% of respondents had children in the household and of those, 64% were public school parents and 26% private school (6% home schooled). Likely voter is defined as a voter who voted in one of the last 3 November elections (’16, ’18, ’20) or a new registrant who says they are certain to vote this fall.”
With Cour’s response in hand, I got out a calculator. 33% of 500 is 165, and 64% of 165 is 105.6. So, 105.6 survey respondents were MNPS parents. I then went back to the formula to calculate sample size. I know there are at least 82,610 public school parents in MNPS (accounting only for one parent per student). The formula results tell me that 493 MNPS parents should have been polled to achieve a margin of error of 4.4 for the portion of the survey data NPEF claims to be representative of the thoughts and beliefs of MNPS parents. Looks like the margin of error for that portion of the results is 10. I learned during my public school education that an acceptable margin of error at a 95% confidence level should be between 4 and 8. I’m starting to question whether or not I should trust this survey data as a reliable measure of MNPS parent “perceptions of local public schools.”
I was particularly puzzled by the assertion that 41% of Nashvillians who are aware of the state’s new school funding formula, TISA (Tennessee Investment in Student Achievement), are in support of it. A lot of folks I know believe that TISA will underfund Nashville public schools, and I know that Nashvillians verifiably support investing in our students, so it seems odd to me that so many Nashvillians would support the formula. I already know that NPEF supports TISA, because I received an email from them on April 15, 2022 stating just that. They sent that correspondence out even though it contradicted the already-stated stance of MNPS district leaders on TISA.
The thing I find most interesting about NPEF is that their NTEE (National Taxonomy of Exempt Entities) code is B11 (Single Organization Support: Educational Institutions and Related Activities), and the mission they state on their FY20 990 filing (the most recent one publicly available) with the IRS is “to ensure every child in Nashville has access to a great public education,” so clearly that “single organization” they’re supposed to be supporting is Metro Nashville Public Schools. And yet – they do things that contradict that, such as supporting legislation that clearly isn’t in the best interests of public school students in Nashville and conducting surveys about education without collaborating with the very organization they’re created to support.
But – I digress. Back to the issue at hand: the press release about the survey findings and the puzzling assertion that 41% of Nashvillians who are aware of TISA support it.
Once again, I reached out to NPEF CEO Cour, and she responded promptly to my request to share the exact questions used in the questionnaire. One question about TISA reads “How much have you seen or heard about Tennessee’s new school funding system, the Tennessee Investment in Student Achievement?” The answer options are “A lot,” “Some,” “Not too much,” “Nothing at all,” and “Don’t Know.” Only those who responded “A lot” or “Some” were asked the next question about TISA: “Do you favor or oppose the new Tennessee school funding formula, the Tennessee Investment in Student Achievement?” They were then asked if they “strongly” or “somewhat” favored or opposed “the new funding formula.”
I went back to Cour to ask how many respondents answered “A lot” or “Some” to the question about how much they’d seen or heard about TISA. She responded that 150 “said they had heard of TISA.” So that means that 150 were asked the question about favoring or opposing the formula. An average of 61.4% of registered voters meet the “likely voter” standard as defined by Cour, so that gets the population down to 297,676. Looks like we have a different margin of error than the one reported in the NPEF press release on that TISA data, too; it’s an 8 when you only have 150 responses for a population that size.
But I think the problem with that data runs a bit deeper than just the higher margin of error. The question: “How much have you seen or heard about Tennessee’s new school funding system….” I’m pretty sure that’s what’s known as a bad screening question. The only precursor to being asked to state an opinion about TISA is to have heard about it.
Let’s play a game; how many things have you heard about yet you still know nothing about that thing? I’ll go first; I’ve seen and heard a lot about cryptocurrency, but I don’t know anything about it. I haven’t even tried to learn about it (and I definitely wouldn’t be qualified to answer a question about it intended to get at the overall perception of a specific population regarding crypto).
What was it Samuel Clemens (pen name Mark Twain) said about statistics? It’s a quote often used to allude to statistics being lies, but when you read through the entirety of Clemens’ assertions, you come to understand that he wasn’t criticizing the collection and interpretation of quantitative data to increase understanding and even help inform decisions. He was chastising those who do so incompetently.