Stat testing and weighting
Tuesday May 5, 2009 by Yvonne Goldberg
Hoping someone can help. We conduct a tracking study with quarterly waves, data is weighted to reflect actual population breakdown. We compared and stat test waves against eachother in our SPSS Dimensions software.
We have never had problems before with stat testing reliability with weighted data and/or with this software except with this particular study and we cannot determine why, so have resorted to manual stat testing in the meantime via an external software.
The only "abnormality" with this data is that there are some missing responses for most questions (but not the one question off which weight factors are determined). This is because our client fields this and has chosen to not force responses to some questions. We do remove nonresponders from the base for each question; there are only about 2% for a given question. Sample size for each wave ranges from 1200 to 5000.
Stat testing, however, is not working between the waves. It is not picking up clear instances of sig differences, and vice versa. In one particular table, where the sample size for a given wave for that question is 200+, it is not showing sig testing at all like it does for sample sizes of less than 30.
I'm at a loss and SPSS tech support was unable to help, not understanding the issue and giving a standard (non-applicable) response from their tech database despite us sending them documentation.
Can anyone help? Any insight would be appreciated.
» Post to this Topic
Stat testing / Bonferroni correction
Wednesday May 6, 2009 by Ian L. Straus
This is a reply to: Stat testing and weighting
Check to see whether the Bonferroni correction is checked to be on or off.
I've had a similar experience with multi-wave data: The more waves you have, the harder it is for a given difference to be significant because the Bonferroni correction accounts for the chance of a spurious random result, which gets really high if you have a high number of potential comparisons.
Decide whether what you want to test for is a signifcant difference vs. the single previous period, rather than a significant difference from ALL previous periods.
You may want to take the same data, select just the two most reent waves, and run the same table and tests. Just to test what's going on.
» Reply to this Post