Thank you for posting these sample questions. I must say these certainly help to explain some statistics which used to get cited often around here regarding PISA and/or Canada being the best english reading country.  I am not the kind of person that will take an actual statistic and disregard it b/c it contradicts my understanding of truth, but I must admit that the Finland “balanced literacy” success story has kept me and my wife up many a night.  The statistic didn’t seem right.  We have searched to find out how the PISA could have gauged Finland to be ‘top dog’, despite their using faulted instructional practice and progressive classroom models. 

  I never found the smoking gun, but I did find some curious facts about Finnish society though, (that they are struggling with intense race issues as the country has been almost uniformly white for so long, and now is dealing with black immigration very roughly) and a few things about PISA that raised my eyebrows (that the test’s translation from English into foreign languages allows for the editor to alter questions if they deem the content to be ‘not familiar’ to their cultural group), but no obvious reason could be found.  It seemed that a country using progressivist ideology as their chief instructional model (Finland) had proven to be more successful than countries that used traditional methods and rote learning. 

  My conclusion was something like Juvenal’s question “Quis custodiet ipsos custodes?”.  I figured that the test’s authors had just written the test such that it would highlight spurious ‘skills’, or that it was written at too low a level.  I have seen district piloted tests which tested students on the reading strategies that the district had been using.  The tests came back with the students responding with the same jargon, just as they were taught to -data was gathered, meetings were had, district heads nodded smugly…. success! 

Waitaminit… 

Only there was this niggling little problem still showing up in the classes… kids can’t read!  But the data showed success?  Wha? Unfortunately reality showed the failure.

If you put kids through meaningless exercises and then test kids on the same meaningless things, you can make the data seem to point to a Truth which in fact is a lie.  The students are just showing that they learned what you taught them.  It doesn’t show that they are good readers by the standard of reality. 

This realization is what my (albeit short) sampling of some PISA questions also has lead me to believe.  The test is testing you at an extremely low level of reading comprehension.  If you come out on top of reading at this level, that is not good enough in my book.  It’s just relativism.

Consider:  If your horse won a race against a bunch of horses with founder that are ready for the glue factory, does that mean your horse is really a winner?  Or how about this: you are taking propecia and you grow back 40 baby hairs on your otherwise bald pate; perhaps a better than average result.  Should you consider that ‘achievement’  akin to having a lion-like mane of glory up top? 

Hmmmm, I always tried to aim for something slightly better than this in my endeavours.

And I’m hoping that all teachers will continue to aim higher!

In listening to OECD’s Andreas Schleicher speak about PISA (he is the public face of the assessment tool), he comments that he is aware of the fact that our current measurements, PISA included, tends to go for the low-hanging fruit. He doesn’t deny that.

He also comments that the organization is working hard to develop assessments that measure the “deeper work” that many, many of us are working towards.

So many of our systems are now attuned to the assessments by which they are being publicly judged. The fact that the public has been taught to voraciously devour them, despite the lack of nutritional value, speaks volumes about many things, including the tendency for us to eat whatever is put in front of us.

If we get to the point where our local international assessments actually assess the higher hanging fruit, we might see the change that we need.

Believe me, I’m not advocating more testing. I am saying that, if the tests we administer are destined to define our systems and our thinking about the work that is done within those systems, we need to ensure that those tests are truly reflective or what we want and value.