Cognitive Science, Education, Technology, etc.

I'm off to St. Louis to speak that the National Council of Teachers of Mathematics conference. My talk is called What Works for the What Works Clearinghouse. It'll talk about the goals of WWC, what they are looking for in an educational study, why the accepted the Morgan and Ritter paper and how teachers can contribute to the effort.

The talk is on Thursday at 12:30. Stop by and say hello.

I was recently pointed to an article by Alan Schoenfeld in Educational Researcher. The article recounts why Schoenfeld resigned from his position as a "senior content advisor" at the What Works Clearinghouse.

Its an ugly story. Basically, Schoenfeld wrote an article mildly critical of the WWC for a special issue of a journal about the WWC. The Department of Education withdrew funding for the journal, so the article never got published.

This is really sad.

First, Schoenfeld's criticism of the WWC was exactly the kind of constructive criticism that they should welcome. He basically said that the WWC needs to focus more on the construct validity of evaluations. (If you don't want to follow the link, the "construct validity" issue is basically about whether the exam used to test students really corresponds to what we mean when we say a student knows mathematics).

For what it's worth, I think Schoenfeld's criticism goes too far. Sure, you can argue that the FCAT (the Florida state exam) isn't a good measure of mathematical knowledge, but it's clear that the FCAT is a measure of what the state of Florida considers mathematical knowledge (or, at least, as close a measure as they were able to produce, given all the constraints on creating the exam). So, sure, warn people that they may or may not care about FCAT scores. But show them how the kids did on that measure.

Except that, well, here's where the politics gets in the way. You see, Schoenfeld's criticism isn't really of the WWC; it's of NCLB itself. The WWC has an easy way to address construct validity. They can get guys like Schoenfeld to evaluate the FCAT (and other exams) and see if it aligns with what NCTM thinks math is about. And you can get Mathematically Correct to evaluate the FCAT (and other exams) and say whether that's what they think math is about. Hell, they could set up a Wiki and let anyone blab on about how great or pathetic a particular exam is.

And then teachers and administrators can go to the WWC and see which curricula seem to do well on what they, personally, think math is about.

But that's not how NCLB works. Teachers and administrators don't get to say what their mathematical goals are. Only the state gets to say that. And the state says it by constructing exams that embody those goals.

So, a school district might like a curriculum that shows strong performance on the FCAT, but if the district's in Maryland, and the curriculum doesn't do well in Maryland, then the district would be foolish to use it.

Bringing up the issue of construct validity just makes this flaw in NCLB too obvious. You just can't have people questioning whether the tests that are at the heart of NCLB accountability are really testing what we say they're testing. And that, I bet, is the Department of Education's real problem with the criticism.

If you'll bear with me, though, here's my final twist. On both the WWC and NCLB fronts, the Department of Education is being way too sensitive. I'll bet the reality is that most exams don't differ too much from each other. Carnegie Learning has been fairly reckless in supporting evaluations on any "reasonable" measure of mathematics - ETS, NWEA, FCAT, SAT, Iowa, whatever. The fact is, there's a core of mathematics in common in these and, I think, we address that core. Sure, we do better on more problem-solving focused exams than skills-based ones, but we tend to do well on all of them. That's our goal. And that should be everyone's goal.

My biggest reason for skepticism about Qatar's Education City was that you just never heard about it in the US media. Well, CNET has just run a nice series about Education City.

I think the articles pretty well reflect what I saw and heard there. One aspect deserves highlighting:

Early on, the Qatar Foundation and the government agreed not to interfere with academic policies or admissions, two key requirements for participating universities.

This is really the key to understanding that Education City is for real. The universities take this academic freedom very seriously. When a rumor spread that a Jewish faculty member thought that he wouldn't be able to go to the campus, the Dean immediately got involved (the rumor wasn't true).

That's why education is the perfect way to pull the country (and, perhaps, the region) ahead. You either allow the free and open exchange of ideas by all people or you'll never be world class.

The What Works Clearinghouse may be the best idea that the US Department of Education has ever had, and I'm not saying that just because I have a paper in it.

The What Works Clearinghouse masquerades as a website intended to help teachers and administrators sort through the piles of what passes for "research" in the educational community so that they can find out what has been shown, through careful experimentation, to actually work.

But the real mission of the What Works Clearinghouse is to raise the bar - to change the conversation about effectiveness in education. So, when the principal of North High School tells the local paper that they started having math pep rallies and now test scores are up, South High School doesn't just start doing that. They call North and ask "Do you really think it was the pep rallies? What about those 3 new teachers you hired? What about the afterschool tutoring program you started? Do you have any proof that the pep rallies caused the rise in test scores, or was it something else?"

And when that starts happening, the educational market moves. Schools stop buying textbooks or computer software that haven't been proven to increase student learning more than what they're currently doing.

The thing is, the WWC is working.

It hasn't reached principals and adminstrators yet, but the publishers see it coming. They're scared. Terrified.

They know that they need to get some research done, fast.

But they don't quite get it yet.

Consider the What Works Clearinghouse's report on Middle School Math. The WWC identified 10 studies that were either very well designed or moderately well-designed. Of those 10 studies, though, only two show statistically significant results. In other words, its not enough to do a well-designed study. You also need to show, in your study, that your educational materials are actually better than the comparison. And that's hard to do.

Part of this may be that the studies were too small or the the data wasn't analyzed properly, but part of it is due to the fact that its really hard for educational materials to make a difference. How much of a difference does a textbook really make? Compared to a teacher? Compared to school funding or administration?

The fact is, most textbooks are the same. So, when you compare them to each other, you don't see much difference. Lots of other factors are more important in mathematics achievement.

This isn't to say that educational materials don't matter. They matter a lot. But you can't expect to produce educational materials that a just a bit different from what students are currently using and see a big difference. You need to create educational materials that are really different.

And here's where the publishers really don't quite get it.

They think that getting in the WWC is a factor of marketing. They're afraid that, if they're not in there, they won't be able to sell.

But getting into the WWC is up to development, not marketing. You need to produce different materials to produce a different effect.

Consider this press release from Pearson. The release claims that they did a WWC-quality study for their Pearson Prentice Hall Algebra 1 text and that everything's great. The press release points to this report.

Here's what the report says:

• Pearson paid PRES Associates to do a WWC-quality study for them.


• Students improved from pre-test to post-test.


• High-performing students improved the most (from pre- to post-test, not relative to the control group)


• Students with Limited English Proficiency, special ed students and females didn't improve any more or less than other students.


• Students using their text were more satisfied with their educational materials and felt better about math, relative to the control group.

The report bullets "Users of the program consistently performed as well as students who used other programs."

In other words, their text did not work any better than what they compared it to. There isn't any indication that even high-performing students, who are singled out as improving the most, improved relative to the control group.

The main result, improvement from pre- to post-test, is a red herring. Considering that this was a year-long study, I'd certainly hope that students improve from pre- to post-test. In a year of math class, they'd better learn something. Improvement from pre- to post-test is explicitly not what the What Works Clearinghouse cares about.

In the scientific community, this would be a disappointing result. In the educational community, its a press release.

Is it better to be organized than smart?

There's a stunning article in this month's Psychological Science. Duckworth and Seligman present a study that shows that self-discipline is more important to academic success than IQ.

This is a very interesting result, but it needs some follow-up.

Whenever you read about some personality variable (self-discipline, confidence, integrity, etc.) being studied, it's important to remember that the study must be relying on some way to measure that variable. We all have lots of connotations about what self-discipline means (careful readers will notice that I substituted "organized" for "self-disciplined" in my first sentence), but what this scientific result really means is that results on some group of tests which are claimed to measure self-discipline correlate with academic success (which is also measured in some way). In this case, the study used a combination of measures, including a questionnaire, reports from parents and teachers and a kind of goofy test where kids were asked whether they'd like a dollar right now or the promise of $2 in a week. Birds-in-hand and present value calculations notwithstanding, the correct answer is, apparently, to wait.

So, do the measures used correspond to what I'd consider self-discipline? I think so, but I'd be more convinced if they found that self-discipline could be distinguished from other characteristics that also, presumably, lead to academic success (like desire to succeed academically, belief in future success, belief in the value of studying, accurate self-assessment, etc.). To some extent, I suspect that the self-discipline measures are tapping aspects of all of these personality characteristics, so its hard to tell which one is really the crucial one to success.

There's a related issue with the outcome measures. What counts as academic success? Again, Duckworth and Seligman use multiple measures. Some seem fine (GPA, performance on an achievement test, admission to a selective high school), but others seem more like mediators than outcome measures. One of the outcome measures was the time of the day that students begin homework. Students with high self-discipline begin homework earlier. That's fine, but perhaps the time when you begin homework is also a predictor of success. In other words, you might start your homework earlier because you have high self-discipline, but you also might behave that way because you have an involved parent. And maybe it doesn't matter why you started your homework early, just that you did.

This stuff is important because, if its true that self-discipline is what really matters, then the next step is to figure out how to increase students' self-discipline. If having an involved parent leads to early homework which leads to academic success (just as strongly as student self-discipline), then let's focus on getting the kids homework started early, rather than trying to change self-discipline.

Finally, there's the issue of what IQ means. One argument against the importance of this study is that there's a difference between being smart and having a high IQ (notice how, in the first sentence, I substituted "smart" for IQ? Well, its just a blog). This is, fundamentally, the argument against any use of IQ - it doesn't capture all of what we mean when we say someone's smart. So, this is the same problem as with self-discipline -- does the measure match what we think it means? However, I'm less concerned with the IQ measure in this study. You may or may not agree that IQ=smart, but you probably have a good idea of what an IQ test is measuring. You've taken one - or a test something like one. And whatever it is that an IQ test measures, it seems pretty close to whatever it is that schools measure.

So finding something that beats IQ in predicting of academic success really is a stunning result.

The trip to Doha, Qatar was very successful - we even got press coverage.

Qatar's ambitions for Education City are bolder than I thought. In addition to the college campuses, Education City hosts (or will host) a charter school, a technology park (Microsoft, Rand and Price Waterhouse Coopers are all there), even the Al Jazeera children's channel.

They're building a large support network around the college campus idea. In order to prepare students for the Education City colleges, they have an Academic Bridge Program, which is basically a prep-school year. They've also set up a charter school system which teaches in English. This isn't strictly linked to Education City, but they're counting on some of those graduates to feed the schools.

Cornell and Virginia Commonwealth are currently the only two schools that have their own buildings. The CMU, A&M and Georgetown campuses are under construction. There's also a large area that appears to be reserved for more campuses, so stay tuned. Stuff's going on there.

I'm off tonight to visit Education City in Qatar.

In principal, Education City may be one of the best big ideas in the world. Qatar is a small country on the Persian Gulf. They've invited several prestigious universities (currently Carnegie Mellon, Cornell, Georgetown, Texas A&M and Virginia Commonwealth) to set up satellite campuses there. The idea is to provide a first-class postsecondary education in the Arab world. This can help build a future for Arab countries when the wells run dry, and it does it in a way that builds on the Arab world's historic role as a center of learning in the world. If this works, kids in the Arab world will be able to see their future in a way that builds on the past but allows them to participate in a progressive future.

So, why haven't you heard of it? Why isn't Tom Friedman writing about it? Google for it, and you see just about no coverage in the west, other than press releases by the universities (this MSNBC article is an exception).

I'm not sure. Maybe it's too new. Maybe this vision is controversial in the Arab world and viewed with skepticism in the Western world. Maybe it's too much to believe that this big vision could come out of a country as small as Qatar. Maybe this is too big a vision to be driven by a country as small as Qatar. Maybe its some kind of scam.

I don't know, but I'm going there to find out. At the very least, we'll help some kids learn some math.