Cognitive Science, Education, Technology, etc.

NCTM

2006-04-26T09:17:00.000-04:00

I'm off to St. Louis to speak that the National Council of Teachers of Mathematics conference. My talk is called What Works for the What Works Clearinghouse. It'll talk about the goals of WWC, what they are looking for in an educational study, why the accepted the Morgan and Ritter paper and how teachers can contribute to the effort.

The talk is on Thursday at 12:30. Stop by and say hello.

Troubles at What Works

2006-04-09T20:28:00.000-04:00

I was recently pointed to an article by Alan Schoenfeld in Educational Researcher. The article recounts why Schoenfeld resigned from his position as a "senior content advisor" at the What Works Clearinghouse.

Its an ugly story. Basically, Schoenfeld wrote an article mildly critical of the WWC for a special issue of a journal about the WWC. The Department of Education withdrew funding for the journal, so the article never got published.

This is really sad.

First, Schoenfeld's criticism of the WWC was exactly the kind of constructive criticism that they should welcome. He basically said that the WWC needs to focus more on the construct validity of evaluations. (If you don't want to follow the link, the "construct validity" issue is basically about whether the exam used to test students really corresponds to what we mean when we say a student knows mathematics).

For what it's worth, I think Schoenfeld's criticism goes too far. Sure, you can argue that the FCAT (the Florida state exam) isn't a good measure of mathematical knowledge, but it's clear that the FCAT is a measure of what the state of Florida considers mathematical knowledge (or, at least, as close a measure as they were able to produce, given all the constraints on creating the exam). So, sure, warn people that they may or may not care about FCAT scores. But show them how the kids did on that measure.

Except that, well, here's where the politics gets in the way. You see, Schoenfeld's criticism isn't really of the WWC; it's of NCLB itself. The WWC has an easy way to address construct validity. They can get guys like Schoenfeld to evaluate the FCAT (and other exams) and see if it aligns with what NCTM thinks math is about. And you can get Mathematically Correct to evaluate the FCAT (and other exams) and say whether that's what they think math is about. Hell, they could set up a Wiki and let anyone blab on about how great or pathetic a particular exam is.

And then teachers and administrators can go to the WWC and see which curricula seem to do well on what they, personally, think math is about.

But that's not how NCLB works. Teachers and administrators don't get to say what their mathematical goals are. Only the state gets to say that. And the state says it by constructing exams that embody those goals.

So, a school district might like a curriculum that shows strong performance on the FCAT, but if the district's in Maryland, and the curriculum doesn't do well in Maryland, then the district would be foolish to use it.

Bringing up the issue of construct validity just makes this flaw in NCLB too obvious. You just can't have people questioning whether the tests that are at the heart of NCLB accountability are really testing what we say they're testing. And that, I bet, is the Department of Education's real problem with the criticism.

If you'll bear with me, though, here's my final twist. On both the WWC and NCLB fronts, the Department of Education is being way too sensitive. I'll bet the reality is that most exams don't differ too much from each other. Carnegie Learning has been fairly reckless in supporting evaluations on any "reasonable" measure of mathematics - ETS, NWEA, FCAT, SAT, Iowa, whatever. The fact is, there's a core of mathematics in common in these and, I think, we address that core. Sure, we do better on more problem-solving focused exams than skills-based ones, but we tend to do well on all of them. That's our goal. And that should be everyone's goal.

Education City in the news

2006-01-21T17:53:00.000-05:00

My biggest reason for skepticism about Qatar's Education City was that you just never heard about it in the US media. Well, CNET has just run a nice series about Education City.

I think the articles pretty well reflect what I saw and heard there. One aspect deserves highlighting:

Early on, the Qatar Foundation and the government agreed not to interfere with academic policies or admissions, two key requirements for participating universities.

This is really the key to understanding that Education City is for real. The universities take this academic freedom very seriously. When a rumor spread that a Jewish faculty member thought that he wouldn't be able to go to the campus, the Dean immediately got involved (the rumor wasn't true).

That's why education is the perfect way to pull the country (and, perhaps, the region) ahead. You either allow the free and open exchange of ideas by all people or you'll never be world class.

What counts in What Works

2006-01-16T20:57:00.000-05:00

The What Works Clearinghouse may be the best idea that the US Department of Education has ever had, and I'm not saying that just because I have a paper in it.

The What Works Clearinghouse masquerades as a website intended to help teachers and administrators sort through the piles of what passes for "research" in the educational community so that they can find out what has been shown, through careful experimentation, to actually work.

But the real mission of the What Works Clearinghouse is to raise the bar - to change the conversation about effectiveness in education. So, when the principal of North High School tells the local paper that they started having math pep rallies and now test scores are up, South High School doesn't just start doing that. They call North and ask "Do you really think it was the pep rallies? What about those 3 new teachers you hired? What about the afterschool tutoring program you started? Do you have any proof that the pep rallies caused the rise in test scores, or was it something else?"

And when that starts happening, the educational market moves. Schools stop buying textbooks or computer software that haven't been proven to increase student learning more than what they're currently doing.

The thing is, the WWC is working.

It hasn't reached principals and adminstrators yet, but the publishers see it coming. They're scared. Terrified.

They know that they need to get some research done, fast.

But they don't quite get it yet.

Consider the What Works Clearinghouse's report on Middle School Math. The WWC identified 10 studies that were either very well designed or moderately well-designed. Of those 10 studies, though, only two show statistically significant results. In other words, its not enough to do a well-designed study. You also need to show, in your study, that your educational materials are actually better than the comparison. And that's hard to do.

Part of this may be that the studies were too small or the the data wasn't analyzed properly, but part of it is due to the fact that its really hard for educational materials to make a difference. How much of a difference does a textbook really make? Compared to a teacher? Compared to school funding or administration?

The fact is, most textbooks are the same. So, when you compare them to each other, you don't see much difference. Lots of other factors are more important in mathematics achievement.

This isn't to say that educational materials don't matter. They matter a lot. But you can't expect to produce educational materials that a just a bit different from what students are currently using and see a big difference. You need to create educational materials that are really different.

And here's where the publishers really don't quite get it.

They think that getting in the WWC is a factor of marketing. They're afraid that, if they're not in there, they won't be able to sell.

But getting into the WWC is up to development, not marketing. You need to produce different materials to produce a different effect.

Consider this press release from Pearson. The release claims that they did a WWC-quality study for their Pearson Prentice Hall Algebra 1 text and that everything's great. The press release points to this report.

Here's what the report says:

Pearson paid PRES Associates to do a WWC-quality study for them.

Students improved from pre-test to post-test.

High-performing students improved the most (from pre- to post-test, not relative to the control group)

Students with Limited English Proficiency, special ed students and females didn't improve any more or less than other students.

Students using their text were more satisfied with their educational materials and felt better about math, relative to the control group.

The report bullets "Users of the program consistently performed as well as students who used other programs."

In other words, their text did not work any better than what they compared it to. There isn't any indication that even high-performing students, who are singled out as improving the most, improved relative to the control group.

The main result, improvement from pre- to post-test, is a red herring. Considering that this was a year-long study, I'd certainly hope that students improve from pre- to post-test. In a year of math class, they'd better learn something. Improvement from pre- to post-test is explicitly not what the What Works Clearinghouse cares about.

In the scientific community, this would be a disappointing result. In the educational community, its a press release.

What do you need to succeed in school?

2005-12-19T14:33:00.000-05:00

Is it better to be organized than smart?

There's a stunning article in this month's Psychological Science. Duckworth and Seligman present a study that shows that self-discipline is more important to academic success than IQ.

This is a very interesting result, but it needs some follow-up.

Whenever you read about some personality variable (self-discipline, confidence, integrity, etc.) being studied, it's important to remember that the study must be relying on some way to measure that variable. We all have lots of connotations about what self-discipline means (careful readers will notice that I substituted "organized" for "self-disciplined" in my first sentence), but what this scientific result really means is that results on some group of tests which are claimed to measure self-discipline correlate with academic success (which is also measured in some way). In this case, the study used a combination of measures, including a questionnaire, reports from parents and teachers and a kind of goofy test where kids were asked whether they'd like a dollar right now or the promise of $2 in a week. Birds-in-hand and present value calculations notwithstanding, the correct answer is, apparently, to wait.

So, do the measures used correspond to what I'd consider self-discipline? I think so, but I'd be more convinced if they found that self-discipline could be distinguished from other characteristics that also, presumably, lead to academic success (like desire to succeed academically, belief in future success, belief in the value of studying, accurate self-assessment, etc.). To some extent, I suspect that the self-discipline measures are tapping aspects of all of these personality characteristics, so its hard to tell which one is really the crucial one to success.

There's a related issue with the outcome measures. What counts as academic success? Again, Duckworth and Seligman use multiple measures. Some seem fine (GPA, performance on an achievement test, admission to a selective high school), but others seem more like mediators than outcome measures. One of the outcome measures was the time of the day that students begin homework. Students with high self-discipline begin homework earlier. That's fine, but perhaps the time when you begin homework is also a predictor of success. In other words, you might start your homework earlier because you have high self-discipline, but you also might behave that way because you have an involved parent. And maybe it doesn't matter why you started your homework early, just that you did.

This stuff is important because, if its true that self-discipline is what really matters, then the next step is to figure out how to increase students' self-discipline. If having an involved parent leads to early homework which leads to academic success (just as strongly as student self-discipline), then let's focus on getting the kids homework started early, rather than trying to change self-discipline.

Finally, there's the issue of what IQ means. One argument against the importance of this study is that there's a difference between being smart and having a high IQ (notice how, in the first sentence, I substituted "smart" for IQ? Well, its just a blog). This is, fundamentally, the argument against any use of IQ - it doesn't capture all of what we mean when we say someone's smart. So, this is the same problem as with self-discipline -- does the measure match what we think it means? However, I'm less concerned with the IQ measure in this study. You may or may not agree that IQ=smart, but you probably have a good idea of what an IQ test is measuring. You've taken one - or a test something like one. And whatever it is that an IQ test measures, it seems pretty close to whatever it is that schools measure.

So finding something that beats IQ in predicting of academic success really is a stunning result.

Doha Details

2005-12-18T22:30:00.000-05:00

The trip to Doha, Qatar was very successful - we even got press coverage.

Qatar's ambitions for Education City are bolder than I thought. In addition to the college campuses, Education City hosts (or will host) a charter school, a technology park (Microsoft, Rand and Price Waterhouse Coopers are all there), even the Al Jazeera children's channel.

They're building a large support network around the college campus idea. In order to prepare students for the Education City colleges, they have an Academic Bridge Program, which is basically a prep-school year. They've also set up a charter school system which teaches in English. This isn't strictly linked to Education City, but they're counting on some of those graduates to feed the schools.

Cornell and Virginia Commonwealth are currently the only two schools that have their own buildings. The CMU, A&M and Georgetown campuses are under construction. There's also a large area that appears to be reserved for more campuses, so stay tuned. Stuff's going on there.

Education City

2005-12-02T13:21:00.000-05:00

I'm off tonight to visit Education City in Qatar.

In principal, Education City may be one of the best big ideas in the world. Qatar is a small country on the Persian Gulf. They've invited several prestigious universities (currently Carnegie Mellon, Cornell, Georgetown, Texas A&M and Virginia Commonwealth) to set up satellite campuses there. The idea is to provide a first-class postsecondary education in the Arab world. This can help build a future for Arab countries when the wells run dry, and it does it in a way that builds on the Arab world's historic role as a center of learning in the world. If this works, kids in the Arab world will be able to see their future in a way that builds on the past but allows them to participate in a progressive future.

So, why haven't you heard of it? Why isn't Tom Friedman writing about it? Google for it, and you see just about no coverage in the west, other than press releases by the universities (this MSNBC article is an exception).

I'm not sure. Maybe it's too new. Maybe this vision is controversial in the Arab world and viewed with skepticism in the Western world. Maybe it's too much to believe that this big vision could come out of a country as small as Qatar. Maybe this is too big a vision to be driven by a country as small as Qatar. Maybe its some kind of scam.

I don't know, but I'm going there to find out. At the very least, we'll help some kids learn some math.

On ugly babies

2005-11-27T22:00:00.000-05:00

So, as I said, Mark Roosevelt called my baby ugly.. The baby's my daughter's school. The Pittsburgh school district hired Rand to rate all the schools in the district, as part of their plan to close some schools down (the population's been declining, so there are too many buildings).

Rand took a "value-added" approach. Basically, this means that, instead of just looking at schools' test scores, they looked at those scores, relative to what you might expect the schools to do, considering their student population. Rand looked at demographic factors and especially emphasized how much students who transferred to or from a school improved, relative to their rate of improvement at other schools.

After running the numbers, they rated schools from 1 to 5, with 5 being the best rating.

So, my daughter's school, Colfax Spanish Academy rated a 1, the lowest rating. This school is in the middle for Pittsburgh Public Schools on an absolute score level, and it has been making AYP.

That's tough to take. The school's population has been changing rapidly in the past few years, so perhaps there are factors that aren't being reflected well in the statistics. After all, only last year, Rand claimed that "The current research base is insufficient to support the use of VAM [value-added modeling] for high-stakes decisions." In fact, one of the issues mentioned in the earlier Rand report is the need to account for residual effects of, for example, previous teachers -- exactly the kind of thing that would be especially hard to estimate when a school's population is changing rapidly.

Still, I doubt that the result is greatly distorted by these factors. Perhaps Colfax should have rated a 2, but probably not a 4.

A couple of interesting aspects of this are worth commenting on:
- I spoke to the principal immediately after the ratings came out. He had exactly the right reaction. His first thought was that this might convince the district to give him the reading coach he's been arguing for.

- I'm not sure what this says about NCLB. By one measure (the one that counts - AYP), the school's doing well. By another (perhaps a more accurate measure), the school's failing. If you're going to live and die by the numbers, you have to make sure you've got the right numbers. I'm not sure we do.

- Parents seem to be taking this in stride. There's a strong desire to dismiss the rating (lots of people have said that it "must" be wrong). Although I do think we need to measure results and act on the measurements, I find myself thinking this way. I'm not sure if this is really being dismissive about the number or if its just (perhaps misplaced) confidence that we'll be able to "fix the problem."

News coverage

2005-11-11T19:55:00.000-05:00

Sometimes, you just fall into things.

I went to a meeting, discussing school closings in Pittsburgh. The guy sitting in front of me happened to be a reporter, so now I'm quoted in the "legitimate press"

Too bad the reporter talked to be before the meeting, not after. Mark Roosevelt, the new Superintendent, was very impressive. He knows how to give bad news: give it straight, don't pull punches and get it over with.

Sure, he called my baby ugly, but that's a topic for a different post.

Changing schools is hard

2005-11-01T15:12:00.000-05:00

I have a lot of respect for what Edison Schools are trying to do, but this report shows how hard it is to make a difference.

You might not think the title is particularly impressive ("Analysis Finds Gains in Edison Schools, But Model Is No Quick Fix"), but it actually gives an optimistic description of the results. Basically, the study shows that Edison can do as well as the management it replaces, if they're given time (5 years!) to overcome the drop in scores resulting from the transition. Given that, typically, schools select Edison because things are just not working, this is not exactly a ringing endorsement.

Diversity update

2005-10-03T15:44:00.000-04:00

Gary Plano reminded me of an interesting aspect of his data. If you're not an ELL student, it pays to be in a class with lots of ELL students, but only if you're in a Cognitive Tutor class.

The explanation, presumably, is that, in Cognitive Tutor classes, its more acceptable to talk with other students than in traditional classes.

The implication for policy is even more interesting. Diversity improves education, but only if the educational culture is structured to let it do so.

Education: our lowest priority

2005-09-27T10:30:00.000-04:00

If you live in Georgia, you (or your kids) are not in school today. Why? The governor decided to close schools for two days, in order to save gas.

I'm all in favor of saving gas, especially when supplies are short due to the hurricanes and other events, but isn't the real message here that missing a few days of school doesn't matter? The governor could have declared a state holiday, with all government offices closed. He could have asked businesses to close. But government offices and businesses are important. Schools aren't.

Diversity and educational effectiveness

2005-09-25T10:40:00.000-04:00

Like many parents, I send my daughter to a diverse public school, in part, because I value the opportunities she gets to interact with children from different backgrounds. I suspect most parents believe that, to the extent that a diverse school helps educationally, it is probably in "soft" skills, something along the lines of Teaching Tolerance.

But I'm now thinking about the impact of school diversity on more academic subjects like mathematics and science. It's a timely issue. As Jonathan Kozol's new book, The shame of the nation: The restoration of apartheid schooling in America points out, schools are no more racially integrated now than they were 30 years ago. And the situation is getting worse. Parents from all backgrounds dislike bussing, but its hard to see how you integrate schools when America's cities and suburbs are becoming increasingly segregated.

On the other hand, according to the NY Times [free registration required], bussing works. Or, more precisely, bussing appears to have worked in Raleigh. They've implemented a policy to integrate schools by income (not directly by race, though the result is more racial integration), and test scores are up across the district.

Both Kozol's book and the Times article about Raleigh assume the same mechanism for how school diversity results in school improvement. When the school's parents are spread across the spectrum of income and political power, then resources will be more evenly spread across the district, so no schools get "left behind."

It's a good theory. It's probably true. But redistribution of wealth isn't the kind of thing that resonates with the wealthy and politically powerful. No wonder schools are re-segregating.

But what if diverse schools really were more effective schools? What if diversifying a school didn't just help kids get along with others but it actually improved mathematics learning for all kids in the school?

Some recent data suggests that this might be the case.

Gary Plano studied the impact of Cognitive Tutor on students in a district Seattle. He found especially strong effects on English Language Learners. But a more recent analysis (submitted for publication but not out yet) shows that the benefit extended to students in high-ELL classes, even though they are not themselves English Language Learners.

I figured that these students, though not technically ELL, were lumped into those classes due to poor language skills, and that would explain the results. But Dr. Plano thinks I'm wrong. He offered the same explanation that Bill Hadley independently came up with. Students in high-ELL classes spend more time talking to each other. The ELL students are always asking the other students for help, and the non-ELL students get used to explaining the math, which helps them learn.

Its a much more subtle and interesting explanation than mine. If it's true, it's the start of a powerful argument for figuring out how to increase diversity in schools.

Catching up to Singapore

2005-09-16T10:17:00.000-04:00

Tom Friedman's one of the smartest guys out there, on a lot of topics. And he's exactly right about Singapore [NY Times - may need to register].

Singapore reminds me of a smart tech like Apple. If you haven't heard, Apple killed their most successful product because they thought they could do better.

Singapore's doing the same thing. They're tops in the TIMSS results, but they know they'll get beat, if they don't try to improve. They're not afraid to change what they're doing, if they see a better way. And they're always working on a better way.

The embarrassing US performance in these international comparisons is not because we're getting worse in math. In fact, NAEP results show small improvements over time. Its just that much of the rest of the world has improved much more rapidly.

In the US, we're still debating about whether we should go back to basics -- back to the supposed glory days of the 1950's. The problem is that we're already back in the 50's, in terms of achievement, and that isn't good enough anymore.

I don't know anything about HeyMath (though I guess I should), but I know that we at Carnegie Learning have been seeing a lot of interest from the Singapore government about what we're doing with technology and math. They're paying close attention to what's being shown to work in the What Works Clearinghouse.

The attention goes both ways. We've been imitating a lot of the diagramming techniques used in Singapore math texts in our Bridge to Algebra product. And it works.

Easing up on NCLB

2005-09-03T13:58:00.000-04:00

The Department of Education has decided to make an exception for Chicago in its rules about tutoring. Basically, the "Supplemental Education Services" part of the law requires failing schools to offer to reimburse parents for tutoring services. The schools aren't normally allowed to provide the tutoring themselves, since, presumably, they won't do any better in their tutoring role as they did in their regular teaching.

Chicago made the case that they could provide tutoring much cheaper than the private sector. They're probably also able to make the case that the private tutoring firms weren't educationally effective, either.

Although this does appear to pull some teeth out of NCLB, the SES provisions were always pretty weak. Its difficult to hold these providers to high standards of effectiveness, so we shouldn't be surprised to find private companies charging $2000 to show the kids Garfield the Movie.

I don't know what to do about this mess, but I do know that the the Hoover Institution's recommendations will only make it worse. They propose strengthening the SES law, the better to use it as punishment for failing schools. They're right that the schools have a disincentive to give their money away to private firms, but they're wrong if they think that invisible-hand magic will automatically produce better instruction in the private sector than students were getting in the public. They say:

education providers have strong incentives — and few impediments — to make these moments educationally rewarding. Unlike the regular school day, the afterschool program is voluntary, not compulsory. Education providers, to secure their revenue flow, must find ways to persuade students to attend.

Call me a skeptic (please), but l doubt that "persuading students to attend" has much to do with providing students with effective instruction. Look at what Platform Learning did in response to their Garfield scandal: they hooked up with Russell Simmons. Does that sound like they're focusing on effective education or on "persuading students to attend"?

The SES law won't work unless there is clear accountability for the tutoring firms. Throwing up $2.5 billion to anyone in the private sector who says they can teach kids isn't going to work.

Thanks to Susan Ohanian for pointing me to the Education Next article, which is really a good summary of what's going on in SES, even though their recommendations are all wrong.

Justice in Kutztown

2005-08-31T09:42:00.000-04:00

Update: It looks like the Kutztown 13 case is going to be resolved reasonably. No one's going to jail.

Free the Kutztown 13

2005-08-29T13:50:00.000-04:00

Seems a bunch of bunch of high school kids are up on charges because they “cracked” the administrative password on the laptops they were issued. In this case, the students were greatly assisted by the fact that some of the laptops had the administrative password taped to the back, and some students (the "Apple Corps") were told the passwords and all the laptops used the exact same passwords. Oh, and the password turned out to be the address of the school.

The kids shouldn't have done this, but this comes close to entrapment. Suspend them for a few days, start using a reasonable security policy and get on with your lives.

You can't give them away

2005-08-29T10:49:00.000-04:00

Who would have thought it was so hard to give away computers?

Hot on the heels of a near-riot when Henrico County decided to sell their used iBooks for only $50, the Pittsburgh Public schools are also running into trouble giving away their PCs.

In both cases, the problem is that the school districts valued the old PCs at much less than the market (Henrico at $50; Pittsburgh said they were "garbage").

Hasn't anyone heard of ebay?

Teddy’s great-grandson

2005-08-28T16:15:00.000-04:00

Speaking of new superintendents, The national media has played the story of Mark Roosevelt becoming Pittsburgh's superintendent as a story about the trend towards hiring non-traditional superintendents. But there's non-traditional and then there's non-traditional. Roosevelt isn't a businessman or a general who has shown leadership but whose career has had nothing to do with education. His whole career has been about education, though more from the policy side than the operations side. I'm hoping for the best.

Judging the book

2005-08-27T21:30:00.000-04:00

We picked a cover for Bridge to Algebra. For me, the experience was like picking a china pattern when I got married. My initial reaction was that I didn’t care, but as soon as I saw some of the possibilities, there’s a lot of "I can't live with that one."

The cover is a picture of a bridge. Surprised?

Trust the experts

2005-08-27T21:23:00.000-04:00

Linda Seebach of the Rocky Mountain News garden-pathed me with her advice to Denver's new superintendent. She starts by advising him not to trust so-called experts. Sounds like good advice. Education's full of self-proclaimed experts who promote their pet theories with little concern for evidence.

But check out her solution -- Trust a different set of experts.

This is not a fix for education's problems; it's just a volley in the turf war between education professors and mathematicians. The reason education is awash in fads and unfounded theories is precisely because education tries to solve its problems by abandoning the experts who weren't able to help for another set of experts who promise to do better. The solution is not to turn to a new group of experts. The solution is to try things out, measure the results, keep what works and get rid of the rest. Sound familiar? It's the scientific method.

Mapping the Pittsburgh schools

2005-08-27T21:17:00.000-04:00

Cool site of the week: The Visual Information Center at the University of Pittsburgh. Pittsburgh’s a shrinking city and in the process of closing schools. This group produces maps showing all sorts of information about the schools in the district – where the students live, where the free-and-reduced-lunch students live, etc. My daughter's a dot on this map, but I'm not saying which one.

Is Educational Software Dead?

2005-08-26T13:47:00.000-04:00

The NY Times (yeah, you gotta register, but its free) reports that the educational software market is in decline. They particularly focus on home software for young kids.

What’s going on? The educational software industry has, once again, promised more than they can deliver. Most of the stuff in this category is lousy. At the recent SIIA EdTech conference, Ed Coughlin told a depressing tale of two software packages, Reader Rabbit and DaisyQuest. You’ve probably heard of Reader Rabbit, but DaisyQuest has disappeared from the market. Guess which one has research demonstrating its effectiveness? And which one is still on the market.

Competitive markets work when the criteria for success in the market values in the short term what works for them in the long term. This hasn’t happened in educational software (or, more broadly, in education). In the short term, consumers are attracted by flash and (assumed) engagement. In the long term, though, they want effectiveness. But effectiveness is unrelated to flash and only related to engagement in ways that aren’t always obvious to consumers. So they buy software once for the flash and then don’t return.