A recent research study released by the Institute of Education Sciences found that the use of selected reading and math software (for an experimental group of teachers) did not improve student test scores when compared to students of classrooms that did not use those software (the control group of teachers). Read about the study here: http://ies.ed.gov/ncee/pubs/20074005/execsumm.asp

My thoughts after the jump.

The study is interesting because it used an experimental design that randomly assigned classrooms to an experimental group (those that used the software) and a control group (normal classrooms that did not use the software). An experimental study is the best way to isolate the effects of one variable… in this case the use of reading and math software… Then by comparing the test scores of the groups, the study presents a compelling finding that the use of reading and math software does not improve student achievement in a significant way.

This finding is similar to other studies that show little or no effect of computers on student achievement… I use the term student “achievement” and not “learning” here because what we are measuring is student performance on standardized tests; while how we define and measure if a student has learned something differs widely across education, and we still argue bitterly about what “learning” really is. Anyway, studies continually show that computers do not improve student achievement.

How do we as technology advocates respond to such discouraging results of technology? I’ll present my thoughts on a few options:

1. Accept the results and denounce any benefits of computers on the learning process -  I think this response would be pretty simplistic and naive, as practitioners and ed-tech advocates clearly can see that technology makes a difference in our kids’ lives, and can also do so for the schooling process.
2. Argue for other benefits of technology – This response includes such arguments as 1) “Our kids need to develop 21st Century Skills” in order to 2)”Compete in the new world economy and society or something like 3)”Our kids are digital natives and need computers in order to make school and learning relevant to them” etc. etc. etc.  While these arguments may have some validity (and I question that sometimes), they stand on their own and are not directly related to student achievement… That is, if you believe that we need to have computers in the classroom because kids need to learn how to use a laptop in preparation for when they get a job 18 years later… then that is an argument that stands on its own and is not the same as believing that technology improves the learning of students.  Those assertions will be critiqued on separate grounds such as “will students be using laptops 18 years from now?” or is “spending massive amounts of money on technology, versus other things that need resources that may be more effective, a good thing to do”?
3. Realize the limitations of the study and find more questions to research that build on these findings – I haven’t read the study in detail, only the executive summary.  However, I have two initial thoughts on the study.  First, the teachers in the experimental group used the software very little… the researchers estimate that the software classrooms used technology about 10% (give or take) of the total instructional time.  Given the small amount of use, it’s not surprising to find that the software did not make a significant difference.  My question is, would using reading and math software for significant amounts of time, where they are the central element of the pedagogy, make a difference?  This is a question of pedagogy, and time spent using technology….  Second, the design of educational software studied and other technologies in general is often not based on anything we know about learning… actually we still know very little, definitively, about effective learning… so it’s not surprising that the current reading and math software out there may not make a difference.  Are they designed based on ineffective theories of learning?  This is  question of what we know about learning, and instructional design….

I’m working on a paper for class that tries to answer some of these questions and am coming to a few conclusions:

1. Educational software, and computing, on its own does not make a difference in student learning.  This finding is clear.
2. Cultural, economic, and social justifications for having computers in schools are separate arguments; they are different statements than saying that computers improve students’ learning.
3. Computing may not make a difference for a few reasons.  First, we don’t use computers and applications nearly enough in classrooms to expect that they make much of a difference – a question of pedagogy and time spent with technology.  Second, we don’t know that popular software and applications are designed using sound and effective theories of learning.  In other words, the products we have to use may just suck in general.
4. This leads to a chicken-or-egg question for what we do next… Do we try to increase computer usage amongst teachers? We may be doing harm by having people use technologies that don’t make a difference.  Or, do we design computer applications that are based on sound learning principles and are proven to be effective?
5. I say we focus on understanding the effective ways to learn, and designing applications that facilitate that… then figure out how to scale up that research to real classroom contexts.  In an ideal world we would design, test, and develop effective software in a small experimental setting…. THEN, we would do a study like the one just released, where we test the effectiveness of that software in larger settings across classrooms.