In Focus

Measuring Arts Achievement: New Goals, New Tools

Dr. Susanne Harnett has overseen many studies of arts-education programs—both federally and privately funded—at Metis Associates. Here, she answers questions about how arts performance can be measured, and whether arts learning has a broader impact on academic achievement.

Q. The issue of measuring student progress is taking center stage in education today, and with it has come a growing interest in assessing achievement in the arts. How is evaluating arts growth different from measuring academic achievement?

A. Measuring arts performance can be challenging, because there are concerns about whether an evaluator can be objective. Yet, if you think about it, teachers have always assessed student writing and essay tests, which is much more difficult than grading multiple-choice questions. In all areas where subjectivity is at play, inter-rater reliability (ensuring that various “judges” assess performance in the same way) is crucial. Assessments in the arts often examine students’ background knowledge as well as their performance skills.

It’s widely thought by researchers that arts education not only strengthens students’ arts skills specifically, but helps students acquire a set of mental habits that can be transferred to academic subjects such as English language arts (ELA) and math. A number of the projects Metis is evaluating aim to impact on these so-called “pro-cognitive skills,” so we have developed a range of tools to measure those related to “habits of mind” as well as actual arts learning.

Q. Let’s talk about measuring actual arts learning. How do you go about evaluating art?

A. Our work with Arts Achieve offers some examples of how it can be done. Arts Achieve is a large project funded by two U.S. Department of Education grants—one from the Arts in Education Model Development and Dissemination program (AEMDD) and one from the Investing in Innovation (i3) Fund. It brings together the New York City Department of Education (NYCDOE) Office of Arts and Special Projects with five premier cultural organizations whose consultants and coaches partner with arts teachers in more than 40 elementary, middle, and high schools. Metis is overseeing a rigorous evaluation of this program, which involves assessments in visual art, music, dance, and theater. The assessments align with grade-level benchmarks for learning that were established by the NYCDOE and published in the Blueprints for Teaching and Learning in the Arts. In addition to direct arts learning, we’re examining whether there is a transfer of skills to students’ academic performance.

During the first school year of this five-year project (2010–11), teams from the NYCDOE and arts organizations collaborated to develop the assessments and the rubrics that adjudicators use to assess students’ performance on a variety of tasks. Metis is contributing to this process by assisting with the establishment of inter-rater reliability. To ensure agreement in the ratings, adjudicators participate in trainings where they watch videotapes of student performances and then rate them together using the rubrics. We help them to talk about what to look for in the work and discuss any disagreements. At times, the rubrics are changed so that everyone can agree on how to use them. These methods have attracted the interest of administrators at the New York State Education Department who are interested in issues of inter-rater reliability in performance assessments in all academic areas and of arts educators who are eager for valid and reliable tools that can be used in future evaluations of arts-education programs.

Q. The issue of assessing the transfer of pro-cognitive, arts-related skills to academic subjects is controversial. Why?

A. Some arts professionals believe that learning in the arts should be a goal in itself—that we should not have to prove any carryover benefit beyond these disciplines themselves. But, in the past few decades, there has been growing recognition that development of certain mental habits—which are often taught through arts subjects—is crucial for success in higher education and careers. Although the exact list of habits varies, the common denominator is that students need to think independently, to critique and value evidence, to collaborate, and to be creative—all skills that are reflected in the Common Core State Standards. Metis is looking at these skills in the four evaluations it is conducting for AEMDD-funded programs and in other studies to understand whether the “soft” or “pro-cognitive” skills conveyed in arts instruction have an impact on academic learning, and if so, how?

Q. How do you go about assessing this transfer from the arts to academic achievement?

Classroom Inc.

Metis used a rigorous experimental design to evaluate Framing Student Success: Connecting Rigorous Visual Arts, Math, and Literacy Learning. This New York City schools program integrated high-quality visual arts education into math and ELA curricula.

A. One example can be seen in Metis’s evaluation of Framing Student Success: Connecting Rigorous Visual Arts, Math, and Literacy Learning, which linked New York City public schools with local artists to help schools integrate high-quality visual arts education into math and ELA curricula. The program was provided to students in grades 3 to 5 in three high-poverty, low-performing public schools. Metis collaborated with Studio in a School to develop a Studio Habits of Mind rubric based on the work of Project Zero, an educational research group at the Harvard Graduate School of Education that promotes arts as a central part of education. The group has broken down the kinds of thinking that arts teachers can promote. Metis’s assessment of Framing Student Success used a five-point rubric to measure progress in five of these habits, including students’ ability to engage and persist in a project for a sustained period of time; envision images that will help guide their work; observe closely their own work and that of others; reflect on their process, intentions, and decisions; and stretch and explore— trying new things, taking risks, and learning from their mistakes. Twice a year, teachers in this program are using this rubric to assess their students along these indicators.

Metis also collaborated with the Eastern Suffolk Boards of Cooperative Educational Services (BOCES) to develop a similar checklist for the Creative Classroom Collaboratives (C3) project, a four-year arts integration program for students in second through fourth grade that is funded through an AEMDD grant. This program involves teacher collaboration with outside artists and training in “aesthetic education” and “21st century learning models.” The students participate in a number of activities, including viewing dance, music, storytelling, and theater performances and engaging in performing arts activities linked to ELA and 21st century skills. The checklist that Metis developed enables teachers to assess their students’ 21st century skills, such as creativity and innovation, collaboration, critical thinking and problem solving, and communication. These are the types of skills that have been theorized to transfer to academic areas. The first participants are currently in third grade, and Metis plans to examine their state test scores this year in relation to the scores of students in similar schools. Metis also will examine whether there is a link between improvements in pro-cognitive skills and academic progress.

Q. What have been the chief findings on the transfer effect?

A. Again, the Framing Student Success evaluation provides some examples. Metis used a rigorous experimental design to test the effectiveness of the project for participating students, classroom teachers, school-based visual arts teachers, and school administrators. Standardized test scores were used to compare growth in math and ELA in these schools and three control schools.

The most pronounced finding was an increase in math scores among participating students. In terms of the Studio Habits of Mind, fifth graders demonstrated the most growth in “reflecting.” It is notable that they also had significantly higher scores than students in the control group for reflection in their written responses to artwork on the benchmark arts assessments. Reflection came through in multiple areas as having been instilled through the project. This is the meaning of “transfer” of a habit of mind to an academic area.

Some direct assessments of whether arts instruction has a bearing on ELA and math have not been conclusive. But this type of assessment might miss the point. Even though transfer may not be evident in standardized test scores, it may be the case that these tests are not measuring critical thinking and other higher-order thinking skills that the arts are helping to promote. Furthermore, the effects of arts education could be seen in measures of confidence and self-esteem, which are also important to future success. Our evaluation of Framing Student Success produced a number of individual success stories, in which students who had not been successful in other academic areas found opportunities to be successful and build their confidence through the program.

Q. How do these types of assessment figure into a larger movement away from testing rote knowledge and toward performance-based measures?

A. Since 2010, 45 states and Washington, D.C., have adopted a set of Common Core State Standards for ELA and math achievement. In conjunction with these standards, a consortium of 28 states called the Partnership for Assessment of Readiness for College and Careers (PARCC) developed a new set of assessments to measure performance—including skills like critical thinking—instead of rote learning. The PARCC assessments will be ready to administer in the 2014–15 school year, and New York is one of the states that have committed to using them.

Q. What do you think are the most important next steps for the field?

A. Metis is working on a number of arts-education projects, and we’ve developed some really good tools that address key questions. We will definitely be applying these tools to future undertakings. One of the central questions is establishing inter-rater reliability. Multiple-choice tests are easy because there are right and wrong answers. However if you’re assessing a piece of art or writing, it becomes very important to build solid rubrics and to establish inter-rater reliability.

We also need to clarify how arts skills are connected to pro-cognitive or soft skills and measure them reliably. Overall, we need more data to document the importance of arts education in children’s development. Should we concern ourselves with carryover of skills to the academic context, or should we be more concerned with showing the importance of arts education in its own right? And, does arts education support college and career readiness? Arts education is more than making art; it is about being a good audience member, building confidence, exploring creativity, and trying new things—in short, being a well-rounded person.