Expertise | Evaluation
Carnegie Hall-Weill Music Institute: Evaluation of Arts-in-Education Programs
From enhancing students' music awareness to improving entire school climates, Ensemble ACJW—a program of Carnegie Hall, the Juilliard School, and the Weill Music Institute in partnership with the New York City Department of Education—is an ambitious arts-in education-program that is making a demonstrable impact on students' lives.
Ensemble ACJW is a group of young, highly skilled professional musicians who are accepted into a two-year program to build their performance careers, expand their teaching abilities, and engage with the communities around them. The centerpiece of the program is a fellowship in which 20 musicians are each assigned to one New York City public elementary, middle, or high school, where they work with music teachers and students to raise the level of instrumental instruction.
Each musician-school partnership is unique, but over the two years, the artists typically coach teachers, lead small or large student learning groups, and perform for the classes and schools. The fellows also lead community projects in the neighborhoods where the schools are located.
Beginning in the 2011-12 school year, Metis conducted surveys and interviews with participating students, teachers, and fellows to examine the impact of this program on both students and the school communities to which they belong. In the second year of the evaluation, Metis began comparing the 20 selected “treatment” schools with similarly situated comparison schools. Over the time that Metis has been working with the program, Ensemble ACJW has touched more than 2,200 school children. Metis is exploring changes in student engagement in music as well as in their academic achievement, self-confidence, motivation, and behavior.
Metis is also undertaking an evaluation of Link Up, a highly participatory national and international program for students in grades 3 through 5 under the auspices of Carnegie Hall's Weill Music Institute (WMI). WMI provides teachers with curriculum and supplementary teaching materials, including teacher guides, student materials, concert scripts, and concert visuals. Using these materials, students learn to sing and play an instrument in their classrooms. At the end of the program, they have the opportunity to "link up" with a professional orchestra—Carnegie Hall in the case of the New York City program—to play the pieces that they learned during the year.
In 2012–13, WMI hired Metis to begin an evaluation of the program. In the first year, Metis worked with WMI and a group of pilot teachers from five schools located in Manhattan, the Bronx, Brooklyn, and Queens to develop a music-assessment tool that could be used to evaluate the program's impact on students' music skills. The tool helps teachers assess students' performance skills, music literacy, knowledge of orchestra instruments, and ability to develop simple music compositions. The aim, ultimately, is to have tools that teachers may use formatively and summatively to assess students' progress. These tools may be packaged together with the curriculum so that data from the program can be collected and monitored from the inception, by individual schools, and by the program as a whole.
Metis also worked closely with program staff to develop a number of other evaluation tools in 2012–13, including a classroom observation form, surveys for teachers, and focus group protocols for students. These tools were all piloted over the course of the school year, and preliminary data were collected and analyzed.
In the 2013–14 school year, Metis is continuing to work with the program to further refine and expand the music-assessment instrument and its use. A new group of pilot teachers from schools around the country is participating in this refinement process, and will then use it with their students. Metis will collect, analyze, and report back to WMI additional data through classroom observations, interviews with national orchestra directors, and focus groups with students from these sites.