In the 2016-17 school year, Citizen Schools completed the second year of our initiative to pilot and evaluate digital courseware across our national network to learn more about how best to integrate technology into Expanded Learning Time (ELT) for middle school students. Under the leadership of Amy Hoffmaster, Citizen Schools’ Director of Program Innovation, in year two of the pilot our primary tasks and learning goals included:

• Continued creation of a network within our existing network of expanded day sites to test and evaluate digital courseware deployed as part of their expanded day academic support strategies (in ELA and/or Math) focused on quality of implementation and understanding best practices and stubborn challenges to effective implementation.

• Offering resources and coaching at the pilot sites to facilitate planning and execution of the product testing. Year two of the pilot will include more targeted training for Citizen Schools’ front-line program staff, and an emphasis on applying data gleaned from using from the digital courseware to drive academic support strategy during the expanded day staffed by Citizen Schools and, at some school sites, also linked closely with Math or ELA courses during the school day.

• Participation in the Learning Assembly, a national network of organizations piloting and evaluating ed-tech innovations with schools across the country.

National Reach

As detailed in the list below, a total of 11 Citizen Schools ELT schools serving middle grade students participated in year two of the pilot, reaching over 2,000 students – more than double our initial target of 1,000 students. Five of the 11 schools previously participated in year one of the pilot, generating valuable knowledge about implementation quality and efficacy beyond a launch year. And for the first time in any of Citizen Schools’ regions nationally, all five of the Citizen Schools ELT sites in the New York region participated in the pilot, allowing for additional capacity and focus on blended learning implementation as a regional priority.

School Implementation Snapshots 

Renaissance School of the Arts (NYC of Education, New York)

At Renaissance School of the Arts in New York, Curriculum Associates’ i-Ready is provided by the school and Citizen Schools uses it for its academic programming twice each week. Citizen Schools staff members differentiated lessons for each student, based on the diagnostic data reports that i-Ready populates. The Citizen Schools campus director and principal met regularly to look at i-Ready usage rates. Teachers and Citizen Schools Teaching Fellows came together to take part in professional development focused on two high priority focus areas: social emotional learning and blended learning, specifically the i-Ready program. In addition to having shared professional development, first shift teachers and Citizen Schools Teaching Fellows at Renaissance School also have joint planning time every week, organized by content area.

Carter G. Woodson Elementary School (Chicago Public Schools, Illinois)

At Woodson, a K-8 school, Think Through Math, is now a required part of the curriculum for all students in the school, after its use during Citizen Schools ELT hours showed a relationship between Think Through Math Scores and scores on Chicago Public Schools’ NWEA math assessment. That early information helped to build buy-in from first-shift teachers and the principal. Citizen Schools found that students who scored 80% or above on post-tests on Think Through Math also scored in significantly higher percentiles on the district’s summative NWEA math assessment. Think Through Math was then used both during the ELT hours and during the traditional day, with students earning a report card grade for it.

Joseph A. Browne School (Chelsea Public Schools, Massachusetts)

Citizen Schools uses the MIND Research Institute’s ST Math, a program developed to boost math skills through visually animated representations of mathematics concepts, which makes it ideal for students who have limited English proficiency. The program aligns with the school’s goal of enhancing students’ computer literacy; because ST Math has similarly formatted questions to what students will see in their state-administered assessments, they will be more comfortable with the format when they are tested. At the Browne, the Citizen Schools campus director and team implemented a connected set of innovative approaches to support student growth in Math, with blended learning at the core, including:

• Realigned program priorities to dedicate an additional 7 hours per week to math instruction and practice

• Utilized a blended learning model with ST Math, combining computer-based learning with targeted small-group instruction, to increase students’ individual time on task with math content

• Utilized a pre-written math curriculum to meet students where they are and to anticipate and target key areas of misunderstanding

• Coordinated a complex system of differentiated instruction which allows students to work on skills that are highest-leverage for them in groups who move at a similar pace

• Utilized a stringent data-to-action cycle, in which students were routinely assessed and their instruction or support adjusted based on their performance

• Engaged her team in a comprehensive conferencing and Early Warning Indicator program which allowed students to build close mentoring relationships with adults

• Co-planned a separate math curriculum to challenge students who were already performing at grade level so that they continued to develop their flexible problem-solving skills

Math interim assessments place each student in a diagnostic level that corresponds to the Math content knowledge and problem-solving skills expected for that student’s grade level (i.e. “On Level, 1 Level Below, 2 Levels Below,” etc.). At the end of the school year, over one-third of students made at least 50% more growth than is typical for one school year – a substantial improvement over previous years.

KEY Findings linked to the pilot’s learning goals

The implementation learning goals emphasized strengthening academic support programming for students, improving instructor expertise and experience using digital courseware, and improving data collection and utilization of that data. Communication and collaboration with school partners and digital courseware vendors were critical in each of these areas of emphasis.

After analysis of the detailed implementation data – including usage reports, site visit data, and teacher and student surveys – and student achievement data, one major finding is the importance of consistent usage of the primary digital courseware as – not surprisingly – efficacy is reduced if students are not using the product regularly as they are unlikely to cover the necessary content aligned with their grade level standards and learning goals. This finding is consistent with a recent national evaluation of ST Math (note: this evaluation did not include Citizen Schools sites) that found that for schools that implemented ST Math consistently, the results were significant. The evaluation found that student impact results were most significant at the schools “that used ST Math above minimum thresholds (where more than 85% of students used the program and on average completed 50% of their grade level content during the year).” Citizen Schools is committed to tracking usage rates overall and relative to minimum thresholds going forward, building on the processes we implemented in year two of the pilot to begin doing that.

The other major findings linked to our learning goals are:

• Strategies for strengthening digital courseware vendor relationships include sustained collaboration, including trainings and feedback loops, and beta testing products to provide vendors with targeted product feedback. For example, Citizen Schools participated in a beta test of MIND Research Institute’s ST Math to test a version of the program differentiated for middle grade students from elementary grade students.

• Implementation of station-rotation model: Preliminary evidence of academic gains through combining guided reading with blended learning.

• As the school snapshots above highlight, stronger implementation on campuses where extended day blended learning programming is tied to students’ classroom activities during the school day.

• Initial innovation in applying project-based learning to extended day academic work through Newsela consistent with Citizen Schools’ commitment to real world project-based learning in its signature apprenticeships.

• Some early successes in professional development but recognition of barriers to scheduling and prioritization if blended learning is not a school-wide priority.

• Consistently learned that it can be challenging for Citizen Schools leadership and campus staff to set meaningful, motivating and realistic campus-wide academic and usage goals within their platforms due primarily to a lack of insight and support on what reasonable yet ambitious progress or mastery might be in an implementation with sufficient student dosage/content coverage. As a result, Citizen Schools national and regional leadership are prioritizing providing more scaffolding to Citizen Schools campus leaders when creating goals within their blended learning product(s) and beginning training and orientation from Citizen Schools and vendor training teams earlier in the summer to provide more time for staff to build familiarity with the digital courseware in the context of their respective classrooms, student learning needs, and student growth goals.

Looking Forward

Consistent with the overall strategy at Citizen Schools to provide our regions with more autonomy to innovate their ELT partnerships and program models locally, currently in the 2017-18 school year, regions have taken an increased leadership role in the vision, oversight and support of their respective digital courseware initiatives. In the New York region in the 2017-18 school year, not only are all of the Citizen Schools ELT sites utilizing digital courseware, but all of the sites are using the same courseware (i-Ready) and focused on the same content area (Math), providing even deeper regional alignment, support, peer learning, and impact.

Also, Citizen Schools nominated five outstanding school and district leaders from its national network – including from some of the schools and districts highlighted here – to attend the recent ASU+GSV Summit joining other innovative district leaders and principals to participate alongside 3,500 entrepreneurs, business leaders, investors, philanthropists, university leaders, policymakers, and nonprofit leaders in meaningful interactions with leaders across the education ecosystem.

Citizen Schools looks forward to sharing future updates on how digital courseware implementation provides students with the personalized learning opportunities they need and deserve to be engaged and successful in middle school and beyond.