The EAA Exposed: An investigative report

Product promotion

What is the response of the EAA to criticisms that it has too many inexperienced teachers and a stubborn reliance on educational software that was rife with problems?

In essence, they say, the real proof is in the results. And, within months of beginning operation, key administrators were already proclaiming that, according to internal testing, their new approach was achieving remarkable results.

The stunning scores coming out of the EAA were eagerly touted by Agilix and the School Improvement Network.

"During the 2012-2013 school term, 59% of students achieved 1.5 or more year's growth in Reading and 58% of student achieved 1.5 or more year's growth in Math," boasts a "case study" posted on the Agilix website.

Those impressive gains, said the people running the EAA, were achieved by combining a change in philosophy with tailor-made technology. EAA students have longer school days and a longer school year than their DPS counterparts. They also had schools that embraced Student Centered Learning, described as an approach that, using the definition employed by the EAA, met students where they were academically rather than forcing them to march lock-step through the traditional, age-based grade system.

And then there was the technology provided by Agilix and SINET. Instead of a teacher with a marker standing in front of a whiteboard and students with textbooks dutifully taking notes, the computer would provide the bulk of their educational experience. Teachers would serve primarily as an adjunct, helpers who facilitated and provided a watchful eye. Teachers too had software to assist them in their professional development, a SINET product called PD360 — with PD standing for "professional development."

Before long, those two companies partnered, with taxpayers kicking in $100,000 to pay for Buzz and PD 360 to be merged, allowing it to be marketed as a single product for customers interested in a package deal. (Contrary to what EAA documents say, a SINET executive told us that Buzz and PD360 had been merged in Kansas City.)

Moreover, key EAA officials devoted much time and energy to promoting both the EAA model and the technology they were using to achieve truly remarkable success.

A symbiotic relation emerged. The companies needed the EAA's students to do well in order to prove the effectiveness of their products when making sales pitches to other schools and districts; the EAA needed the companies to do well, so that more money could be funneled back into the product improvement that would, conceivably, promote the kind of student advancement that would attract even more kids, and the state funding attached to them, to the EAA classrooms.

None of this is idle speculation or hyperbole from critics.

Testing, testing

The emails provided us clearly show company officials time and again asked top EAA administrators — especially Covington and Esselman — to assist in efforts that would help promote their products, from making speaking appearances at conferences to coming up with data on a moment's notice to help provide fodder to an influential blogger. Promoting the work being done by the EAA, and promoting the products being used, were all one and the same.

As for the promises that successful sales would return to the district, there is this telling email from a Agilix official to Mary Esselman:

"Sorry I've been so out of touch the past several months — though don't think you and the EAA aren't top priority — it's nearly all I discuss, promote and focus on. I've just had to make sure SINET is owning the project, getting familiar and able to support and drive it so I can focus on driving sales that drive more money to perfecting your model."

In other words, achieving stellar test results was vital to the success of the EAA model. And the scores really shined, reflecting remarkable success.

At least the ones obtained through something called the "Scantron Performance Series," tests internally administered by the EAA.

There is, however, a shadow that's been cast over all those sparkling Scantron scores.

Numerous teachers interviewed by the ACLU told us that, because of intense emphasis on producing positive test results, students were allowed to re-take tests when they failed to perform well. It was described as standard procedure throughout most, if not all, EAA schools.

Asked by the ACLU if she ever became aware of these types of improper testing procedures, Esselman responded: "Yes. We were made aware at a public meeting and immediately made the necessary steps with our school leaders to address this issue."

Adding more darkness to those shadows is this fact:

In stark contrast to the internal test results are the state's standardized achievement tests, known as MEAP. The most recent MEAP results show that a high majority of EAA students are either stagnating in terms of reaching math and reading proficiency, or falling even further behind.

Asked about those contradictory results, Esselman responded: "The EAA has adopted a student-centered model of teaching and learning that allows teachers to personalize instruction to meet the needs of each child. Our teachers are focused on individual student progress rather than teaching to the state test."

The irony in that is that persistently low MEAP test scores were the reason schools were taken out of the Detroit Public School System and put under the control of the EAA.

Even so, for Cory Linton, an executive vice president at School Improvement Network, the proof of the EAA's success and the success of the software that is a focal point of its system, is in those Scantron scores.

"These kids are getting the best educations of their lives," he said in a phone interview.

When asked about the many bugs in Buzz when it arrived in Detroit, he said those kinds of problems are typical when software gets rolled out. It is part of the process. He also contended that the emails we've examined only reflect postings to a help desk set up specifically to deal with problems quickly and efficiently.

It's not much different than people having problems with an iPhone, he explained. There are bound to be some problems when using any technology, no matter how well it is designed. And that it's not fair to focus only on the problems being emailed to a help center, because it distorts the picture, providing an inaccurate portrayal of what was going on in the big picture, or how well the software works and how effective it is overall.

One thing that he's wrong about, though, is the scope of the documents used to compile this report.

Big picture, big problems

Rather than just pleas for help in dealing with software snafus, many of the emails we examined dealt with bigger-picture issues involving Esselman and the two companies.

One particularly damning email actually came from a SINET employee.

In November 2013 — at which point Buzz had been the keystone of education in the EAA for 14 months — the School Improvement Network's Brian Chandler sent a message to the EAA's Mary Esselman saying this about the work Agilix was doing, and the performance of Buzz, which by that point had been upgraded twice since first being introduced in Kansas City:

"We have reiterated to them [Agilix] again and again they have done a terrible job meeting schedules and expectations. ... In the end, they have finally admitted the current version of Buzz is little more than a hack that really can't be extended much further without a significant rewrite."

"It is hard to defend language like that," Linton admitted.

Esselman's reaction at the time, recorded in an email, was to say she found Agilix's self-evaluation of Buzz to be "hugely concerning," and that her first call about the situation would be to the "governor."

It's not known if that call was ever made, or what the governor's reaction might have been.

But that same email had Esselman being told that the big fear, at least in the eyes of this SINET employee, was that sales efforts in Kentucky would bear fruit too soon, and they would be forced to deliver the "hack" model currently being used in Detroit rather than the upgrade expected to be installed in Detroit in about four months' time.

Another four months of some of Michigan's most vulnerable students being forced to use a product that, as the SINET employee admitted to Esselman, was clearly not "satisfactory to your needs at the EAA."

In a long exchange of emails with the ACLU, SINET Vice President Cory Linton wrote:

"SINET never believed it was a hack. Unfortunately, the email does exist — but that was not an official SINET position. To me, the most important question is what is the impact on the students? Are they having a better learning experience or not? I realize this is very debatable and the data can be interpreted many ways — this is why it is so important to go visit the schools and talk to the students directly, so you can make your own judgment about whether the schools are being successful or not. I can tell you unequivocally — as an official spokesperson for the company — that SINET's motive from the beginning was to help these disadvantaged students have a positive learning experience, and be prepared for a successful career or college after leaving these schools."

As for talking with teachers and students, we have done that.

Inside view

During the course of our investigation, the ACLU interviewed a dozen current and former teachers, one former administrator, and several students. Many of the teachers asked to remain anonymous, saying that, because they lacked the protections afforded by a labor union, they feared retaliation by the EAA if they spoke on the record. Others feared that the EAA would blackball them. Others, however, did agree to go on the record.

One of them is Jordan Smellie, a tech-savvy former music teacher now working in the IT industry.

"Buzz overall I would describe as a travesty. To say it was incomplete when it arrived is giving it too much credit," he said. "The software was in a state that any other firm would have never released. The design was poor, front to back, top to bottom. The user experience was horrendous. It was incredibly slow, if it worked at all."

Nearly everyone interviewed talked about the dearth of content when Buzz first arrived in the fall of 2012, which is contrary to Mary Esselman's unequivocal written assertion that Buzz arrived on time and fully formed.

Two of the teachers we interviewed — both of whom no longer work for the EAA — provided signed affidavits detailing accounts of their EAA experiences. One of them is Delbert Glaze, who formerly taught at Nolan Elementary School.

He recounted for us how EAA administrators had thrown into a Dumpster textbooks left behind by DPS as it vacated the building to make room for the EAA. So he had to use Buzz. But by January of 2013, he noted, "once the Buzz platform had advanced through all of the Web pages for a particular unit, Buzz instructed the students to complete a multiple choice test. When the test was available, students were allowed to re-take the test as often as necessary and thus ultimately figure out the right letter to mark and move toward the next lesson. However, at various points throughout the year, the tests would disappear from the Buzz platform. As a result, students were unable to progress in their curriculum and, in some instances, were forced to completely restart the lesson and thus fall several weeks or months behind."

Instances of test scores and other material disappearing are well-documented in the emails obtained through the Freedom of Information Act documents that serve as the foundation of this report.

Brooke Harris, who formerly taught at Mumford High School, signed a statement that mirrors many of the statements made by Glaze.

Among other things, she noted that during the first year Buzz was being used, "The Buzz platform did not have content for certain elective classes for the entire school year."

Despite that, she stated, administrators either applied heavy pressure or offered incentives for teachers to abandon traditional teaching methods and embrace Buzz and its companion technologies.

"I was told that in the student-centered model, my role as a teacher was primarily to supervise students to make sure they were using Buzz."

And then there are the first-person accounts of life in an EAA school provided to us by the three children of Detroit resident Christina Lee. Sitting on a couch in the living room of their rented home on Detroit's west side, they explained why their mom pulled them out of Nolan, a school for children in kindergarten through eighth grade, after attending for two years.

"Nobody did what they were supposed to do," said Destiny, 15. "Kids were getting on Facebook, Instagram, porn sites."

Porn sites?

"Yeah, you know, like the ones where you do video chat."

("Buzz is a digital platform that functions inside of the firewall that is provided through the Detroit Public Schools IT network. The firewall is in place to filter inappropriate content. This firewall was in place at the time of launch and continues to be place today," Mary Esselman responded in writing when asked if children using Buzz were able to access pornography sites.)

As with teachers who left the EAA, she might be classified as "disgruntled." She was caught using a phone to videorecord girls staging a fight in a bathroom. She was, according to her mother, suspended for 180 days.

"I know I did wrong," Destiny said. "But 180 days? I begged them to let me come back to school, but they wouldn't let me."

Deon, 14, said, "I didn't get to learning anything. If there's a teacher to help me, I can catch on. But being on a computer all the time doesn't help me. I was doing basic math, over and over. Simple stuff, like what is 3 + 3. Just sit on the computer all day."

The youngest, Aynnya, now 7, recounted the challenge of trying to get help working on a computer. Part of that stems from the fact that her family has no computer at home.

"I'd tell my teacher, 'I don't get this,' and she'd say, 'You have to figure it out for yourself,' and I'd say, 'But I don't get it,'" she recounted, her eyes catching fire from the anger that experience still kindles.

Inquiring minds

In an email responding to questions about bugs in Buzz, Cory Linton explained that there are bound to be problems, and that is the nature of the software business.

"Like all software, as you have more usage, you continually refine and make it better. This is software industry standard practice," he wrote in an email responding to questions from the ACLU. "A best-selling business book, The Lean Startup, explains this process in detail. But the basic idea is that you build software, release it to end users, get feedback, make adjustments, and release again. It is an iterative process, which is why you see so many 'Updates' on applications on your smartphone.

"You can only figure out so much in a lab, and then you learn more when real users use the software. This doesn't mean the users are guinea pigs. I spent 9 years at Microsoft, and many years doing this exact process in the Microsoft Word development team. With every version, we would immediately visit clients and see what they wanted changed/improved, and then incorporate that feedback into the next version — which usually took 18-24 months to release. That process has been followed closely with 'Buzz' — with a new version coming out every 18 months or so."

Having obtained the documents much of this report is based on, Pedroni is well aware of what they contain, and doesn't easily swallow claims that what went on at the EAA should be considered the unavoidable complications that come with creating and deploying software.

"Wouldn't the 'end users' have deleted the Buzz app off their smartphone a long time ago and gone with something else?" he asked. "If the app was a 'hack' for 18 months at least, according to its developers, I would have deleted that app from my smartphone after realizing it didn't do what was being promised.

"The last thing I'd do is use it as the almost sole basis of teaching children I loved, forcing them to endure it for hours and hours a day, week after week, month after month, sacrificing them so they could make it 'better' for some for-profit, private venture. This product was supposed to already be known for lifting schools from the bottom 5 percent. Instead, the system Gov. Snyder single-handedly created brought in a product that was already in its second iteration but still a hack. My smartphone doesn't have space for that kind of poor quality and I'd never force my children to be guinea pigs to assist it in becoming more than a hack."

Linton, recognizing both the controversy and the significance of what the EAA represents, offered an idea.

"I would say, 'Let's see what the impact on the students is.' Is anything more important that that?

"What is their attitude about learning, better or worse? Are there more or less discipline incidents? Do their parents notice a difference in their children? And, most importantly, are the students learning more or less? There is much debate over the data ... we should be focusing on accurately measuring the learning so that we can see what the impact on these students is. I would say we should be pushing to have a neutral, third-party come in and look comprehensively at the impact of the EAA's new educational approaches and systems, and look at all aspects of education and the impact on the student. Such a comprehensive analysis would focus on the big picture so that we all can learn what worked in the EAA, what needs to be tweaked, and so forth — looking at such measures as student learning, student and parent attitudes, culture and climate of the school, student attitudes about their future careers and attending college, and so forth."

We agree that the parents of EAA students and the public at large need to know if the claims of astounding gains being made are actually true, and what the effects of this new learning model really are.

We also agree that more transparency is necessary.

And with Covington now gone, the EAA wants to put the focus on the future. “Everything is under review,” spokesman Mario Morrow said. “It is a new day for the EAA.”

As things move forward, however, one question needs to be answered: How do the leaders involved in creating the EAA, the board charged with overseeing it, and the educators responsible for running it justify allowing a terribly flawed program to be used for two years on 10,000 children in schools that were already deemed to be failing?

Curt Guyette is an investigative reporter for the ACLU of Michigan. His work, which focuses on emergency management law and open government in Michigan, is funded by a grant from the Ford Foundation. You can find more of his reporting at aclumich.org/democracywatch. Guyette can be contacted at 313-578-6834 or cguyette@aclumich.org.

To read some of the emails, continue to next page.