Investigating what AI tools actually do with what your students put into them

In 2024, the Los Angeles Unified School District rolled out “Ed,” an AI-powered assistant for students. Months later, the company behind it went into financial trouble and the platform shut down abruptly. A whistleblower engineer at the company had already warned district and state officials about data privacy problems before the collapse, and had been ignored. When it was over, parents and advocates were left with no clear accounting of how student data had been handled (Primary reporting: EdSurge, July 2024 and The 74, July 2024)

This wasn’t a fringe platform. It was a district-endorsed, district-deployed tool used by students in one of the largest school systems in the country. And when it fell apart, no one could say with confidence what had happened to the data collected from hundreds of thousands of children.

That story isn't an edge case. It's the current state of AI in education. In December 2024, the PowerSchool breach compromised the records of approximately 62 million students and 9.5 million educators — names, addresses, Social Security numbers, grades, medical alerts, and attendance data. It is the largest breach of children's data in U.S. history. The attacker gained access through a single compromised employee password. Third-party vendors were at the center of both incidents. For more information, you can check out: PowerSchool's official disclosure, Security.org analysis, or Proskauer legal summary.

In addition to these scenarios is “Shadow AI” — unapproved apps teachers and students use independently. These apps remain nearly invisible to school IT teams while potentially storing student data indefinitely or using it to train commercial models (see, eSchoolNews, July 2025).

58%	of educators have received no AI training. Most teachers making AI decisions in their classrooms are doing so without guidance on what those decisions mean for student data. Education Week survey, via Chalkbeat, 2024

0×	FERPA — the primary federal law protecting student data — has been enforced by withholding federal funding exactly zero times since it was signed in 1974. Elizabeth Laird, Center for Democracy and Technology, via Axios, 2025

What follows is a framework for asking the right questions — about the tools you're already using, before your students' data is already somewhere you didn't intend. Try it out and provide your feedback on how it went!

– The Shift

Most teachers evaluate AI tools the same way they evaluate any resource: Is it useful? Does it save me time? Do students engage with it? Those are reasonable questions. They’re just not the only questions.

The shift is from “is this tool useful for learning?” to “who benefits from this tool, and what do they get that my students don’t?”

As Axios reported in August 2025, AI companies are “constantly seeking data to train their models.” They’re not required to say exactly where they get it. When a teacher pastes a student’s writing sample into a free AI tool to generate feedback, that sample may become training data. The company grows its model. The student receives — maybe — useful feedback. The asymmetry is rarely named.

Armando Martinez, an English and media literacy teacher in Albuquerque, models the disposition this shift requires. A self-described AI enthusiast who uses it frequently for lesson planning and generating materials, he described to Chalkbeat how he draws explicit lines on when to use AI: he intentionally does not use AI for IEPs or grading student work. Not because he’s anti-technology. Because he’s deliberate about what student data travels and where.

That deliberateness is what data literacy looks like in practice when it comes to AI. Not avoidance. Not uncritical adoption. Informed, intentional use based on actually knowing what a tool does with what you put into it.

– A Proactive Experiment

Think of this as a personal investigation or a research project you conduct about the tools already in your classroom. The five questions below are grounded in guidance from the Future of Privacy Forum and reporting by K-12 Dive and Chalkbeat. Run them on one tool this week.

Q1 Does this tool use student inputs to train its model?
→ Why this matters: This is the core question. While some tools built specifically for education may have some protections in place, it is important to notice when/if that company is using student or your content to improve their model(s). Your information is valuable.

Q2 How long is student data retained, and what is the deletion policy?
→ Why this matters: Khan Academy deletes chat data after 365 days. Many platforms don’t publish this at all. If you can’t find the deletion policy, that’s information. A tool that won’t tell you how long it keeps student data probably keeps it longer than you’d want.

Q3 Is this tool district-approved? Does your district have a Data Processing Agreement with this vendor?
→ Why this matters: FERPA and COPPA require that any vendor processing student data operate under a formal agreement with the district — typically a Data Processing Agreement — that restricts how that data can be used. Most shadow AI tools have no such agreement in place. That means the district may be out of compliance, and you may be creating that exposure without knowing it.

Q4 What student information does this tool actually need — and what am I giving it beyond that?

→ Why this matters: A tool that generates feedback on writing samples doesn’t need a student’s name. A tool that suggests interventions doesn’t need a student’s IEP. The Future of Privacy Forum notes that “a name combined with a school ID number, disability status, or even a writing sample could easily identify a student” in a small district. Anonymizing inputs where possible is the simplest protection.

Q5 If this tool disappeared tomorrow, what would happen to the data it holds?
→ Why this matters: The LA Unified “Ed” story is the test case. When a platform shuts down — financially, legally, or by acquisition — data obligations become murky. Ask where data is stored, whether it would be returned or deleted, and what the contractual obligation is on shutdown. If there isn’t one, that’s the answer.

Not All AI Tools Are the Same

A rough spectrum, from most to least protective of student data:

TOOL TYPE	PRIVACY POSTURE
Built for education, privacy-focused	Strongest protections; still ask the five questions
Built for education, verify carefully	Safer but depends on how you use them; check DPA status
Consumer tools, repurposed	Policies vary by version; do not enter student PII
Unapproved / shadow AI	No DPA; unknown data handling; highest risk

– In the Room

Armando Martinez’s approach is worth sitting with. He didn’t arrive at his lines by reading a policy. He arrived at them by thinking through the question: what does this tool do with what my students put into it? For IEPs and grades, the answer was unacceptable. For lesson brainstorming, it wasn’t.

That’s the kind of deliberate reasoning this series is designed to build — a judgment formed by you, based on understanding how these systems work.

One teacher described her process after running the five questions on an AI feedback tool she’d been using for two months: “I found out my district had no DPA with them. The terms of service said inputs could be used for ‘product improvement.’ I stopped pasting student writing into it. I still use it for generating example texts.” That’s not avoidance. That’s data literacy.

– Where This Could Go Wrong

Individual data literacy won’t fix systemic failures. The PowerSchool breach wasn’t caused by a teacher pasting something into the wrong tool. FERPA not being enforced isn’t a teacher problem. District partnerships designed partly to acquire student data aren’t something individual teachers can veto.

But here are a few things individual teachers can do:

• Run the five questions on tools before using them with students.
• Anonymize student inputs wherever possible. A writing sample doesn’t need a name attached to produce useful feedback.
• Escalate DPA questions to your admin. “Does our district have a DPA with this tool?” is a reasonable professional question, not a complaint.
• Participate in school and district conversations about AI policy. Those conversations are happening right now and teachers should be in the room.
• What requires collective action: updating FERPA for the AI era, enforcing existing law, building district procurement processes that actually protect students. Those are worth pushing for at the union level, at the school board level, and at the state level.

A 2025 Student Privacy Compass analysis found that state AI guidance for schools is overwhelmingly superficial. Most states say little more than perfunctory statements about the importance of data privacy, without giving schools the specific, actionable direction they need. That gap will be closed by educators who understand what's at stake and name it clearly.

– Try It Out

Our Learning Lab for the Week: Let Us Know How It Goes!
The experiment	Pick one AI tool you currently use with students. Run the five questions on it. You may need to look up the terms of service. Budget 15 minutes.
What to look for	Does it use inputs for training? What’s the retention/deletion policy? Does your district have a DPA? What student data are you actually providing it?
Share it	What did you find? Drop it in the comments: which tool, what you learned, and what (if anything) you’re changing as a result. This is how we build collective data literacy.

What did you find when you ran the five questions? Which tool surprised you — in either direction? This series grows through what you share.

---

This is part of Accingo's Learning Lab, where we provide small, structured experiments you can run inside a normal teaching week.