It’s 4:30 PM on a Thursday. A school district leader is tasked with selecting a new Tier 2 literacy intervention by Monday morning. Where should they look to help them make their decision? Knowledge brokering research tells us that they will likely do some combination of three things: (1) Reach out to a social contact for advice, (2) Review the vendor-provided materials, or increasingly: (3) Open up ChatGPT. So after chatting with a friend, and looking over the website, this school leader opens a browser and types “What is the most effective evidence-based reading program for third graders with dyslexia?” into ChatGPT.
The era of the “AI-as-Consultant” is officially here. For knowledge brokers—those of us who inhabit the bridge between academic research and classroom reality—this shift is both an opportunity and a significant professional anxiety. Educators and leaders are increasingly turning to Large Language Models (LLMs) for “best practice” answers because these tools offer something research papers rarely do: immediacy and actionable clarity.
But as we know, clear and coherent do not always mean correct. If we want AI to be a force for good in education, we have to move beyond general-purpose models and start demanding LLMs that are grounded in the rigors of meta-analysis and high-quality evidence synthesis.
The Allure—and Danger—of the AI Answer
Educators are some of the busiest professionals on the planet. When a teacher asks for a strategy to manage classroom behavior, they don’t want a 40-page literature review on socio-emotional learning; they want a three-step plan they can use tomorrow. LLMs provide that plan in seconds.
However, general LLMs are trained on the “average” of the internet. In the world of education research, the “average” includes outdated fads, debunked theories (hello, learning styles), and marketing materials from edtech companies. For an LLM to be a true partner in knowledge brokering, it cannot simply summarize the internet; it must be trained to prioritize high-quality evidence.
Specifically, the future of educational AI lies in the integration of meta-analysis. Instead of giving a singular anecdote or a broad generalization, an evidence-aligned LLM should be able to synthesize effect sizes across multiple studies, providing a nuanced view of what works, for whom, and under what conditions.
The Gold Standard: Integrating What Works Clearinghouse (WWC) Standards
To transform an LLM from a “fancy Google” into a reliable research assistant, it must be programmed to recognize and prioritize the standards set by the What Works Clearinghouse (WWC). This ensures that the advice given to a principal isn’t just popular, but proven.
To achieve this, the model’s underlying logic should filter evidence through the WWC parameters. To be included in a high-quality synthesis, a study must first meet strict design standards: it should ideally be a randomized controlled trial (RCT) or a quasi-experimental design (QED) that demonstrates rigorous baseline equivalence between the intervention and control groups. The model must evaluate the attrition of the study—ensuring that too many participants didn’t drop out in a way that biases the results—and confirm that there are no confounding factors, such as a single teacher delivering the intervention to only one group. Furthermore, the evidence must be combined using a statistical framework that accounts for real-world complexity.
There is a pathway for this, if large funders were to support efforts to integrate meta-analysis into LLMs, so that the general model can perform fast and accurate responses based on rigorous meta-analysis principles.
From Gatekeeper to Navigator
As knowledge brokers, our role is evolving. We are no longer the sole gatekeepers of information; we are the auditors of the algorithms and the architects of their application.
If a principal uses a research-aligned LLM to identify a Tier 2 intervention, the AI provides the what. But the how—the messy, human work of scheduling, teacher buy-in, and cultural alignment—remains. This is where the knowledge broker fits into the “phone a friend” model. Even the most sophisticated AI cannot replace the “savvy principal,” the instructional coach, or the formal broker who understands the specific constraints of a school’s ecosystem.
There is a profound opportunity here to advocate for the formalization of the “Broker” role within school districts. If the AI handles the heavy lifting of evidence synthesis, the human broker is freed to act as a high-level implementation partner. They become the “expert friend” on the other end of the line, helping leaders navigate the nuances that a model might miss:
- Contextualization: “The AI suggests this program based on a high effect size in urban settings, but how does that translate to our rural district’s staffing?”
- Adaptive Support: “The evidence-based three-step plan looks great, but how do we support Mr. Miller in Room 4 who is struggling with the new software?”
- Integrity Monitoring: Ensuring the “persuasive prose” of the AI doesn’t lead to lethal mutations of the original research design.
Integrating WWC standards into an LLM’s logic acts as a “validity filter,” ensuring the model ignores flashy but unproven marketing and focuses only on studies with rigorous designs and equivalent control groups. Similarly, utilizing REML (Restricted Maximum Likelihood) addresses a core challenge with simple effect size averages: when you pool results across studies conducted in wildly different contexts—different schools, different student populations, different teacher training levels—a straight average can be deeply misleading, masking the fact that an intervention works brilliantly in some settings and barely at all in others. REML accounts for this by modeling the variance between studies, not just within them, producing a more honest estimate of an intervention’s true effect and the range of outcomes a district might realistically expect. Think of it as the difference between a weather forecast that says “the average temperature in the U.S. is 55°F” versus one that accounts for the fact that it’s 90°F in Phoenix and 20°F in Minneapolis—the latter is simply more useful for deciding what to wear. In practical terms, this means the AI can tell a principal not just “this program has an effect size of 0.4” but “this program has an effect size of 0.4, with meaningful variation depending on implementation context”—a far more useful and honest answer.
By advocating for AI tools that integrate WWC standards and sophisticated meta-analytic models like REML, we aren’t trying to automate the broker out of a job. Rather, we are upgrading the broker’s toolkit from a manual shovel to a precision instrument.
Note: The views, opinions, and positions expressed by guest bloggers on this site are theirs alone and do not necessarily reflect the views, opinions, or positions of the Education Knowledge Broker Network or its employees.