Have you been DDoS’d by a large-scale model today? We once marveled at how the rise of generative AI drove players like OpenAI, Anthropic, and Google to an almost frenzied thirst for internet content. Whether it’s the conversational prowess of ChatGPT and DeepSeek or the reasoning capabilities of Claude, all of these rely on vast amounts of data to fuel them. However, this “take-what-you-want” approach is placing a heavy burden on small and medium-sized internet services, even evolving into an invisible DDoS crisis.
The Open-Source Community’s Nightmare: Relentless AI Crawlers
In fact, it’s not just content and social media sites suffering from “large-model DDoS attacks.” Recently, the open-source community has begun complaining that omnipresent AI crawlers are devouring project bandwidth like a swarm of locusts.
Just as 2025 began, software developer Xe Iaso reached a breaking point—not because of buggy code or a crashed CI/CD pipeline, but because their Git repository was rendered nearly unusable by AI scrapers. Traffic spiked abnormally, frequent outages became the norm, and defenses were overwhelmed. Even standard measures like robots.txt, blocking user agents, and IP blacklists proved useless.
“They lie, disguise themselves as browsers, and use residential broadband IPs with proxies,” Iaso said. In the end, they had to hide their service behind a VPN and build a custom “Turing Gate” system called Anubis, requiring all visitors to solve a computational puzzle to access the page. “I don’t want to shut down this Gitea service, but I’m really at my limit,” they admitted.
This isn’t just Iaso’s nightmare. More and more open-source maintainers are finding their projects dragged into a persistent “distributed denial-of-service” attack—DDoS—by hordes of AI crawlers. According to a recent LibreNews report, some projects see up to 97% of their traffic coming from AI companies. Bandwidth is being leeched for free, project stability is plummeting, and maintainers—already stretched thin—are struggling to keep up.
Crawler Chaos: The “Invisible Attacks” from AI Companies
It’s not just Iaso sounding the alarm—frontline project operators are raising red flags too:
- The Fedora Pagure project had to block the entire Brazil region due to abnormal traffic.
- GNOME GitLab adopted Iaso’s Anubis system, and the results showed that only 3.2% of access requests passed verification—the rest were almost entirely automated scrapers.
- KDE GitLab was once knocked offline entirely by traffic from Alibaba Cloud IP ranges.
Even more absurdly, these crawlers don’t just “scrape once and leave.” They return every six hours, like some kind of digital nomadic tribe. Dennis Schubert, an operator for the Diaspora project, called this behavior “a DDoS on the entire internet.”
Beyond technical strain, there’s a real financial cost. The Read the Docs project, for instance, blocked AI crawlers and saw bandwidth usage drop 75%—from 800GB to 200GB—saving $1,500 a month.
Open-Source Projects: We’re Not Your Free AI Training Set
The behavior of AI crawlers can be summed up in one line: “robots.txt is a suggestion, and IP bans are a moral quiz.” These crawlers spoof browser info, switch proxy IPs, and bypass all protocol restrictions—all for one goal: to keep scraping data.
Martin Owens of the Inkscape project lamented, “It’s not just ‘China’s DDoS’ anymore—it’s bots pretending to be browsers from a ton of companies.” He now maintains an “epic blocklist” and warned, “If you work at an AI company, you might already be unable to access our site.”
Adding to the headache, some open-source projects are now receiving AI-generated fake bug reports. Daniel Stenberg, maintainer of the Curl project, noted in his blog that these reports seem plausible but are entirely fabricated, effectively wasting developers’ time.
Drew DeVault, founder of SourceHut, put it bluntly: “They don’t just grab the main page—they scrape git blame, commit histories, every single line of code.”
Who’s Scraping? And Why?
According to traffic analysis from Diaspora, OpenAI accounts for a quarter of all requests, Amazon 15%, and Anthropic 4.3%. Some sources remain unidentified, with highly sophisticated spoofing capabilities.
The purposes vary. Some scrape to train large models, others to enable AI assistants to “real-time” fetch data for user queries—or perhaps both. The repeated scraping suggests this isn’t a one-off training run but a “continuous sampling” mechanism—AI companies are using open-source content from across the web to daily update their models.
As for accountability, the industry consensus is clear: “AI firms with billions in funding cling to public resources without taking responsibility.” Developers on Hacker News have called out the big players’ attitude: “Who still cares about open-source ethics?”
Resistance Begins: From Puzzle Gates to Trap Systems
Facing this “legal but rogue” behavior, a group of developers has started fighting back.
Beyond Iaso’s Anubis, “Aaron” developed another tool—Nepenthes (pitcher plant)—which creates fake webpage mazes to trap AI crawlers in endless loops, wasting their computational resources.
Cloudflare has also rolled out its own “AI Labyrinth” system. Instead of blocking or banning, it redirects AI into mazes of AI-generated content, draining their computing power and bandwidth.
Meanwhile, the community launched the ai.robots.txt project, offering a blacklist of AI crawlers and a one-click config file to help websites defend themselves.
But these are still stopgap measures. AI crawlers continue to evolve, even mimicking normal user behavior. Most critically, the majority of AI companies have stayed silent on the issue, showing no willingness to collaborate with the open-source community.
In Closing: Large Models’ Appetite Outgrows What Open Source Can Feed
The rise of AI should have been a moment of pride for the open-source community—after all, much of the tech stack and training data came from here. But now, this “primitive accumulation” has turned into outright plunder.
If AI companies keep ignoring the rules and scraping global open-source projects “by the hour,” the day will come when these infrastructures—once the backbone of AI’s rise—start collapsing one by one.
When that happens, the open-source community won’t be the only victim.
