Pao Ramen

AI is eating the Internet

“You see? Another <something> ad. We were just talking about this yesterday! How can you be so sure they’re not listening to us?” – My wife, at least once a week. Internet advertising has gotten so good, it’s spooky. We worry about how much “they” know about us, but in exchange, we got something future generations may not: free content and services, and a mostly open Internet. It is unprecedented Faustian bargain, one that is now collapsing. A Memory of Summer: The Ad-Based Utopia At the epicenter of the modern Internet sits Google. Forget the East India Company, Google, with an absurd +$100B in net income, is arguably the most successful business in history. By commanding nearly 70% of the global browser market and 89% of the search engine market, they dominated Internet through sheer reach. How did this happen? A delicate balance of incentives where every player on the Internet got exactly what they wanted: Consumers enjoyed free access to virtually unlimited information and online services. Content creators got a steady stream of visitors flowing from search engines, which they could monetize via ads or conversions. Advertisers gained demand for their products at a reasonable price, thanks to unprecedented targeting precision. Google crawled the entire Internet, and site owners didn’t just allow it: they optimized for it. Users found what they wanted (most of the time, for free), while advertisers happily footed the bill because digital ad targeting finally began to solve the old John Wanamaker dilemma: “Half the money I spend on advertising is wasted; the trouble is I don't know which half.” Later on, Meta (then Facebook) doubled down on the same ad-driven premise. The social graph plus cross‑device identity made its targeting scary good. And advertisers flooded in. Google saw the threat but Zuck went full Carthago delenda est. By the late 2010s, the Google-Facebook duopoly ran the ad‑funded Internet: Google caught “I’m looking for it” intent; Meta created “make me want it” demand, and both printed money. Winds of Winter: The Transformer revolution Even the mightiest empires fall, and Google unknowingly sowed the seeds of its own undoing. In 2017, Google researchers published “Attention is all you need,” introducing the Transformer architecture. Originally meant to improve translation services, this innovation also led to enhance search quality with models like BERT. When OpenAI’s ChatGPT launched (as a free research demo in late 2022), it spread like wildfire, reaching 100 million users in just two months. The fastest adoption of any consumer application in history. Suddenly, millions of people started replacing the search box with a chat box. Why type a query and sift through ten blue links (from which 2-5 are ads) when an AI could synthesize an answer for you? The Internet’s longstanding equilibrium began to topple. Today’s landscape is one of turmoil, change, excitement, and fear. Content is no longer traded for traffic; content has become training data. And data is the fossil fuel of AI. This shift has unleashed a wave of new players crawling the web, often offering nothing in return. Much like early Google News aggregating snippets (which infuriated publishers), AI companies now repurpose the content they ingest and serve it directly in chat responses or summaries, without sending users to the original source. It’s breaking the ad-based Faustian bargain. Content creators, realizing their work is being pillaged instead of traded, are rapidly fencing off their gardens. Paywalls are rising higher than ever, and anti-bot measures are getting more aggressive. In an ironic twist, human users often get caught in the crossfire of bot hunting Inquisition (CAPTCHAs, login walls, broken RSS feeds), as sites desperately try to distinguish real readers from scraping bots. The once-open web is curling back into a fragmented, medieval patchwork of walled cities. Walled cities against the botsEven Google, realizing its core search business is at risk, has decided to commit seppuku rather than be usurped by outsiders. Google is now rolling out AI-generated answers in search results, essentially cannibalizing its own traffic. “Sure, Search might die, but I’d rather kill it myself first.” Like the dolphins in The Hitchhiker’s Guide to the Galaxy, they’re waving as they swim away: “So long, and thanks for all the crawled content.”. Early evidence from Google’s experiment is sobering: when Google shows AI answers, organic results suffer a ~50% drop in clicks. In other words, content creators are getting drastically fewer visitors because Google’s AI is answering the query directly, and advertisers, seeing organic reach dry up, have little choice but to pour more money into ads for visibility. Turns out the obituaries were premature: Google ad revenue is up 10%. Meanwhile, Meta has been fighting its own wars. Apple’s App Tracking Transparency (ATT) measures dealt a heavy blow to Meta’s ad targeting in 2022, an estimated %37 decrease on click-through rates. Zuckerberg’s costly bet on the Metaverse didn’t pan out, and the core ad-based model remains under threat. Zuck is now in the arena, waving an American flag while fighting everyone with his Open Source AI strategy. But the Chinese labs are playing the same game, and despite GPU bans, they are taking over the benchmarks. And then there’s Cloudflare, quietly rubbing its hands like an old-school gangster offering protection. “Your content is safe with us,” they say, while simultaneously launching browser automation to those with crawling needs. In July 2025, Cloudflare announced a new service to block AI crawlers by default unless they pay up or get permission. It’s offering a potentially lucrative third way for the Internet’s future. It’s not a classic paywall. It’s not ads. It’s essentially “Pay to Crawl”: think of it as the digital equivalent of a nightclub’s ladies’ night: humans get in free, bots pay cover. Reddit, Stack Overflow, and major news publishers are already striking data licensing deals, turning this data monetization model into reality. In effect, if data is fossil fuel, the web’s knowledge is being fracked at all costs. Lastly, as generative bots overrun the public web, humans are retreating to more intimate corners. The cozy web of private chats, invite-only forums, and niche communities. When every other social post or blog could be written by ChatGPT, people naturally gravitate to spaces where real identity and authenticity are ensured. It’s the chess problem: robots may play perfectly, but nobody shows up to watch robots play each other. This has lead to a resurgence of email newsletters, influencers, small Discord communities, and Substacks where a named human is accountable for the content. robots dominate the open webA Dream of Spring: The Internet to come Eventually, the dust will settle. Some players will win (and take all), others will fall. But a new equilibrium must emerge. One that rebalances the interests of content creators, consumers, and advertisers. I could stop here and call it a day. But if you’ve made it this far, you deserve more than an open‑ended conclusion. The past and present are easy. An LLM can already write them better than I can. The future? That’s exposure. I’ll make the call and own it if I’m wrong. Looking ahead, the collapsing ad-funded, crawl-for-traffic model seems likely to resolve into three intertwined markets: Ad-based Consumer AI: Whether it’s an AI chatbot, personal assistant, or generative search hybrid, the winner of the consumer AI category will become a new Super-Aggregator. They’ll replace search (and perhaps even browsers, which every company seems to be building) as the gateway through which billions access information and services. And they will monetize primarily via advertising. Why ads? Again? Because it’s too much money for anyone to leave on the table. In 2024, Google made ~$265B from ads. To match that with subscriptions alone would be herculean: it’d take roughly 13 billion monthly $20 subscriptions. That’s 5 billion short of Earth’s population, and demographic collapse and virtual waifus won’t save that math. The business logic points toward a free (or low-cost) consumer AI service with a massive user base, monetized by targeted advertising, just as search and social networks were. The Open Cozy web: Trust in faceless content and large media will erode. When any article might be AI-generated slop, people will increasingly trust individuals and communities instead. There will be just too much content to be able to sift the signal from the noise. Influencers, domain experts, and friend networks will become the sources people rely on more than generic brands. We’re already seeing the shift: readers flock to personal Substack newsletters from writers they trust, and niche, closed communities thrive while broad public forums struggle with bot moderation. In this future, human provenance is the key problem to solve. We’ll need individual “fingerprints” to verify that specific humans created specific pieces of content. Once we can reliably prove and label human‑made work, I’m optimistic that the pendulum will swing back from centralized, walled platforms to a constellation of decentralized, open spaces. Pay for Crawl: Quality human-generated data will become scarce and more valuable, especially as the Internet’s data gets increasingly contaminated by AI output. Models are already being trained on AI-generated content, and like royal inbreeding, the results aren’t looking good. Pay‑per‑Crawl will displace content‑licensing. Licensing builds walls: everything behind a fence, more logins, more friction, less traffic. Pay‑per‑Crawl strikes the perfect balance: humans can roam freely, but bots pay at the gate. Perhaps browsers will carry a kind of “human token” to prove you’re not a bot (a modern twist on HTCPCP). The open web won’t die, but it’s likely to have ID checks at the gate. robots showing ID at the gateWho will be the consumer AI winner? Two frontrunners stand out to me, for very different reasons: OpenAI: They currently have the strongest consumer AI brand. Despite the clunky name, “ChatGPT” has become synonymous with AI for hundreds of millions of people. OpenAI’s move to integrate web browsing into ChatGPT (with source citations and clickable results) is a clever attempt to repair relations with content creators by sending traffic back. But to truly reach Google-like gargantuan revenues, they would need to introduce advertising or sponsored results. OpenAI’s partnership with Microsoft (which brings Bing’s search index and advertising platform into the mix) hints that ads could be coming. If OpenAI can leverage its head start in user base, but it faces the challenge of scaling revenues without alienating users: not everyone will tolerate ads in their assistant. Google: Don’t leave out the current king. Google has decades of experience integrating ads in a way users tolerate, and it already has relationships with millions of advertisers. It also still controls Android, Chrome, Gmail, Maps, YouTube, and other personal platforms that give it a ridiculous data and distribution advantage. While Google was caught unprepared by the launch of ChatGPT, by 2025 it launched capable large models of its own (Gemini, Imagen, Veo, etc.), demonstrating serious technical muscle. Critically, Google doesn’t have to justify a sky-high startup valuation: it’s immensely profitable already and can afford to play the long game. This might allow Google to adopt AI more deliberately, balancing the interests of advertisers, content creators and consumers as it transitions. In the end, Google’s biggest advantage is that it controls the default channels (your phone, your browser, your email) and can bake its AI into all of the above. That, combined with a less constrained business model (they won’t mind cannibalizing some revenue to protect their moat), makes Google a formidable contender in the consumer AI game. My bet is on Google pulling through. That’s it, I said it. So, the bargain that ruled the Internet for two decades is ending. The free ride of free content, propped up by unseen ad targeting, is giving way to something new, something perhaps more balanced, but also more fragmented. “Software is eating the world,” Marc Andreessen declared in 2011. Now, “AI is eating the Internet”. The only questions are: who gets to digest the value, and how will the meal be shared? What do you think? What’s your take? The winter winds are howling, but perhaps, just perhaps, a new spring is coming.

Jul 28