What Machines See

Search engines navigate a web with a potentially infinite number of addresses under hard constraints of bandwidth and computational budget. In the January 2007 essay «C'è un buco nella tela del ragno», the analysis originated from core engineering realities: a crawler manages its web exploration via two synchronized queues:

• The Priority Queue: selects which URLs to fetch first, weighing document authority and historical update frequency.
• The Politeness Queue: schedules requests to each origin host, preventing resource exhaustion or denial of service.

To process hundreds of pages per second, early crawling pipelines fetched raw HTML responses exclusively, omitting CSS and JavaScript.

The primary architectural discrepancy of the web is the divide between raw source code and final visual rendering. The crawler parses markup delivered in the HTTP response; the user experiences the interactive layout rendered by the browser.

This gap historically birthed manipulative tactics: delivering divergent content to bots versus humans (cloaking), or hiding keyword blocks via CSS techniques (matching font and background colors, or off-screen positioning like left: -3001px).

For automated algorithms, distinguishing a legitimate navigation dropdown—hidden via display: none until user hover—from manipulative hidden text was non-trivial.

These techniques must be studied to recognize them and keep them strictly away from production systems, not to deploy them, adhering fully to Google's search spam policies.

Because automated parsers could not independently assess human visual context and intent, search engines paired algorithmic indexing with structured evaluations by trained human reviewers.

As clarified by Danny Sullivan for Google, human quality raters do not manually alter individual website rankings: their role is scoring sample sets to determine whether broad algorithmic updates improve or degrade overall search utility for real users.

While Googlebot progressively introduced headless Chromium rendering to evaluate client-side JavaScript, the fundamental asymmetry between source markup and rendered output has reopened with AI scraping and retrieval agents.

According to the benchmark study published by Vercel in December 2024, major AI crawlers (including OpenAI's GPTBot, Anthropic's ClaudeBot, and PerplexityBot) do not execute JavaScript during scraping, with the notable exceptions of Gemini — which relies on Googlebot's rendering pipeline — and AppleBot.

A website that relies entirely on client-side rendering (such as an unrendered Single Page Application) risks appearing substantially blank to these agents. While crawlers can read static JSON-LD metadata embedded in initial payloads, dynamic client-rendered body copy remains unextracted.

Because AI crawler behavior evolves rapidly, treat benchmark data as architectural guidance: inspect your server access logs to see which bots are visiting and whether they fetch only raw HTML or also request JavaScript assets.

To ensure content discoverability across AI models, the foundational architectural answer is returning to web fundamentals: delivering complete semantic markup and structured data directly in the initial HTTP payload delivered by the server.

This is where the 2007 analysis connects directly with modern AI systems engineering: cloaking in the AI era relies on the exact same asymmetry between browser and machine, with text invisible to human visitors but ingested directly by automated models. The fundamental shift is that hidden text no longer packs keywords, but embeds natural language instructions directed to the model (indirect prompt injection).

This is an empirical reality documented in active production:

• The first documented case (December 2024): in the Guardian case, cataloged in the OECD AI Incidents Monitor, hidden text on a web page actively altered what ChatGPT Search reported about that source.
• Google's Common Crawl study (April 2026): in Google's Common Crawl study (April 2026, Google Security), researchers identified hidden directives crafted specifically for SEO manipulation—prompting the assistant to recommend a particular enterprise over competitors. In the malicious category, instances surged 32% between November 2025 and February 2026, though remaining largely unsophisticated.
• Unit 42 field evidence from Palo Alto Networks (March 2026): in the Unit 42 report by Palo Alto Networks, «Fooling AI Agents: Web-Based Indirect Prompt Injection Observed in the Wild», researchers documented 22 techniques observed in live use, including a phishing domain propelled into model recommendations and fraudulent ad copy approved past an AI moderator.

The concealment methods mirror those of 2007: CSS rules (display: none), extreme off-screen offsets, and font colors matching page backgrounds. Modern vectors expand this to text embedded within HTML attributes (data-*, aria-label, alt, comments) or dynamically constructed via client scripts post-load.

For enterprise leadership and engineering teams, this carries two critical mandates:

1. Those tempted to use it face the identical risks of historical black-hat SEO, amplified: Google treats these maneuvers as a security issue and trains its models, starting with Gemini, to recognize and ignore them; hidden text remains strictly prohibited under search spam policies. A domain discovered manipulating assistants risks losing credibility with the very systems it seeks recommendations from.
2. Those architecting agents that read the web must treat external content as untrusted data, never as instructions: trusted system instructions (system prompts) must be strictly segregated from unverified web context, mandating human confirmation (Human-in-the-Loop) before executing consequential actions, such as financial payments, outbound transmissions, or record deletions.

A practical audit requires just three steps to verify whether your web architecture suffers from asymmetry between raw source and browser rendering:

1. Source Inspection: open a content page (not just the home page) and use View Page Source (Ctrl+U, or Cmd+Option+U on Mac): the text you read on screen must be present in the raw HTML payload.
2. AI Crawler Emulation: have your technical team run curl -sL -A "GPTBot" <url> and compare the terminal output against the rendered browser page: if the HTTP response is empty or lacks article content, AI bots see an empty page.
3. Hidden Directive Audit: instruct the team to scan codebases and stylesheets for hidden text via display: none, off-screen positioning (e.g. left: -9999px), or HTML attributes (aria-label, alt, data-*) containing non-pertinent natural language directives.