The robots.txt Moment for AI: Why Every Website Will Need an Open MCP Manifest

A departure from Optimizely into the emerging agentic web

In 1994, a Dutch software engineer named Martijn Koster proposed a simple text file that would change the web forever. The robots.txt standard gave website owners a way to communicate with automated crawlers — telling them what they could and couldn't access. It was elegant, decentralized, and it worked because everyone agreed to follow the same convention.

Thirty years later, we're at another inflection point. But this time, the crawlers aren't just indexing content — they're taking actions on behalf of users. And right now, there's no robots.txt equivalent to help AI agents understand what a website can do.

I believe that's about to change.

The Problem: AI Agents Are Flying Blind

If you've watched any demo of an AI agent booking a flight, ordering food, or filling out a form, you've probably noticed something: it's impressive, but also brittle. The agent is essentially screen-scraping, clicking buttons, and hoping the UI hasn't changed since its last training run.

This approach has fundamental limitations. Every website interaction requires the agent to interpret visual layouts, guess at functionality, and navigate interfaces designed for humans. It works — sometimes — but it's slow, error-prone, and doesn't scale.

The web needs a better way for sites to declare: "Here's what I can do, and here's how an AI agent can do it."

What's Already Emerging

Before I share where I think this is headed, it's worth understanding the fragmented landscape that's forming right now. Several parallel efforts are attempting to solve pieces of this puzzle:

llms.txt is a proposal for a markdown file at the root of websites that helps LLMs understand site content. Think of it as an AI-optimized sitemap — it tells language models where to find important documentation and how content is organized. Over 600 websites have adopted it, including Anthropic, Stripe, Cloudflare, and Perplexity. But llms.txt is about content discovery, not action capabilities.

WebMCP is a W3C proposal from the Web Machine Learning Community Group that allows websites to expose JavaScript functions as "tools" that AI agents can invoke. It's built around a browser API (navigator.modelContext.registerTool()) that lets developers define actions with natural language descriptions and structured schemas. This is powerful, but it requires navigating to a page and executing JavaScript to discover what's available.

MCP (Model Context Protocol) from Anthropic standardizes how AI models connect to external tools and data sources. It's often described as "USB-C for AI" — a universal interface for tool integration. MCP is powerful for connecting agents to backend services, but it doesn't address public web discovery.

A2A (Agent-to-Agent Protocol) is Google's open standard, and it's the closest thing to what I'm envisioning — so it deserves a deeper look.

A2A: Close, But Designed for a Different Problem

Google's A2A protocol, announced at Cloud Next '25 with backing from over 50 technology partners, is an ambitious attempt to standardize how AI agents communicate. It includes a concept called an "Agent Card" — a JSON document hosted at /.well-known/agent.json that describes an agent's capabilities, skills, API endpoints, and authentication requirements.

That sounds a lot like what I'm proposing. So why do I think there's still a gap?

A2A is designed for agent-to-agent collaboration, not website-to-agent discovery.

The protocol assumes both sides of the interaction are autonomous AI agents — systems that can reason, negotiate, and collaborate on complex tasks. Google's documentation describes A2A as enabling "true multi-agent scenarios" where agents work together without sharing memory, tools, or context.

The architecture reflects this. A2A is built around tasks with full lifecycles — submitted, working, completed, failed. Tasks can be long-running, with agents streaming updates to each other via Server-Sent Events. One agent acts as the "client" initiating a request; another acts as the "remote" agent fulfilling it. There's support for push notifications, webhooks, and complex negotiation flows.

This is powerful for enterprise workflows. Imagine a hiring pipeline where a sourcing agent finds candidates, a scheduling agent books interviews, and a background-check agent runs verifications — all coordinating autonomously through A2A. Google explicitly designed the protocol for these multi-agent orchestration scenarios.

But here's the thing: most websites aren't agents.

When I ask my AI assistant to "find me a red dress under $200 on Nordstrom," I don't need Nordstrom to be an autonomous agent that negotiates and collaborates. I need a simple way for my assistant to discover that Nordstrom has a product search API, understand what parameters it accepts, and invoke it on my behalf.

That's a fundamentally simpler interaction pattern:

A2A solves agent-to-agent collaboration beautifully. But before agents can collaborate with a website, they need to discover what that website can do. That's the missing layer.

The Missing Piece: Open MCP Manifests

Here's the pattern I see emerging. Just as we have:

• robots.txt — tells crawlers what they can't access
• sitemap.xml — tells crawlers what content exists
• llms.txt — tells LLMs what content is important

We need:

• /.well-known/mcp.json — tells agents what actions are available

I'm calling this concept an Open MCP Manifest — a static, machine-readable file at a well-known location that declares the capabilities a website exposes to AI agents.

The key differences from A2A:

1. Websites, not agents. The manifest describes a website's capabilities, not an autonomous agent's skills. The site doesn't need to reason or negotiate — it just needs to expose structured tools.
2. Tool-oriented, not task-oriented. Instead of complex task lifecycles, the manifest simply declares: here are the tools, here are their parameters, here are their endpoints. Stateless, simple, RESTful.
3. Discovery-first. The manifest enables pre-flight discovery. An agent can fetch it with a single GET request, understand what's possible, and make routing decisions before any navigation or authentication.
4. Consumer web focus. Designed for the everyday web — e-commerce, content sites, SaaS products — not just enterprise agent orchestration.

The manifest format could potentially be compatible with A2A's Agent Card schema, allowing sites that want richer agent interactions to evolve toward full A2A support. But the baseline should be simple enough that any website with an API could publish one.

What Would This Look Like?

Imagine you ask an AI assistant: "Find me a red dress under $200 for a summer wedding and add it to my cart on Nordstrom."

Today, the agent would need to navigate to Nordstrom, visually parse the interface, figure out how search works, interpret product listings, and attempt to interact with the cart — all through the same interface humans use.

With an Open MCP Manifest, the agent could first fetch nordstrom.com/.well-known/mcp.json and discover:

{
  "name": "Nordstrom",
  "version": "1.0",
  "description": "Fashion retailer with designer and everyday brands",
  "tools": [
    {
      "name": "product_search",
      "description": "Search for products by keyword, category, color, price range, occasion, and other attributes",
      "inputSchema": {
        "type": "object",
        "properties": {
          "query": { "type": "string", "description": "Search terms" },
          "category": { "type": "string", "enum": ["dresses", "shoes", "accessories", "..."] },
          "color": { "type": "string" },
          "maxPrice": { "type": "number" },
          "occasion": { "type": "string", "enum": ["casual", "wedding", "formal", "..."] }
        },
        "required": ["query"]
      },
      "endpoint": "/api/mcp/v1/search",
      "rateLimit": { "requests": 100, "period": "hour" }
    },
    {
      "name": "add_to_cart",
      "description": "Add a product to the shopping cart",
      "inputSchema": {
        "type": "object",
        "properties": {
          "productId": { "type": "string" },
          "size": { "type": "string" },
          "quantity": { "type": "integer", "minimum": 1 }
        },
        "required": ["productId"]
      },
      "endpoint": "/api/mcp/v1/cart/add",
      "authRequired": true,
      "scopes": ["cart:write"]
    }
  ],
  "authentication": {
    "type": "oauth2",
    "authorizationUrl": "https://nordstrom.com/oauth/authorize",
    "tokenUrl": "https://nordstrom.com/oauth/token",
    "scopes": {
      "cart:read": "View cart contents",
      "cart:write": "Add/remove cart items",
      "orders:read": "View order history"
    }
  },
  "policies": {
    "humanApprovalRequired": ["checkout", "payment"],
    "maxRequestsPerSession": 500
  }
}

Now the agent knows exactly what's possible. It can search products with structured parameters, understand that cart operations require authentication, and respect the site's requirement for human approval before checkout.

Why This Matters Now

Several trends are converging to make this not just possible, but necessary:

Traffic patterns are shifting. Vercel reports that 10% of their signups now come from ChatGPT. This isn't an anomaly — it's the beginning of a fundamental change in how people discover and interact with web services. Sites that are "agent-friendly" will capture this emerging traffic.

The standards bodies are paying attention. The W3C Web Machine Learning Community Group is actively working on WebMCP. Google's A2A protocol includes /.well-known/agent.json for agent discovery. The MCP community is discussing well-known URI conventions for server discovery. The pieces are moving into place.

Agents are becoming capable enough to use structured tools. The latest generation of language models can reliably parse JSON schemas, make API calls, and chain together multi-step workflows. The bottleneck isn't agent capability anymore — it's the lack of structured interfaces to work with.

The SEO-to-AEO transition is beginning. Just as Search Engine Optimization became essential for web visibility, Agent Engine Optimization (or whatever we end up calling it) will become essential for agent visibility. Sites that expose structured capabilities will be the ones agents can actually help users interact with.

Design Principles for Open MCP

If this standard is going to work — and I believe something like it will emerge within the next few years — it needs to be built on solid principles:

Static and declarative. The manifest should be a simple JSON file that can be fetched with a single HTTP GET request. No JavaScript execution, no complex handshakes. This enables pre-flight discovery and caching.

Progressive enhancement. The manifest should work alongside existing approaches. Sites can expose server-side MCP endpoints, reference WebMCP tools on specific pages, or provide simple read-only tools. Start simple, add complexity as needed.

Trust and transparency. The manifest should clearly declare authentication requirements, rate limits, human-approval requirements, and scope boundaries. Both users and agents need to understand what they're agreeing to.

Protocol agnostic. While the manifest format might be inspired by MCP, it shouldn't be tightly coupled to any specific protocol version. The web moves slowly; we need room for evolution.

Backwards compatible. Sites should be able to adopt this incrementally. A minimal manifest with just one or two tools is still useful. Complexity should be opt-in.

How This Fits With A2A and MCP

To be clear: I'm not proposing something that competes with A2A or MCP. These protocols solve different problems, and they can work together.

Think of it as a stack:

A website could start with a simple Open MCP Manifest exposing a few tools. As their needs grow, they could adopt MCP for richer tool integration. And if they eventually want to offer autonomous agent capabilities — where their service can reason, negotiate, and collaborate on long-running tasks — they could evolve toward full A2A support.

The manifest format could even be designed as a subset of A2A's Agent Card, making the upgrade path seamless. A basic manifest might look like a "lightweight Agent Card" that doesn't claim the full autonomous capabilities A2A assumes.

What Different Sites Might Expose

The beauty of a standardized manifest format is that every site can expose capabilities appropriate to its domain:

E-commerce Sites

1. Product search with structured filters (category, price, size, color)
2. Inventory availability checks
3. Cart management (add, remove, update quantities)
4. Wishlist operations
5. Order status lookup (authenticated)
6. Store locator

SaaS Platforms

1. Account status and usage queries
2. Configuration lookups
3. Documentation search
4. Support ticket creation
5. Integration status checks
6. Feature availability queries

Content and Media Sites

1. Article/content search
2. Topic-based navigation
3. Subscription status (authenticated)
4. Newsletter signup
5. Event registration
6. Comment submission (authenticated)

Financial Services

1. Account balance queries (authenticated, high-security)
2. Transaction history (authenticated)
3. Branch/ATM locator
4. Rate calculators
5. Appointment scheduling
6. Document request submission

Each of these represents actions that users regularly ask AI assistants to help with. Today, the agent has to fumble through human interfaces. Tomorrow, it could invoke purpose-built tools.

The WebMCP Connection

I mentioned earlier that the W3C is working on WebMCP. After reviewing their proposal, I believe Open MCP Manifests, WebMCP, MCP, and A2A are all complementary rather than competing approaches.

WebMCP focuses on runtime tool registration — JavaScript running on a loaded page that registers functions with the browser. This is powerful for interactive, stateful scenarios where the user and agent are collaborating in the same interface.

Open MCP Manifests would focus on discovery and pre-flight — allowing agents to understand a site's capabilities before navigation, enabling routing decisions and workflow planning.

The WebMCP proposal actually acknowledges this gap. Their documentation notes: "One disadvantage of the current registration approach is that the browser must navigate to the page and run JavaScript to discover tools... it may be beneficial to have a way to know what capabilities a page offers without having to navigate to the web site first."

That's exactly what I'm proposing. The manifest could even reference WebMCP tools that become available after page load, as well as signal whether the site supports A2A for more complex interactions:

{
  "tools": [
    {
      "name": "product_search",
      "type": "api",
      "endpoint": "/api/mcp/v1/search"
    },
    {
      "name": "virtual_try_on",
      "type": "webmcp",
      "page": "/products/{productId}",
      "description": "Interactive tool available after loading product page"
    }
  ],
  "protocols": {
    "a2a": {
      "supported": false,
      "note": "Simple tool invocation only; no autonomous agent capabilities"
    }
  }
}

This makes the relationship explicit: the site exposes tools for agent consumption but isn't claiming to be an autonomous agent capable of A2A-style task negotiation. Sites that do want full A2A support could set "supported": true and point to their Agent Card.

What Needs to Happen

For this vision to become reality, several things need to align:

Community consensus on format. The manifest schema needs input from browser vendors, AI providers, and web developers. Ideally, this would align with or extend A2A's Agent Card format rather than creating yet another competing standard. This probably means RFC-style proposals, W3C involvement, and coordination with Google's A2A working group.

Early adopter momentum. Just as llms.txt gained traction through adoption by prominent sites, Open MCP Manifests would need champions willing to implement and iterate publicly. E-commerce platforms and developer-focused SaaS companies seem like natural early adopters.

Agent platform support. AI assistants from Anthropic, OpenAI, Google, and others would need to start checking for and utilizing these manifests. Google's investment in A2A suggests they're already thinking about discovery; the question is whether simpler manifest-only discovery gets supported alongside full A2A.

Tooling and validation. Generators, validators, and testing tools would make adoption easier. Think of the ecosystem around OpenAPI/Swagger — we'd need similar infrastructure for capability manifests.

Security and trust frameworks. As agents start taking actions on behalf of users, we need robust patterns for authentication, authorization, and abuse prevention. A2A has already done good work here that could be leveraged.

The Opportunity for Forward-Thinking Teams

If you're running a website with meaningful functionality — e-commerce, SaaS, content, services — this is worth thinking about now. Not because you need to implement anything today, but because the architecture decisions you make over the next few years will determine how easily you can participate in the agentic web.

Questions to consider:

• Do you have clean, documented APIs for your core functionality?
• Could an AI agent meaningfully interact with your service through structured calls?
• What actions would you want agents to be able to take on behalf of authenticated users?
• What guardrails would you need to prevent abuse?

The sites that will thrive in an agent-mediated web are those that treat AI interfaces as first-class citizens alongside human interfaces — not afterthoughts bolted on when the pressure becomes unavoidable.

A Prediction

Within three years, I expect we'll see:

1. A convergence around A2A's Agent Card format (or something compatible with it) for capability discovery, with a "lite" profile for simple tool exposure
2. Major browser vendors implementing WebMCP or similar APIs for runtime tool registration
3. At least one of the major AI assistants checking for capability manifests as part of standard web interactions
4. E-commerce platforms and CMS systems offering built-in manifest generation
5. An emerging discipline of "Agent Experience Optimization" alongside SEO

The exact form this takes might differ from what I've outlined here. Perhaps A2A's Agent Card becomes flexible enough to serve both full agent-to-agent scenarios and simple website tool discovery. Perhaps the W3C WebMCP effort incorporates declarative discovery. Perhaps something entirely new emerges.

But the need for static, declarative capability discovery is clear, and the ecosystem is moving in this direction. Whether it's called Open MCP Manifests, Agent Cards, or something else entirely, the pattern will emerge.

Conclusion

The web has always evolved through agreed-upon conventions. Hyperlinks, forms, cookies, REST APIs, OAuth — each represented a moment where the community said "this is how we'll solve this problem" and then built on that foundation.

AI agents interacting with websites is the next frontier. Right now, it's the wild west — screen scraping, brittle automation, and hope. But a better pattern is emerging. Google's A2A protocol shows that major players are thinking seriously about agent discovery and communication. The W3C is working on WebMCP for runtime tool registration. The pieces are moving into place.

What's needed is a simple, accessible entry point — a way for any website to say "here's what I can do" without needing to become a full autonomous agent. Whether that ends up being a lightweight profile of A2A's Agent Card, an extension of the WebMCP proposal, or something new, the pattern will emerge because the need is clear.

The sites that prepare for this future — by building clean APIs, thinking through agent use cases, and participating in emerging standards — will be the ones that thrive as the web continues to evolve.

The robots.txt moment for AI is coming. The question is whether you'll be ready when it arrives.

Interested in discussing this further? I'd love to hear from others thinking about the agentic web. The W3C Web Machine Learning Community Group is a great place to follow WebMCP development. Google's A2A protocol has its own community and documentation at a2a-protocol.org. And the MCP community continues to evolve the tool integration story.

If you're building in this space or have thoughts on how website-to-agent discovery should work, let's connect.