The crawler

SurfingDogBot

SurfingDogBot is the crawler of the Surfing Dog network. It reads only what a business publishes for machines, to learn how an assistant can find the business and reach it: robots.txt, the homepage, up to three more of the site's own pages about what it offers (services, prices, opening hours, a menu, about, or one of a shop's product pages; never a contact, legal, account or checkout page), the documents for machines those pages point to (an agent inbox manifest, an MCP server, an API description, an AI catalog), and a few files agent standards put at fixed places: llms.txt, an agent inbox manifest, an A2A agent card, an MCP server card, and a shop's UCP commerce profile.

It never fills in a form, never logs in, never buys, books or orders anything, never calls a tool, and never solves a CAPTCHA. When a site asks it to prove it is human, it stops.

How to recognise it

Every request carries this User-Agent:

SurfingDogBot/1.0 (+https://network.surfingdog.ai/bot)

Its identity is checked by signature, not by address: every request is signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519), and the keys it signs with are published in its key directory at https://network.surfingdog.ai/.well-known/http-message-signatures-directory. No address ranges are published; a request that says it is SurfingDogBot and is not signed by one of those keys is not ours.

How it behaves

  • It obeys robots.txt (RFC 9309), under the product token SurfingDogBot, or the rules for every crawler (*) when there are none for it. It reads the robots.txt of every host it visits, www and other subdomains included, before asking that host anything else. If robots.txt cannot be read because of a server error, it reads nothing.
  • It honours Content-Signal: search=no stops it; it records ai-input=no (and Content-Usage that says no to AI), so that what it found on that site is not used as AI input; and it records ai-train=no. If your site tells AI systems not to use or train on its content, we don't name you or list you publicly.
  • By default it sends at most one request every 10 seconds to a host, and slower when Crawl-delay asks. It backs off on 429 and 5xx answers and honours Retry-After.
  • It paces itself per server as well as per host, since one server often hosts many sites. For its crawl, one request at a time to a server address (an IPv4 address, or an IPv6 /64) from all our machines together, and at most one every 15 seconds, however many of the server's sites it is reading; our checks of the doors it found keep to the same pace, and wait while it is reading that server. A 429, 503 or 403 from a server rests the whole server: at least a minute, longer when its Retry-After asks, up to 15 minutes (the host that asked is left alone for as long as it asks). A check someone asks for (below) reads its one site at the check's pace instead, beside the crawl, and only one check reads a server at a time, from all our machines; a free tool someone runs on a site reads that one site at the check's pace too, one request at a time from the machine that runs it. The shared front ends of large platforms and content delivery networks, which answer for a great many unrelated sites at once, get one request at a time from each of our processes, apart by an interval of their own; a 429 there keeps that machine's crawl away from that address, and its checks away from the site that answered it, for two minutes at least, twice as long each time it comes back to another 429, up to an hour.
  • It does not probe for files. It asks a site only for what the site itself publishes or points to: robots.txt, the homepage, the pages and documents for machines they link to, the sitemaps its robots.txt lists, and the documents for machines it found on the site before, where it found them; and for these files alone without being pointed to them, each at most once a visit: /llms.txt, /.well-known/agent-inbox.json, /.well-known/agent-card.json, /.well-known/mcp/server-card.json; on a site that looks like a shop, /.well-known/ucp and /.well-known/acp.json; beside an MCP server the site points to, that server's card; and on a Shopify store, the MCP endpoint Shopify gives every store. A check someone asks for (below) also asks for /.well-known/mcp.json, which the MCP tester reads. It never asks for admin pages, backups, configuration files or any other file it was not pointed to. A free tool someone runs on a site reads only what that tool tests, on that one site: the MCP tester also /.well-known/mcp.json, the shop tester /.well-known/acp.json, and the inbox tester the inbox file on the site's inbox. subdomain and its _agent-inbox DNS record.
  • It reads a limited number of pages per site, each of limited size, and never follows a link off the site while it reads it. A door the business itself declared on another host (when it registered, in a registry or in its manifest) is checked there, read-only, as a door on the site would be: that host's robots.txt is read before anything else, and the same pace and limits apply.
  • When a site's llms.txt names its doors for agents (an MCP server, an API, a product feed, a link that fills a basket), it checks the ones on the site and its subdomains as it checks any door, read-only and within the same limits. A feed is read no further than its first few kilobytes, a link that would fill a basket is noted and never opened, and one on another site is not checked.
  • It records the names of the actions a site offers AI agents (the names of its tools and operations, and the short description a tool gives itself), to learn which kinds of actions sites offer. Never what an action returns: it calls none, but for the catalogue search and the A2A test below.
  • When a site's UCP commerce profile names a catalogue search, it asks it once, as an agent would, to learn whether the business sells anything there: one search for three products, wherever the profile says the catalogue is, that host's robots.txt read and obeyed before anything else. It names itself by its own UCP profile (/bot/ucp), which says it searches and looks up catalogues and does nothing else, and signs the search with keys that profile lists. It counts what comes back (how many products, how many with a price, how many for sale) and keeps no product's name, picture or description. It never fills a cart, starts a checkout or books, and sends nothing about a person. A catalogue host that is busy or asks it to wait is left alone for as long as it asks.
  • When a site's A2A agent card names an endpoint on the site, or on the host the site named the card on, it asks that endpoint, as an agent would, whether it answers A2A at all: tasks/get for a task id it made up (GetTask, A2A 1.0's name for the same, first when the card names A2A 1.x), and the other name when the endpoint knows only that one, so two requests at most, for up to three cards a read. It never asks the card's own address, the site's homepage or the address of another of the site's doors. It reads no task that exists and changes nothing. It never sends a message, starts a task or signs in. It asks after it has read the site's pages; an endpoint robots.txt keeps it from is not asked, and one that shows a bot check or asks it to wait is left alone.
  • When a shop's homepage is too long for it to reach the shop's product links, it reads the sitemap the site's robots.txt lists, and the sitemap of its products that names, to find one product page, each no further than its first 64 KB.

How it finds sites

Besides the sites it is asked to read, it finds sites on its own. When it reads a site, it notes which other sites that site's pages link to, and which other sites the site's documents for machines name (an AI catalog, an agent card, an MCP server card, llms.txt, a commerce profile). It fetches nothing extra to do this. It skips links marked nofollow, ugc or sponsored, and every link on a page that asks for that with <meta name="robots" content="nofollow"> (or none, or name="SurfingDogBot") or an X-Robots-Tag: nofollow header (for every crawler, or SurfingDogBot: nofollow).

  • A site found this way is visited later, on its own, like any other: its robots.txt first, with the same pace and limits, and after that no more often than any other site.
  • This is switched off at the moment: no site is added this way.
  • It never adds a site that opted out, and it leaves out social networks, search engines, link shorteners and other big platforms.
  • Which site led it to yours is kept for the network's operators only, never published, and deleted if either site opts out below, or when the crawler next finds that either site's robots.txt keeps it out.
  • To keep it away from your site, disallow it in robots.txt or opt out below. Either works whether or not it has visited you yet.

Listed on Surfing Dog Network?

The network lists a business it found on the business's own website only when an AI can already ask, book or buy there: the site publishes a machine door for agents (an agent inbox, an MCP server, an A2A agent card, an API, UCP or ACP checkout) that answers asks, bookings or orders. A site that offers only a phone, an email or a contact form is not listed.

  • What is shown: the facts the business published about itself, on its own site or at its own doors, each with where it was read and when: its name, what it does, its town, region and country, its opening hours, its languages, the payments it shows, and its doors with what each takes. From rules version 12 (/v1/ranking), where the site gives none of its own, an entry may also show the area, kind and name of its one place in public maps data (Overture Maps, © Overture Maps Foundation), once the crawler has matched that place to the site (the place's name on the site's own pages, with its town, postcode or phone): each is labelled maps, and none is shown for a business with more than one place. Never a phone number or an email address, no names of people other than the business's own name (a sole trader's business may carry theirs), and never the street or the exact spot: an entry the business has not claimed shows its town and country, and is found by distance only to within a few kilometres.
  • Every such entry says so: found on its own website · not a member · checked <date>, with a link to this page.
  • An entry found on a site whose robots.txt tells AI systems not to use or train on its content (ai-input=no or ai-train=no in Content-Signal, or the same in Content-Usage) is not listed while nobody has claimed it.
  • Every kind of business is listed this way except adult services, gambling, weapons, tobacco, cannabis and vaping, payday loans and bail bonds.
  • To correct the entry, or claim it: from the business's own AI, call update_business or register_business on the network's MCP server, with proof that the domain is yours.
  • To remove it: opt out below. It needs no proof: the crawl stops, everything found on the site is deleted, and an entry nobody claimed goes with it. An entry its owner claimed is hidden while the site is not crawled; it goes for good when its owner takes it out, with its claim token or a proof as strong as its claim.

Products

A member whose inbox lists its products here (its owner switches it off in the inbox) has them found with search_products and get_offers. The network reads the inbox's signed catalogue when the inbox says it changed, checks the signature, the domain and the page before it keeps anything, and each page it read is kept 30 days with its signature, as the record of what the business said. A product is shown only while its business is in the directory and its catalogue was confirmed in the last six hours, and never a stock count: in stock, out of stock or on sale. The same barcode is one product, and a pack of it (an offer of several identical units, its multipack) is another, never in one list with single units; products are never matched by name. Listing products is free, and nothing paid moves one: each offer takes its business's place in the directory's published order. Businesses of a kind the directory does not list list no products.

Claiming and correcting an entry

A business's own AI says "put my business on Surfing Dog" and calls register_business: it declares the business's doors and facts, and proves the domain is its own. A business with no entry yet registers the same way. The proof is one of these, and an entry shows its label, as in proof: domain:

  • domain: a token the network gives, put on the domain: in the file /.well-known/surfingdog-claim, in the inbox manifest's member claims, or in a DNS TXT record at _surfingdog-claim.<domain>;
  • key: the network's challenge, signed by a key the domain publishes (its manifest's receipt keys, or /.well-known/jwks.json);
  • code: eight digits sent to an address at the domain itself (not at a subdomain, and never at a mail provider anyone can sign up to) that the owner types. The network writes to an address when it is typed in that call, and to no other: at most three codes a day, and none for 30 days once one is used. It keeps no copy of the address, just a keyed fingerprint, and never takes an address from a site. A code claim declares no door on another host, cannot turn off an entry found on the site, and cannot be taken over by a code to another address.

A stronger proof (domain, then key, then code) takes over an entry claimed with a weaker one, and the earlier claimant's declaration goes with it. A successful proof gives a claim token, which proves again for 90 days for that entry, to correct it with update_business, turn it off or on, or add and remove doors; a fresh proof ends the earlier tokens of the same proof or a weaker one. A declared door counts once the crawler has checked it, and a door on another host counts only when the site leads to it or the door's own document (its card, profile or manifest) names the business's domain. Taking a claimed entry out needs its claim token or a proof as strong as its claim; opting out below with no proof stops the crawl and hides it.

Checks and the agentic score

Anyone can ask for a site to be checked, at /check or with the check_business tool. A check reads the same pages under the same rules as above, at once rather than in its turn, and at its own pace: someone is waiting for it.

When someone asks us to check a site, we read robots.txt before anything else and obey it, then up to 30 small files, one at a time, at least 250 milliseconds apart on the site's own hosts and at least 10 seconds apart anywhere else. We obey Crawl-delay, Retry-After, 429 and 5xx. A stored result answers for an hour, and a fresh read can be asked for 10 minutes after the last one, or 2 minutes after one that failed for a technical reason.

A site may also be checked again, under the same rules, when a person opens its result page and our last read of it is over a week old, or when an AI agent asks about it with get_business, or its owner opens its inbox sign-up, and the last read is over a day old. These reads take at most half of the day's checks, and never read a site that opted out.

The result is a page at /b/<domain>: whether an AI agent can message the business, book, order, sign a person up, cancel or negotiate there, through which of its doors, and an agentic score by the published rules (/v1/score-rules), worked out from what the site and its doors publish and nothing else, each with its source and date. It is not a certification, it never changes the directory's search order, and nobody can pay for it.

  • Checking a site is not claiming it, and changes nothing about its listing.
  • Anyone with the link can see a result page. A business is named on leaderboards, and its result page may be indexed, once it is agent-ready (askable or above). A local business we found is named once we match its site to its entry on public maps or registers, or its owner proves the site is theirs. Parked, empty or taken-over sites never are. Others are counted, not named. If your site tells AI systems not to use or train on its content, we don't name you or list you publicly. The page of a business that is not named asks search engines not to index it.
  • Its owner can hide the result, its ranks and leaderboard places included: update_business with score_page set to hidden, with the listing's claim token or a proof that the domain is theirs. shown brings it back.
  • A site that opts out below is never checked, and its score, rank and check are deleted with everything else.
  • The badge (/badge/<domain>.svg) is put on a site by its owner, if they want it; this network never places it anywhere.

How to stop it

Add this to your robots.txt:

User-agent: SurfingDogBot
Disallow: /

See how AI crawlers read your robots.txt, each company's by name: the AI crawler checker.

Or opt out here. It needs no proof and takes effect at once: anything found on the site is deleted, its listing too unless its owner claimed it (then it is hidden until the owner decides), and a visit already under way stops within a minute.

For hosting companies: opt out by address range

A hosting company or network operator can keep the crawler away from every site on its servers at once. Write to hello@surfingdog.ai from an address at your company's domain, or from the abuse contact registered for the range, with the address ranges in CIDR notation (an IPv4 or IPv6 network and its prefix length). A person adds them and tells you when it is done. From then on none of our machines sends a request to an address in those ranges: within minutes for a visit already under way, and before any new one. Nothing found on the sites is deleted for this; the site owner can opt out above for that.

Until a range is in, robots.txt (above) or a rule on the SurfingDogBot user agent stops it for any site. Every request it sends is signed (above), so you can tell ours from anyone using the name.

If the crawler caused you a problem, or set off your intrusion detection, write to hello@surfingdog.ai with the time, the domain and what you saw: a person reads every message and answers it.

What it keeps, and your rights

This is the notice under Article 14 of the GDPR, for information not obtained from you.

Who is responsible
The controller is Surfing Dog Lda, which runs the Surfing Dog network. Write to privacy@surfingdog.ai about anything on this page.
Where it comes from
The business's own public website: what the site itself publishes, read as described above; the public registries where the business listed its own agent servers (such as the MCP registry), for the address of its site and its doors; links to its site on other public websites the crawler read, for the address of its site only; and public maps data and business registers (Overture Maps, © Overture Maps Foundation, OpenStreetMap and others), for the address of its site and its place there, which the crawler checks against the site's own pages, and, from rules version 12, the area, kind and name of its one place in Overture Maps' data that its entry may show where its site gives none (What is shown, above). Nothing is bought from anyone.
What is collected
Facts a business has published about itself: its name, address, opening hours, categories, the payment and booking methods it shows, and its machine endpoints; its place in public maps data and business registers, which the crawler checks against the site's own pages (its name, kind, town, postcode and position, and a salted fingerprint of its phone, never the number); and what the network's own reads of those endpoints found (Door history, below). Never personal contact details: no phone numbers, no email addresses, and no names of people other than a business's own name (a sole trader's business may carry theirs). At most it records whether a site offers only human contact.
How a site is built
At each visit it also notes what the pages it read show the site is built with: the website platform or shop software, the booking, payment, chat, analytics and consent tools they load, the content delivery network its answers name, the files for machines it found (such as robots.txt and llms.txt), what its robots.txt says to each AI crawler, and how quickly and securely its homepage answered. This is kept as a dated record for 13 months; counts made from these records, which name no site, are kept for good. For the network's operators only, it also keeps the latest copy of the site's files for machines and of its homepage's markup, without the page's text and with email addresses, phone numbers and messaging links taken out; a homepage that says noarchive is not kept. None of this is published, and opting out deletes all of it at once.
Door history
Each read the network makes of a site's doors (its MCP server, agent card, inbox and the like) is kept 90 days. What those reads found is kept for good, as one line per door per day: how many reads it made, how many answered, failed (and why) or were blocked, how quickly they were answered, the protocol version the door states, and the first and last day a door answered. Beyond the site's own web addresses it holds nothing about a person and no page content, and no agent's report goes into it; nothing is kept of llms.txt files or booking pages. The days a door was seen answering are shown with the business's listing and result, and for a door the network reads every day (a member's, a claimed listing's), from 14 days checked, the share of days every check answered, as a count and a share ("answered 41 of 42 days (97%)"): a day with a failed or a refused check counts as a missed day, and a refused one is shown as refused. An opt-out deletes it at once, with everything else, and nothing more is written.
Why
To help assistants find businesses and know how to reach them, and to show, from the network's own reads, whether a business's doors answer. The legal basis is legitimate interest.
Who sees it
The network's operators; processors that act for the network on its instructions, such as its hosting and the language-model provider described below; and, when the network lists the business, the assistants that ask it and the people using them. It is not sold, and not given to anyone else.
Outside the European Economic Area
A processor may handle some of it outside the European Economic Area. Where one does, the transfer rests on an adequacy decision of the European Commission or on the Commission's standard contractual clauses; write to privacy@surfingdog.ai for a copy of the safeguards.
Read by a language model
When a site's own structured data leaves out its name, what it does or where it is, the text of a few of its own pages (never a contact page, with emails, phone numbers and messaging links taken out beforehand) may be read by a language model that a processor runs for the network, which copies the facts the site states about itself. A fact is kept when the model quotes it word for word from those pages, and dropped otherwise; it is shown marked as probably, and no filter reads it. The text is kept at most 7 days and deleted once read. A site whose robots.txt or Content-Signal says ai-input=no is never read this way.
How long
Until the facts are superseded by a later crawl, or 180 days after they were last seen; an entry found on a site and not checked again within 180 days is deleted with them. A door's history is kept for good (Door history, above). An opt-out deletes all of it at once.
Your rights
You may ask for access to what is kept about your site, have it corrected, have its use restricted, object to it, and have it erased. Opting out above is the quickest way to object and erase at once.
Complaints
You may complain to the data protection authority of the country where you live or work, or where you think the law was broken.
Contact
privacy@surfingdog.ai for your rights; hello@surfingdog.ai for anything else about the crawler.

Listing terms (listing-1)

These are the terms a business agrees to when it registers or claims an entry (agree in register_business). A registration records the version it agreed to. Surfing Dog Lda runs the network.

What the listing is
Being listed is free. The order of the directory follows the published rules at /v1/ranking, which name its main parameters. Nobody can pay to be listed or to move in the order: no payment of any kind, to the network or to anyone else, has any effect on it.
What the business declares
Its facts and doors must be true and its own. The network checks each declared door before it counts, takes only what fits its vocabularies, and says in each answer what it did not take and why.
When an entry is hidden or limited
The network may hide an entry or a part of it, or refuse a registration, when its facts are false or misleading, when a door is not the business's, when the business is of a kind not listed (adult services, gambling, weapons, tobacco, cannabis and vaping, payday loans and bail bonds), when it breaks the law or someone else's rights, or when it is used to abuse the network or other businesses. It gives the reason at the time it acts, in the answer to the business's next register_business or update_business call and as the entry's reason not to be shown.
When a listing ends
The business may end its listing at any time: listing off hides it, and opt_out takes it out. The network gives 30 days' notice, with its reason, before it ends a listing for good, unless the law requires it to act sooner, or the business broke these terms repeatedly or in a way that harms others. The reason is given as above.
Changes to these terms
A change is a new version, published on this page at least 15 days before it takes effect, and named in the listing's answers. A business that does not accept it may end its listing before then.
Complaints
Write to hello@surfingdog.ai with the domain: about a decision on an entry, its order, or these terms. Every complaint is read by a person and answered in plain words within 15 working days. Surfing Dog Lda is a small enterprise, so the duties of Regulation (EU) 2019/1150 to run a complaint system and to name mediators do not apply to it; it answers every complaint all the same, and will agree a mediator with a business when both wish to.
What it keeps
The declaration, the proof's label, a fingerprint of each claim token (never the token), when the business agreed and to which version, and the record of each change. An address typed for a code is kept only as a keyed fingerprint.