Campertunitybot

Campertunity is a search engine

Campertunity is the world's largest search engine for campsites. We are building the largest search database of campgrounds in the world, covering every country and every language, so campers can find and compare places to stay wherever they go.

Like any search engine, we index public web pages and send people back to the sites where the information lives. Campertunitybot is the crawler that builds and refreshes that index.

User-agent

Every request identifies itself with this user-agent:

Campertunitybot/1.0 (+https://campertunity.com/bot)

Some sites are web apps that need JavaScript to draw their pages. For those we load the page in a headless Chrome browser, the same way a visitor's browser would. Its user-agent is a standard Chrome string that ends with the same identifier:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 Campertunitybot/1.0 (+https://campertunity.com/bot)

In robots.txt, use the product token Campertunitybot.

What it does

• We fetch campground websites, and the booking and directory sites that list campgrounds, to check and fill in the facts on our campground listings: name, location, amenities, rules, prices and availability.
• We use AI to summarize short excerpts of those pages into listing content. We don't republish whole pages.
• We cite our sources. Listing pages link back to the campground's own website and to the booking and review sites each fact came from, so traffic and bookings go back to the site that provided the information.
• When a campground owner claims their listing and manages it themselves, the listing shows the owner's own details instead.

robots.txt and crawl-delay

• We read your robots.txt before every request, including every redirect and the scripts and data a page loads in our browser. We don't fetch anything it disallows for Campertunitybot. The only exceptions are a few connection types a page opens by itself in our browser, such as WebSockets.
• We honor Crawl-delay. Page crawling, sitemap crawling and photo checks each wait at least that long between the requests they start to your site. When we render a page in our browser, a quick HEAD check is followed straight away by the page itself, and the page's scripts and data load with it, as in a normal browser visit.
• We keep a copy of your robots.txt for up to one hour, so changes take effect within an hour.
• If your robots.txt can't be reached, or returns a server error or 429 Too Many Requests, we don't crawl your site and try again later.
• We don't use pages marked noindex or none, in a robots, googlebot or campertunitybot meta tag, or in an X-Robots-Tag header, either for all crawlers or scoped to us (X-Robots-Tag: campertunitybot: noindex).

Crawl rate

• Page crawling: at most one page per second to each site, or slower if your Crawl-delay asks for it. Rendering a page in our browser is a HEAD check followed straight away by the page, plus the scripts and data it loads.
• Sitemap crawling: reading a sitemap and fetching the pages it lists, at most one request every two seconds to each site. Pages it renders in our browser are a HEAD check followed straight away by the page and the scripts and data it loads. It is paced separately from page crawling, so a site both are working on can briefly see requests closer together.
• Photos: to check that a listing's photos still exist, we send small HEAD requests, at most 6 at a time to each site, including sites a photo link redirects to. These follow robots.txt and honor a Crawl-delay of up to 2 seconds. If your Crawl-delay is longer, we don't check photos on your site at all.
• All crawling runs from a single server. Our crawler software refuses to fetch sites from anywhere else.

How to block or slow us down

To block Campertunitybot from your whole site, add to robots.txt:

User-agent: Campertunitybot
Disallow: /

To slow it down, for example to wait 10 seconds between our requests:

User-agent: Campertunitybot
Crawl-delay: 10

To keep a single page out of Campertunity, add to that page:

<meta name="campertunitybot" content="noindex">

You can also block or rate-limit our user-agent or our IP address (below) in your firewall, or email us and we'll add your domain to our block list.

IP address

Campertunitybot crawls only from this IP address. Requests using our user-agent from any other address are not from us.

34.182.173.94

Machine-readable list: campertunity.com/bot/ips.json

Share your data directly

Crawling is not the only way to get your campgrounds into Campertunity. If you run a campground, a booking platform or a directory and would rather give us API access to your data, or send us a feed or export, we can list your campgrounds more accurately, cover more of them, and keep prices and availability in sync in real time.

Partners' campgrounds show a "Book on" button that sends campers to their site. Read how it works and tell us about your platform on our partner page.

Contact

Problems with our crawler, or want us to stop crawling your site? Email us:

[email protected]

Please include your domain name. You can also use our contact form. To share your data with us instead, see the partner page.