RVshiftBot
The program RV Shift uses to keep dealer listings current: prices, what’s still for sale and what has sold.
What it reads
Only the public inventory of dealerships whose listings appear on RV Shift, about once a day: their sitemap or inventory feed, the pages of the units we list, the pages or feed records of in-stock units we may add, and the robots.txt of every site it contacts, photo hosts included. It does not fill in forms, log in, or read anything behind a login.
How it behaves
- It always identifies itself as
RVshiftBot/1.0 (+https://rvshift.com/bot/; inventory sync for dealers listed on RV Shift; contact@rvshift.com). It never presents itself as a web browser. - It reads your
robots.txtfirst and follows it, including anyCrawl-delay. It waits at least 1.5 seconds between requests to one site. - If your site answers 401, 403 or 429, it stops for the day, records it, and does not retry under another name.
- Photos are copied only from sites whose
robots.txtallows it.
How to stop it
Add this to your site’s robots.txt; it is honoured on the next run:
User-agent: RVshiftBot Disallow: /
Or email contact@rvshift.com and we’ll take your listings down.