Information for webmasters
- What is the spbot and why is it visiting my site?
- How can I block the spbot?
- Why shouldn't I block the spbot?
- Was it really spbot that visited your website?
- spbot visits my website too often.
- Contact information
What is the spbot and why is it visiting my site?
You may be wondering about "spbot" in your weblogs or web statistics and be curious why it is visiting your site. To block spbot from crawling your site, please read below.
What is SEOprofiler.com about?
SEOprofiler.com offers information about the inbound links of websites. It answers the question: Who links to whom on the Internet?
To get this kind of information, SEOprofiler.com uses a web crawler with the name spbot. Basically, it starts with a list of known URLs from across the entire Internet, then it fetches local links found as it goes. There are several advantages to this approach, most importantly that it creates the least possible disruption to the sites being crawled.
We will not index anything you would like to remain private. All you have to do is tell us. How? By using the official Robots exclusion standard. Example:
User-agent: spbot Disallow: /
You can find detailed instructions on how to block web crawlers on Wikipedia and on the robots.txt page.
Make sure that you don't inadvertently block other crawlers that you don't want to block, for example the Googlebot.
Why shouldn't I block the spbot? What's in it for me?
Before you block spbot, consider the following:
- We do not index your web pages, we just want to find out to which web pages you link to.
- We want to help you to better understand how interlinking affects the position of your web pages in the search results of Google, Bing and other search engines.
- We're 100% open in the way we retrieve our data.
- We play 100% by the rules and we follow the robots.txt protocol.
- The spbot is very bandwidth-friendly. It supports gzip compression, and it will hit your server at maximum only once per minute if your site is very, very popular. Most sites will be visited much less often, for example once a week or once a month.
Here's our offer: you let our web crawler find out to which web pages your website links and in return we help you to find the pages that link to your website, how they link to your website and how these links can be improved to get better rankings on Google for your website.
Just enter your domain name in the search box to get an instant basic analysis:
How can I verify spbot is really spbot?
Other web crawlers can spoof the spbot user agent to make them seem legitimate. They seem to be coming from us but they don't. You can verify the IP adresses to make sure that the spbot that visits your site is actually from SEOprofiler.com.
Our IP addresses start with:
- 50.16.*.*
- 50.17.*.*
- 50.19.*.*
- 67.202.*.*
- 72.44.*.*
- 75.101.*.*
- 107.20.*.*
- 174.129.*.*
- 184.73.*.*
- 204.236.*.*
The best way to block us is using the robots.txt file and the user agent name spbot.
spbot visits my website too often
If spbot visits your website too often, please drop us a line. We don't want to annoy webmasters and we don't want to crawl single websites excessively. Just contact us at the email address below and we'll visit your website less often or improve our web crawler.
Feel free to contact us: spbot _AT_ seoprofiler -DOT- com
