Information for webmasters
- What is the spbot and why is it visiting my site?
- How can I block the spbot?
- Why shouldn't I block the spbot?
- Was it really spbot that visited your website?
- spbot visits my website too often.
- Contact information
What is the spbot and why is it visiting my site?
You may be wondering about "spbot" in your weblogs or web statistics and be curious why it is visiting your site. To block spbot from crawling your site, please read below.
What is SEOprofiler.com about?
SEOprofiler.com offers information about the inbound links of websites. It answers the question: Who links to whom on the Internet?
To get this kind of information, SEOprofiler.com uses a web crawler with the name spbot. Basically, it starts with a list of known URLs from across the entire Internet, then it fetches local links found as it goes. There are several advantages to this approach, most importantly that it creates the least possible disruption to the sites being crawled.
Take the tour to learn more about SEOprofiler.
We will not index anything you would like to remain private. All you have to do is tell us. How? By using the official Robots exclusion standard. Example:
User-agent: spbot Disallow: /
You can find detailed instructions on how to block web crawlers on Wikipedia and on the robots.txt page.
Make sure that you don't inadvertently block other crawlers that you don't want to block, for example the Googlebot.
Why shouldn't I block the spbot? What's in it for me?
Before you block spbot, consider the following:
- We do not index your web pages, we just want to find out to which web pages you link to.
- We want to help you to better understand how interlinking affects the position of your web pages in the search results of Google, Bing and other search engines.
- You get something in return. You will get a detailed report of the websites that link to your website (and how they link to your website). The report will be much more detailed than the information that you get from Google, Yahoo or other sources. Our service will start in Q3/2010.
- We're 100% open in the way we retrieve our data.
- We play 100% by the rules and we follow the robots.txt protocol.
- The spbot is very bandwidth-friendly. It supports gzip compression, and it will visit only 4 to 16 pages of your domain. Then it will wait a long time before visiting your domain again. For most domains, it will take weeks or months until spbot visits your domain again.
Here's our offer: you let our web crawler find out to which web pages your website links and in return we help you to find the pages that link to your website, how they link to your website and how these links can be improved to get better rankings on Google for your website.
Just enter your domain name in the search box to get an instant analysis:
How can I verify spbot is really spbot?
Other web crawlers can spoof the spbot user agent to make them seem legitimate. They seem to be coming from us but they don't. You can verify the IP adresses to make sure that the spbot that visits your site is actually from SEOprofiler.com.
Our IP addresses start with:
- 67.202.*.*
- 72.44.*.*
- 75.101.*.*
- 174.129.*.*
- 204.236.*.*
- 184.73.*.*
The best way to block us is using the robots.txt file and the user agent name spbot.
spbot visits my website too often
If spbot visits your website too often, please drop us a line. We don't want to annoy webmasters and we don't want to crawl single websites excessively. Our bot is relatively new and it might be that some websites or robots.txt files are visited very often if they have many inbound links from some sites.
In that case, contact us at the email address below and we'll add your website to our "fewer visits" list and we'll improve our bot software. spbot should not access your website too often and it should be beneficial to your business.
Feel free to contact us: at spbot _AT_ seoprofiler -DOT- com
