SEOprofiler.com HomeJoin for freeLogin

Information for webmasters


What is the spbot and why is it visiting my site?

You may be wondering about "spbot" in your weblogs or web statistics and be curious why it is visiting your site. To block spbot from crawling your site, please read below.

What is SEOprofiler.com about?

SEOprofiler.com offers information about the inbound links of websites. It answers the question: Who links to whom on the Internet?

To get this kind of information, SEOprofiler.com uses a web crawler with the name spbot. Basically, it starts with a list of known URLs from across the entire Internet, then it fetches local links found as it goes. There are several advantages to this approach, most importantly that it creates the least possible disruption to the sites being crawled.

How can I block the spbot?

We will not index anything you would like to remain private. All you have to do is tell us. How? By using the official Robots exclusion standard. Example:

User-agent: spbot
Disallow: /

You can find detailed instructions on how to block web crawlers on Wikipedia and on the robots.txt page.

Make sure that you don't inadvertently block other crawlers that you don't want to block, for example the Googlebot.

Why shouldn't I block the spbot? What's in it for me?

Before you block spbot, consider the following:

Here's our offer: you let our web crawler find out to which web pages your website links and in return we help you to find the pages that link to your website, how they link to your website and how these links can be improved to get better rankings on Google for your website.

Just enter your domain name in the search box to get an instant basic analysis:

 

How can I verify spbot is really spbot?

Other web crawlers can spoof the spbot user agent to make them seem legitimate. They seem to be coming from us but they don't. You can verify the IP adresses to make sure that the spbot that visits your site is actually from SEOprofiler.com.

Our IP addresses start with:

The best way to block us is using the robots.txt file and the user agent name spbot.

spbot visits my website too often

If spbot visits your website too often, please drop us a line. We don't want to annoy webmasters and we don't want to crawl single websites excessively. Just contact us at the email address below and we'll visit your website less often or improve our web crawler.

Contact information

Feel free to contact us: spbot _AT_ seoprofiler -DOT- com