As a veteran in baidu seo, I can say that baidu spider spider is super lazy, and hard to dealing with. To end this mad spider, I will show you how to block baidu spiders.
Most Chinese SEOers feel baidu spider is crazy because they normal won’t follow the rules you made. For example, spiders should crawl the pages following the rules you build in search engine webmaster (such as sitemap, URL submission etc.) When you go throw server log, you can see that baidu spiders crawl you site like they are in the zoo. Only crawl what they like to crawl, and ignore all other pages which you think is important.
Let’s take look how to use robots.txt to stop baidu spiders.
Put code above into your site’s robots.txt file, will prevent baidu web spider crawling the site.
Please do notice that, I said Baidu WEB spider. Not all baidu spiders.
How many kinds of spiders do they have? Let’s take a look of the list below.
- Baiduspider (for web pages)
- Baiduspider-image (for images)
- Baiduspider-video (for videos)
- Baiduspider-news (for news)
- Baiduspider-favo (baidu favorites)
- Baiduspider-cpro (baidu adsense)
- Baiduspider-sfkr (baidu sem spider)
If you want to Block Baidu, you should block them all like this