蜘蛛百度蜘蛛（baiduspider）

百度蜘蛛baiduspider

Baidu spider, Baidu spider, English name is "Baiduspider", isa Baidu search engine automatic program. Its function is toaccess HTML pages on the Internet and build index databases sothat users can search the pages of your web site in Baidu searchengines.

Common problem

How is the access pressure caused by 1.Baiduspider to a webserver?

Answer: Baiduspider automatically regulates access densitybased on the server' s load capacity. After continuous accessfor a period of time, Baiduspider will pause for a while toprevent the access pressure of the server from increasing. So,in general, Baiduspider does not cause too much pressure on theserver on your site.

2. why does Baiduspider keep grabbing my website?

Answer: Baiduspider will continue to crawl on new orcontinuously updated pages on your site. In addition, you canalso checkwhether the access toBaiduspider in the site accesslog is normal, so as to prevent anyone from pretending to beBaiduspider to grab your website frequently. If you findBaiduspider not normal to crawl your website, please feedbackto webmaster@baidu. com, and please try to give Baiduspideraccess log to your station so that we can track processing.

3. , I don't want my website to be accessed by Baiduspider. What

should I do?

Answer: Baiduspider comply with internet robots protocol. Youcan use robots.txt files to completely ban Baiduspider fromaccessing your web site or to prohibit Baiduspider fromaccessing some of the files on your web site. Note: theprohibition of Baiduspider access to your web site will enablepages on your web site to be searched in Baidu search enginesand all Baidu search engines providing search engine services.Ps: about robots.txt' s writing methods, please see ourintroduction: robots.txt writing method

4. why my website has added robots.txt, but also in Baidu searchout?

Answer: because search engine index database update takes time.Although Baiduspider has stopped accessing web pages on yoursite, it may take two to four weeks before the Baidu searchengine database has been established. Also check to see if yourrobots configuration is correct.

5. , I want my website content to be indexed by Baidu but notsaved by snapshot. What should I do?

A: Baiduspider follows the Internet meta robots protocol. Youcanuse the settings of theweb page meta so that Baidu displaysthe index only for the page, but does not display snapshots ofthe page in the search results.

And update the robots, because the search engine index database

update takes time, so although you have a web page through themeta banned Baidu snapshot of the web page displayed in thesearch results, but Baidu search engine database has beenestablished if the page index information, may need two weeksto be effective online.

6. what' s the name of the Baidu spider in robots.txt?Answer: "Baiduspider" initial B uppercase, and the rest islowercase.

7.Baiduspider how long will it take to grab my page again?Answer: Baidu search engine updated every week, web pagesdepending on the importance of different update rate, frequencyin a few days to a month, Baiduspider will revisit and updatea web page.

The bandwidth jam caused by 8.Baiduspider capture?

A: Baiduspider' s normal crawl does not cause congestion on yoursite' s bandwidth. This may be due to someone posing as Baidu' sspider malicious grab. If you find the agent grab known asBaiduspider and cause bandwidth jam, please contact us as soonas possible. You can feed the information back to the Baidu webcomplaint center, and if you can provide your site, the accesslogs for this time period will be more conducive to ouranalysis.

-----------------------------------------------------------

---

什么是百度蜘蛛

悬赏分 0解决时间 2009年3月15日21 :24

百度爬虫是什么怎么工作的

提问者 四条-一级最佳答案第一百度蜘蛛极为活跃经常看看你的服务器日志你就怀发现百度蜘蛛抓取的频率和数量都非常大。百度蜘蛛几乎每天都会访问我的论坛并且至少抓取几十个网页。我的论坛只开通了不到一个月网页数目还没有完善但是百度蜘蛛的活动已经相当可观了。大量捕获是百度的强项其他任何搜索引擎都没办法相比。但是百度中文网页数目并不是最大的百度蜘蛛抓取的频率和网页更新情况有关。天天更新的网站一定会吸引百度蜘蛛更频繁的访问我有一个非常明显的例子 www.ao l inda. com这个域名比较

老注册已经快一年了开始做了一个学习站感觉更新比较麻烦而且也没有很多时间去维护但是这个学习站是关于电脑方面的虽然内容不多但是页面却不下两W是别人的整站源码-第一天几个好朋友光顾了一下 9ip没想到

第二天早上打开网站居然发现从百度来了100多IP 奇迹百度蜘蛛就有这么神气地点 www.aol inda. com查一下晕了一晚上时间被收录了2000多页 

应该说这个学习站继续做下去有点前途但是我时间还真不够用所以K掉了这个学习站用这个域名做了一个笑话站有留言也有网友上传轻松多了不过这下被收录的页面全部是死链要从头开始了吧但是我又错了第三天这个笑话站又被全面抓取了     -我发现百度对天天更新的站最敏感 彻底换内容更敏感--哈哈看来这个机器人也是喜新厌旧的家伙啊

最近还是因为时间不够又用这个域名改了论坛不知道还有没有奇迹出现–我相信只要内容够多百度蜘蛛也贪你站的内容如果不达到么个数目它可能懒得理你具体多少好象是百度内部机密哈哈

第二我注意了一下蜘蛛似乎更注重页面内的因素。与谷歌更加重视内部有点爬虫类的味道越黑越深它越是喜欢往里钻 –不相信你做100个页面做得再漂亮只要链接没有层次哈哈不好意思你最多就孤零零的被收录可怜的一点点东西。我前两个站开通不到一个月也很少有外部链接但因为本身的结构是比较有层次一些竞争不太激烈的关键词在百度的排名还不错。

第三要想排名靠前 目标关键词应该完整匹配地出现在页面中。比如说你想让你的网站在用户搜索”电脑学习”时出现在前面那么在你的网页上 “电脑学习”这四个字应该完整连续的出现而不能”电脑”出现在第一段 “学习”出现在第二段。

第四百度排名算法是以网页为基础 比较少关注整个网站的主题。联系到上一点这说明百度排名算法中比较注重内部结构缺少完整的语义分析。所以一些目前比较认同的关于网站之间那几个所谓关系到搜索质量的东西并不是百度蜘蛛所最敏感的

第五百度并不被所谓的优化迷惑  GG对优化好象远远没有百度敏感百度尤其反感所谓的优化不知道是用什么方法识别--我的看法是目前最”先进”的优化方法

Baidu seems to not what a big role, so we are doing, the robotis a little brain dead, but the Baidu IT is not to eat plainwhite rice Kazakhstan, to know that he is the world' s mostadvanced Chinese search, GG search, Chinese in this fast - haha, not say it) : no more than!

Sixth: make full use of one of the biggest advantages of Baidu- you may think it' s advantage for us is the difficult thing:Oh, really is available, Baidu included speed can be used todescribe the mass, because of speed, it gives us the space thatcan be used! -back to the optimization:) -while Baidu doesn'tget cold about optimization, it can still work out well ifyou're friendly in your approach-I agree with the right amountof optimization! As far as the optimization is concerned, whatis the best? I can't say 1, 2, 3, either. Oh, but don't forget,because Baidu included too fast, we can often use differentmethods to test the effect, but also to Baidu spider every dayyougive it to playnewtricks, oh, it seems that thismysteriousthing is a little childish Kazakhstan, need someone to lead,love Coucourenao - it seems there is a benefit, if you neverbother to play tricks Station - ha ha, it is very likely thatday spider no longer patronize your site, why?Did K drop it?!- the Baidu spider has a frog' s eye, and the moving object canbe seen far away, and with special attention, the quiet objectmay not be visible around it!

----------------------------------------------

How to query Baidu spider crawling!

Reward points: 5 - solve the time: 2010-1-7 14:21

How can I know?! Baidu spider is to his web page?!

How to search Baidu spider crawling traces?!

Question: kdkj888 - two best answer, now Baidu spider robot isno longer the previous robot, looks smarter, crawling is moreflexible, and today we will use examples to talk to you. First,explosive crawling, I wonder if Baidu spiders like highefficiency crawling, and sometimes Baidu spider can crawlhundreds of times in one or two minutes. I like the station,basically every day will be Baidu spider crawling out severaltimes, at 6 o'clock in the morningonce about crawling 300 times;at 9 o'clock in themorningwhen one is crawling 300 times; therewas also a 13, but a little less, only 200 times; I have time18, about crawling 400 times, also have a 23, only about 250times. Sometimes, when I look at specific crawling records,these explosive reptiles don' t last more than five minutes. Onone occasion, I do not know what the station will be, Baiduspider crawling in two minutes more than 1800 times, I was alittle puzzled, Baidu spider robot computing speed is reallyamazing. But now I basically know what will happen, because thespider crawling on it, after a period of time, the spider tosee whether it is the original operation procedures included,whether what is original, whether it should be included. Two,confirm the crawling crawling way also confirmed that Baidu inlate September began the trial, then what is the confirmationof crawling, refers to your website to update a content afterthe first time Baidu will not give you crawl after the releaseincluded, Baidu spider also conducted second times incomparison in computing, crawling. If you think this isnecessary to update the content included, Baidu spider will bethe third time crawling, under normal circumstances, there willnot be a fourth Baidu spider crawling. After the thirdconf irmation, Baidu spider will slowly to you release included.This confirmation crawl is a bit like crawling with Google.

Baidu spider crawling robot home page or the same, do not knowhow many times a day to crawl home page, other pages, if Baiduthink it is necessary to carry out the calculation, it will besecond times to confirm the crawl. Like my station,

I update every day content, as long as Baidu spider, robotcrawling three times, basically will release included. Thosewho crawled two times would not be released. I haven' t seen itfor four times. Three, stable crawl, stable crawl, refers to24 hours every day, every hour of crawling is not big difference.Stable crawling often appear to the railway station only, forBaidu to think you station is mature, if appear this way youcan crawl, we must be careful, this way you crawl, station willprobably be right down. Second days will be able to see out,the home page snapshot date, will not give you update. Forexample, my station aabc.cn, the amount of crawl in each hour,is almost the same from the chart. Therefore, this station' shome page basically does not appear 24 hours snapshot. Everyday I update the content, will include some. For example, aperson doing anything, without passion, there will be noexplosive force, of course, will not work hard, do not work hard,you say how good results will be. The above said so many, youmay have doubt, Baidu spider to no, how do I know, this is verysimple, you can check the server log records. If you can't checkthe log book, see if there is a record of spider crawling inthe website background. We recommend a dew source CMS the sourcesite background can clearly record the traces of eachbig searchrobot, each robot visiting time, visiting the page to visit thespecific data is analyzed, analyze the 24 hour time period,analysis of each channel, the content for you the analysissection. For each big search robot, like your website which

channel, which section of the analysis, but also to you putforward the remedy of other channels and the suggestion of thesection, which time, add content included fastest, etc. . Insummary, Baidu spider crawling rules for each site is not thesame, only the comparison and analysis of our own seriously,in order to summarize the update site more perfect way, onlywe grasp some rules of Baidu spider, we can put some updates.