r7rrr@r@9s+ J N M MD ' (CCCrr@c(eZdZdZddgZdgZddgZdS) SitemapTesta# robots.txt for http://www.example.com/ User-agent: * Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml Sitemap: http://www.google.com/hostednews/sitemap_index.xml Request-rate: 3/15 Disallow: /cyberworld/map/ # This is an infinite virtual URL space r:r;r<z7http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xmlz2http://www.google.com/hostednews/sitemap_index.xmlN)r4r5r6rr%r,r1r7rrrCrCJs7 J  D ' (CJEGIIIrrCceZdZdZgZgdZdS)RejectAllRobotsTestz(# go away User-agent: * Disallow: / )r<r:/tmp/Nr>r7rrrErE[s'J D 6 6 6CCCrrEceZdZdZdZdZdS)BaseRequestRateTestNcr|j}|j|jzD]}||\}}|||5||||j||}|||j|jo||tj j ||j |jj ||j |jj dddn #1swxYwY dSr$)rr%r,r"r&r0 crawl_delay request_rateassertIsInstancerr RequestRaterequestsseconds)rrr!rparsed_request_rates rtest_request_ratez%BaseRequestRateTest.test_request_rateis9tx'  C//44JE3#U33    !3!3E!:!:Dr7rrrdrds#J =D CCCrrdc"eZdZdZdZdgZdgZdS)AnotherInvalidRequestRateTestzeUser-agent: Googlebot Allow: /folder1/myfile.html Disallow: /folder1/ Request-rate: whale/banana Googlebot/folder1/myfile.html/folder1/anotherfile.htmlNr4r5r6rrr%r,r7rrrfrfs,J E " #D & 'CCCrrfceZdZdZdZdgZdS)UserAgentOrderingTestzMUser-agent: Googlebot Disallow: / User-agent: Googlebot-Mobile Allow: / rgz/something.jpgN)r4r5r6rrr,r7rrrlrls$J E  CCCrrlceZdZdZdS)UserAgentGoogleMobileTestzGooglebot-MobileNr`r7rrrnrns EEErrnc"eZdZdZdZdgZdgZdS)GoogleURLOrderingTestzJUser-agent: Googlebot Allow: /folder1/myfile.html Disallow: /folder1/ googlebotrhriNrjr7rrrprps,J E " #D & 'CCCrrpceZdZdZdgZdgZdS)DisallowQueryStringTestz2User-agent: * Disallow: /some/path?name=value /some/pathz/some/path?name=valueNr>r7rrrsrss&J >D " #CCCrrsceZdZdZdgZdgZdS)UseFirstUserAgentWildcardTestzNUser-agent: * Disallow: /some/path User-agent: * Disallow: /another/path z /another/pathrtNr>r7rrrvrvs&J  D .CCCrrvceZdZdZdgZdgZdS)EmptyQueryStringTestz>User-agent: * Allow: /some/path? Disallow: /another/path? z /some/path?z/another/path?Nr>r7rrrxrxs&J ?D  CCCrrxcZeZdZdZejddZdZddgZ dgZ dS) DefaultEntryTestzOUser-agent: * Crawl-delay: 1 Request-rate: 3/15 Disallow: /cyberworld/map/ rYr:r;r<N) r4r5r6rrrrMrKrJr%r,r7rrrzrzsFJ %11!R88LK  D ' (CCCrrzceZdZdZdZdZdS)StringFormattingTestzUser-agent: * Crawl-delay: 1 Request-rate: 3/15 Disallow: /cyberworld/map/ # This is an infinite virtual URL space # Cybermapper knows where to go. User-agent: cybermapper Disallow: /some/path zxUser-agent: cybermapper Disallow: /some/path User-agent: * Crawl-delay: 1 Request-rate: 3/15 Disallow: /cyberworld/map/c`|t|j|jdSr )r0strrexpected_outputr2s rtest_string_formattingz+StringFormattingTest.test_string_formatting*s+ T[))4+?@@@@@rN)r4r5r6rrrr7rrr~r~s6 JOAAAAArr~ceZdZdZdZdS) RobotHandlerc2|dddS)NizForbidden access) send_errorr2s rdo_GETzRobotHandler.do_GET0s /00000rcdSr r7)rformatargss r log_messagezRobotHandler.log_message3s rN)r4r5r6rrr7rrrr.s2111     rrz&Socket server requires working socket.c:eZdZdZdZejdZdS)PasswordProtectedSiteTestCasec4|tjjt t jdft|_tj d|jj ddi|_ d|j _ |j dS)NrzHTTPServer serving poll_intervalg{Gz?)nametargetkwargsT) addCleanuprrequest urlcleanuprrHOSTrserver threadingThread serve_forevertdaemonstartr2s rrz#PasswordProtectedSiteTestCase.setUp=s~ 1222 -"4a!8,GG !%;,$D) +++   rc|j|j|jdSr )rshutdownrjoin server_closer2s rtearDownz&PasswordProtectedSiteTestCase.tearDownMs?      """""rcX|jj}dtjzdzt |dz}|dz}t j}||| | | d|dS)Nzhttp://:r|z /robots.txt*) rserver_addressrrrrrrset_urlreadr-r()raddrr! robots_urlrs rtestPasswordProtectedSitez7PasswordProtectedSiteTestCase.testPasswordProtectedSiteRs{)-,,s2Sa\\A=( #3355s  ))#z::;;;;;rN)r4r5r6rrr reap_threadsrr7rrrr7sQ  ### "<<#"<<>BBcd|j|tj|dsdndS)Nz{}{}{}r|r:r )rrospathsplitext)rrs rr!zNetworkTestCase.urljs? M4BG,<,rs. &&&&&&))))))::::::::BBBBBBBBDBBBBBM8+<BBB)))))=(2C)))"GGGGG-!2GGG"77777-):777-8'): .....#68I...$-----5---     ]H,=         M8+<     ( ( ( ( (M83D ( ( (     M8+<    5 ( ( ( ( (M8+< ( ( ($$$$$mX->$$$     M83D   =(*; ) ) ) ) )*H,= ) ) )AAAAA=(*;AAA4     )    ,<<<<