# # robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources. # # This file will be ignored unless it is at the root of your host: # Used: http://example.com/robots.txt # Ignored: http://example.com/site/robots.txt # # For more information about the robots.txt standard, see: # http://www.robotstxt.org/wc/robots.html # # For syntax checking, see: # http://www.sxw.org.uk/computing/robots/check.html # # This robots.txt is to exclude everything on UrbanSermons.org except the pages # under the /f/ path. # It should be served via an .htaccess rewrite rule. # See g1smd's post on http://www.webmasterworld.com/robots_txt/3931680.htm. User-agent: * Crawl-delay: 20 Disallow: /a Disallow: /b Disallow: /c Disallow: /d Disallow: /e Disallow: /fa Disallow: /fb Disallow: /fc Disallow: /fd Disallow: /fe Disallow: /ff Disallow: /fg Disallow: /fh Disallow: /fi Disallow: /fj Disallow: /fk Disallow: /fl Disallow: /fm Disallow: /fn Disallow: /fo Disallow: /fp Disallow: /fq Disallow: /fr Disallow: /fs Disallow: /ft Disallow: /fu Disallow: /fv Disallow: /fw Disallow: /fx Disallow: /fy Disallow: /fz Disallow: /f0 Disallow: /f1 Disallow: /f2 Disallow: /f3 Disallow: /f4 Disallow: /f5 Disallow: /f6 Disallow: /f7 Disallow: /f8 Disallow: /f9 Disallow: /g Disallow: /h Disallow: /i Disallow: /j Disallow: /k Disallow: /l Disallow: /m Disallow: /n Disallow: /o Disallow: /p Disallow: /q Disallow: /r Disallow: /s Disallow: /t Disallow: /u Disallow: /v Disallow: /w Disallow: /x Disallow: /y Disallow: /z Disallow: /0 Disallow: /1 Disallow: /2 Disallow: /3 Disallow: /4 Disallow: /5 Disallow: /6 Disallow: /7 Disallow: /8 Disallow: /9