# # robots.txt for http://www.alterno.dk/ # #To exclude a single robot #User-agent: BadBot #Disallow: / #To allow a single robot #User-agent: WebCrawler #Disallow: #User-agent: * #Disallow: / # AltaVista Search #User-agent: AltaVista Intranet V2.0 W3C Webreq #Disallow: /Out-Of-Date #To exclude all files except one #This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "docs", and leave the one file in the level above this directory: #User-agent: * #Disallow: /~joe/docs/ #Alternatively you can explicitly disallow all disallowed pages: #User-agent: * #Disallow: /~joe/private.html #Disallow: /~joe/foo.html #Disallow: /~joe/bar.html # exclude some access-controlled areas User-Agent: Googlebot-Image #Disallow: / User-agent: * Disallow: /admin/ Disallow: /edytor/ Disallow: /Connections/ Disallow: /UsersFiles/ Disallow: /Profil/ Disallow: /Kontakt/ Disallow: /annoncering/ Disallow: /Extranet/ Disallow: /moduler/ Disallow: /images/ Disallow: /setup/ #Disallow: /Out-Of-Date