cst1/openstreetmap-website
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

New robots file from Firefishy.

Browse source
This commit is contained in:
Tom Hughes 2010-02-26 19:02:17 +00:00
parent e19be03313
commit 42b0dd52fd
1 changed files with 63 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

65
public/robots.txt
View file

@ -1,6 +1,67 @@
Sitemap: http://www.openstreetmap.org/sitemap.xml.gz
User-agent: *
Disallow: /user/
Disallow: /traces/tag/
Disallow: /traces/page/
Disallow: /api/
Disallow: /browse/
Disallow: /trace/
Disallow: /edit
Allow: /browse/node/
Allow: /browse/way/
Allow: /browse/relation/
Disallow: /browse
Disallow: /login
Disallow: /geocoder
Disallow: /history
Disallow: /message
Disallow: /trace/
Disallow: /*lat=
Disallow: /*node=
Disallow: /*way=
Disallow: /*relation=
User-agent: Slurp
Disallow: /user/*/traces/
Disallow: /user/*/edits/
Disallow: /user/*/diary/
Allow: /user/
Disallow: /traces/tag/
Disallow: /traces/page/
Disallow: /api/
Disallow: /edit
Allow: /browse/node/
Allow: /browse/way/
Allow: /browse/relation/
Disallow: /browse
Disallow: /login
Disallow: /geocoder
Disallow: /history
Disallow: /message
Disallow: /trace/
Disallow: /*lat=
Disallow: /*node=
Disallow: /*way=
Disallow: /*relation=
User-agent: Googlebot
Disallow: /user/*/traces/
Disallow: /user/*/edits/
Disallow: /user/*/diary/
Allow: /user/
Disallow: /traces/tag/
Disallow: /traces/page/
Disallow: /api/
Disallow: /edit
Allow: /browse/node/
Allow: /browse/way/
Allow: /browse/relation/
Disallow: /browse
Disallow: /login
Disallow: /geocoder
Disallow: /history
Disallow: /message
Disallow: /trace/
Disallow: /*lat=
Disallow: /*node=
Disallow: /*way=
Disallow: /*relation=