A standard for Robot Exclusion
| Date written: | 1994 |
| Authors: | Martijn Koster |
| Resource type: | article |
| Links: | [html] |
Description
This article shows how to control the behavior of webcrawlers visiting a site.
DB information
| Date created: | 0000-00-00 00:00:00 |
| Date evaluated: | 0000-00-00 00:00:00 |
| Categories: | System administration/Setup |