Apache robots.txt 2007-07-11 - By Karl Latiss
Back
On Wed, 2007-07-11 at 02:52 +0000, debu wrote:
> Hi Group,
>
> I was just trying to find all the tags that can be used inside apache robots
.txt, any link, or any addition to the following?
>
> i have these,
>
> User-agent:
> Disallow:
> Crawl-delay:
> Request-rate:
> Visit-time:
>
> Thanks
> Debajit kataki
According to http://www.robotstxt.org/wc/norobots.html only User-agent
and Disallow are valid. All other entries are user-agent specific (see
for eg http://www.ilovejackdaniels.com/seo/robots-txt-file/3/)
--
Karl Latiss <karl.latiss@(protected)>
Atvert Systems
--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@(protected)?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list
|
|
