Apache robots.txt 2007-07-11 - By Waleed Harbi
Back
http://www.redhat.com/robots.txt
On 7/11/07, Karl Latiss <karl.latiss@(protected)> wrote:
>
> On Wed, 2007-07-11 at 02:52 +0000, debu wrote:
> > Hi Group,
> >
> > I was just trying to find all the tags that can be used inside apache
> robots.txt, any link, or any addition to the following?
> >
> > i have these,
> >
> > User-agent:
> > Disallow:
> > Crawl-delay:
> > Request-rate:
> > Visit-time:
> >
> > Thanks
> > Debajit kataki
>
> According to http://www.robotstxt.org/wc/norobots.html only User-agent
> and Disallow are valid. All other entries are user-agent specific (see
> for eg http://www.ilovejackdaniels.com/seo/robots-txt-file/3/)
>
> --
> Karl Latiss <karl.latiss@(protected)>
> Atvert Systems
>
> --
> redhat-list mailing list
> unsubscribe mailto:redhat-list-request@(protected)?subject=unsubscribe
> https://www.redhat.com/mailman/listinfo/redhat-list
>
--
:. Best Wishes
:. Waleed Harbi
-- ---- ---- ---- ---- ---- ---- ---- ---- --
:. Never too old to learn
:. Every why has a wherefore
:. Grasp all, lose all
--
redhat-list mailing list
unsubscribe mailto:redhat-list-request@(protected)?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/redhat-list
|
|
