TXT File o' Mysteries
Posted: Thu Jun 24, 2004 8:11 pm
Just out of curiosity, can anyone tell me what this is? It showed up randomly in my webalizer logs and I am intrigued by it.
it's the little file that tells webbots where they can go and where they can't go in your directory structure. search engines like google use these automated programs to search through and list your website on the engine. the file tells them what pages you want listed on the search engine, and what you don't.
that's about it