just a curiosity, i havn't been working on my comic for a few months, i've come back, begun to switch everything to the new comicgenesis server, and was just wondering what the Robots.txt file is which appears in my /public_html/ folder...
i didn't see any other post relating to it, so i thought i'd throw out the question and see who's listening...
robots.txt
-
Greenlightgo
- Newbie
- Posts: 14
- Joined: Tue Jun 01, 2004 3:56 pm
- Location: somewhere east of you
- Contact:
-
STrRedWolf
- Confuzzled CG Admin
- Posts: 2580
- Joined: Fri Jan 01, 1999 4:00 pm
- Location: undef;
- Contact:
robots.txt is a file which tells (most*) web search engines what can and cannot be indexed. The basics do not index the comic archives nor the images. This cut down on bandwidth usage, especially due to Google and Yahoo.
(* Most -- some search engines and "spiders" don't respect the Robot Exclusion standard. If we find them abusing it, we'll ban the search engine from the servers.)
(* Most -- some search engines and "spiders" don't respect the Robot Exclusion standard. If we find them abusing it, we'll ban the search engine from the servers.)