How does robots.txt work?

Search engines have two main jobs:

  1. Crawling the web to discover content;
  2. Indexing that content so that it can be served up to searchers who are looking for information.

Leave a Reply

Your email address will not be published. Required fields are marked *