What is Enable robots exclusion?

If enabled, the search indexer will respect the exclusions in the robots.txt file. This option is applicable to the Collabnet platform indexer service.