To see the robot.txt file, try accessing the url:
domainname/robots.txt
For example, http://www.netbeans.org/robots.txt