GSEC · Question #10
What file instructs programs like Web spiders NOT to search certain areas of a site?
The correct answer is A. Robots.txt. The robots.txt file is the universally accepted standard for instructing web crawlers and spiders which parts of a website they should not index or visit. It is placed in the root directory of a web server.
Question
What file instructs programs like Web spiders NOT to search certain areas of a site?
Options
- ARobots.txt
- BRestricted.txt
- CSpider.txt
- DSearch.txt
How the community answered
(17 responses)- A82% (14)
- B6% (1)
- C12% (2)
Why each option
The robots.txt file is the universally accepted standard for instructing web crawlers and spiders which parts of a website they should not index or visit. It is placed in the root directory of a web server.
Robots.txt is the Robots Exclusion Protocol standard, defined to allow site owners to communicate crawling restrictions to compliant web spiders and bots. It is placed at the root of a domain (e.g., example.com/robots.txt) and uses Disallow directives to specify restricted paths. All major search engine crawlers honor this file by convention.
Restricted.txt is not a recognized standard file name for web crawler exclusion and would be ignored by web spiders.
Spider.txt is not a defined protocol or standard recognized by any major web crawler or search engine.
Search.txt is not a recognized convention for controlling web spider behavior and has no defined specification.
Concept tested: Robots Exclusion Protocol and robots.txt usage
Source: https://developers.google.com/search/docs/crawling-indexing/robots/intro
Topics
Community Discussion
No community discussion yet for this question.