nerdexam
GIAC

GSEC · Question #10

What file instructs programs like Web spiders NOT to search certain areas of a site?

The correct answer is A. Robots.txt. The robots.txt file is the universally accepted standard for instructing web crawlers and spiders which parts of a website they should not index or visit. It is placed in the root directory of a web server.

Penetration Testing Foundations & Reconnaissance

Question

What file instructs programs like Web spiders NOT to search certain areas of a site?

Options

  • ARobots.txt
  • BRestricted.txt
  • CSpider.txt
  • DSearch.txt

How the community answered

(17 responses)
  • A
    82% (14)
  • B
    6% (1)
  • C
    12% (2)

Why each option

The robots.txt file is the universally accepted standard for instructing web crawlers and spiders which parts of a website they should not index or visit. It is placed in the root directory of a web server.

ARobots.txtCorrect

Robots.txt is the Robots Exclusion Protocol standard, defined to allow site owners to communicate crawling restrictions to compliant web spiders and bots. It is placed at the root of a domain (e.g., example.com/robots.txt) and uses Disallow directives to specify restricted paths. All major search engine crawlers honor this file by convention.

BRestricted.txt

Restricted.txt is not a recognized standard file name for web crawler exclusion and would be ignored by web spiders.

CSpider.txt

Spider.txt is not a defined protocol or standard recognized by any major web crawler or search engine.

DSearch.txt

Search.txt is not a recognized convention for controlling web spider behavior and has no defined specification.

Concept tested: Robots Exclusion Protocol and robots.txt usage

Source: https://developers.google.com/search/docs/crawling-indexing/robots/intro

Topics

#robots.txt#web crawling#OSINT#passive reconnaissance

Community Discussion

No community discussion yet for this question.

Full GSEC Practice