nerdexam
CompTIA

PT0-002 · Question #100

Given the following output: User-agent:* Disallow: /author/ Disallow: /xmlrpc.php Disallow: /wp-admin Disallow: /page/ During which of the following activities was this output MOST likely obtained?

The correct answer is A. Website scraping. The provided output, a robots.txt file, is typically consulted by web crawlers or scrapers to understand which parts of a website should not be accessed, making it most relevant to website scraping.

Reconnaissance and enumeration

Question

Given the following output:

User-agent:* Disallow: /author/ Disallow: /xmlrpc.php Disallow: /wp-admin Disallow: /page/ During which of the following activities was this output MOST likely obtained?

Options

  • AWebsite scraping
  • BWebsite cloning
  • CDomain enumeration
  • DURL enumeration

How the community answered

(29 responses)
  • A
    93% (27)
  • C
    3% (1)
  • D
    3% (1)

Why each option

The provided output, a robots.txt file, is typically consulted by web crawlers or scrapers to understand which parts of a website should not be accessed, making it most relevant to website scraping.

AWebsite scrapingCorrect

The provided output is a `robots.txt` file, which specifies directives for web robots (like crawlers and scrapers) about which areas of a website they are allowed or disallowed to access. Website scraping tools typically parse this file before attempting to extract content, making this output most likely obtained during a scraping activity.

BWebsite cloning

Website cloning involves creating an exact copy of a website; while a cloner might encounter `robots.txt`, its primary goal isn't to read and adhere to these directives for content extraction, but to replicate the site structure and content.

CDomain enumeration

Domain enumeration focuses on discovering subdomains or related domains for a target, not on reading specific files within a known domain for access restrictions.

DURL enumeration

URL enumeration involves discovering valid URLs or directories on a website, but the `robots.txt` output specifically defines disallowed paths for robots, which is more directly relevant to automated content extraction (scraping) than just finding valid URLs.

Concept tested: robots.txt file purpose, website reconnaissance

Source: https://developers.google.com/search/docs/crawling-indexing/robots/intro

Topics

#Robots.txt#Website scraping#Web reconnaissance#Information gathering

Community Discussion

No community discussion yet for this question.

Full PT0-002 Practice