GPEN · Question #38
You want that some of your Web pages should not be crawled. Which one of the following options will you use to accomplish the task?
The correct answer is C. Place the name of restricted Web pages in the robotes.txt file. The robots.txt file is the standard mechanism for instructing web crawlers which pages or directories they should not access on a site.
Question
You want that some of your Web pages should not be crawled. Which one of the following options will you use to accomplish the task?
Options
- AUse HTML NO Crawl tag in the Web page not to be crawled
- BEnable the SSL
- CPlace the name of restricted Web pages in the robotes.txt file
- DPlace the name of restricted Web pages in the private.txt file
How the community answered
(22 responses)- A5% (1)
- B14% (3)
- C77% (17)
- D5% (1)
Why each option
The robots.txt file is the standard mechanism for instructing web crawlers which pages or directories they should not access on a site.
There is no standard 'HTML NO Crawl tag' - the correct HTML mechanism is a meta robots tag with content='noindex,nofollow', which is different from a non-existent NO Crawl tag.
Enabling SSL (HTTPS) encrypts data in transit between client and server but has no effect on whether a crawler is permitted to access a page.
The Robots Exclusion Protocol specifies that web crawlers check the /robots.txt file at the root of a domain before crawling, and entries using Disallow directives tell compliant bots which URLs to skip, effectively preventing those pages from being indexed or crawled.
There is no web standard called private.txt - crawlers do not recognize or respect this filename, so listing pages there would have no effect.
Concept tested: robots.txt Robots Exclusion Protocol for crawl control
Source: https://developers.google.com/search/docs/crawling-indexing/robots/intro
Topics
Community Discussion
No community discussion yet for this question.