EC-Council
EC0-350 · Question #148
WWW wanderers or spiders are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. Search engines like Google, frequently spider web pages for indexing. How…
The correct answer is A. Place robots.txt file in the root of your website with listing of directories that you don't want to be crawled. See the full explanation below for the reasoning.
Question
WWW wanderers or spiders are programs that traverse many pages in the World Wide Web by recursively retrieving linked pages. Search engines like Google, frequently spider web pages for indexing. How will you stop web spiders from crawling certain directories on your website?
Options
- APlace robots.txt file in the root of your website with listing of directories that you don't want to be crawled
- BPlace authentication on root directories that will prevent crawling from these spiders
- CEnable SSL on the restricted directories which will block these spiders from crawling
- DPlace "HTTP:NO CRAWL" on the html pages that you don't want the crawlers to index
How the community answered
(35 responses)- A80% (28)
- B11% (4)
- C6% (2)
- D3% (1)
Community Discussion
No community discussion yet for this question.