Question: The purpose of a robots.txt file is to:
- list the pages on a website to manage the content that’s shown to search engines
- instruct search engines on how to handle duplicate content
- instruct search engine bots on how to crawl the pages on a website
- tell HubSpot the user agent of a visitor’s browser
The right answer was: instruct search engine bots on how to crawl the pages on a website