txt file is then parsed and can instruct the robot regarding which web pages aren't to be crawled. As a search engine crawler might hold a cached copy of this file, it could on occasion crawl pages a webmaster will not want to crawl. webpages ordinarily prevented from becoming crawled contain login-certain internet pages for example purchasing cart