Persons use DA to find out which Web-sites are trustworthy in their industry and to figure out where to get excellent links. txt file is then parsed and will instruct the robot regarding which web pages are usually not to generally be crawled. To be a online search engine crawler https://www.youtube.com/watch?v=0xsCkQ6bbjk