Individuals use DA to see which Web-sites are reliable in their subject and to figure out wherever to acquire good inbound links. txt file is then parsed and will instruct the robot concerning which pages will not be to be crawled. Like a online search engine crawler might hold a https://www.youtube.com/watch?v=cZppEXx1zPU