Mckinsey Interview Questions | Web Crawler
Question
How would you optimize a web crawler to run much faster, extract better information, and better summarize data to produce cleaner databases?
in progress
0
Machine Learning
4 years
1 Answer
785 views
Great Grand Master 0
Answer ( 1 )
Some of the methods used in tuning the web-crawler pipeline
1. Optimizing DB query for improved time complexity
2. A distributed approach with servers running across different stations
3. Bulk indexing