DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING | DWIGHT LOOK COLLEGE OF ENGINEERING | TEXAS A&M UNIVERSITY

 

HOME

ABOUT

COURSES

PEOPLE

PROJECTS

PUBLICATIONS

CONTACT

LINKS

Scalable web crawling

Sponsor: NSF

Abstract

Web crawling is a challenging issue in today's Internet due to many factors. These include the massive amount of content available to the crawler, existence of highly branching spam farms, prevalence of useless information, and necessity to adhere to politeness constraints at each target host. This project investigates scalable and efficient web algorithms that can be used in high-performance search engines to crawl hundreds of billions of pages and keep the overhead manageable. Unlike commercial search engines, our focus is to enable Internet-wide crawls and data mining without access to enormous server clusters or exotically expensive hardware.

Journal Publications

 

bullet

H.-T. Lee, D. Leonard, X. Wang, and D. Loguinov, "IRLbot: Scaling to 6 Billion Pages and Beyond," ACM Transactions on the Web, vol. 3, no. 3, June 2009.

PDF

Conference Papers

 
bullet

S.T. Ahmed, C. Sparkman, H.-T. Lee, and D. Loguinov, "Around the Web in Six Weeks: Documenting a Large-Scale Crawl," IEEE INFOCOM, April 2015.

PDF, PPT
 
bullet

S. Sood and D. Loguinov, "Probabilistic Near-Duplicate Detection Using Simhash," ACM CIKM, October 2011.

PDF, PPT
 
bullet

C. Sparkman, H.-T. Lee, and D. Loguinov, "Agnostic Topology-Based Spam Avoidance in Large-Scale Web Crawls," IEEE INFOCOM, April 2011.

PDF, PPT

 

bullet

H.-T. Lee, D. Leonard, X. Wang, and D. Loguinov, "IRLbot: Scaling to 6 Billion Pages and Beyond,'' WWW, April 2008 (best paper award).

PDF, PPT

Datasets

A cleaned-up PLD (i.e., domain-level) out-graph from the 2007 experiment is available here (14 GB), with the corresponding map (2.2 GB). The graph consists of 8-byte hashes, followed by a 4-byte out-degree and a list of 8-byte neighbor hashes. The map contains an 8-byte hash, a 2-byte domain length, followed by a NULL-terminated string of the corresponding domain. All numbers are in LSB-first order.


     Copyright 2002-2017 IRL at Texas A&M. All Rights Reserved.