nl.utwente.mirex

Class AnchorExtract

java.lang.Object

nl.utwente.mirex.AnchorExtract

public class AnchorExtract
extends java.lang.Object

Runs MapReduce job: Extracts anchor text from HTML documents. The input path should contain files (or should be a file) on the Hadoop file system formatted as Web Archive (WARC) files. The output consists of gzipped, tab separated files containing: WARC-TREC-ID, URL, anchor text1, anchor text 2, etc. Only finds anchors for documents inside the collection. Documents in the collection without inlinks are not listed. Anchor texts are cut after more than 10MB of anchors have been collected for one page to keep the output manageable. This MapReduce program is described in:

Djoerd Hiemstra and Claudia Hauff. "MIREX: MapReduce Information Retrieval Experiments" Technical Report TR-CTIT-10-15, Centre for Telematics and Information Technology, University of Twente, ISSN 1381-3625, 2010

Since:

0.1

Author:

Djoerd Hiemstra, Guido van der Zanden

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	AnchorExtract.Combine -- Combiner: Glues local anchor texts together.
static class	AnchorExtract.Map -- Mapper: Extracts anchors.
static class	AnchorExtract.Reduce -- Reducer: Glues anchor texts together, and recovers TREC-ID.

Constructor Summary

Constructors

Constructor and Description
AnchorExtract()

Method Summary

Methods

Modifier and Type	Method and Description
static void	main(java.lang.String[] args) Runs the MapReduce job "anchor text extraction"

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AnchorExtract

public AnchorExtract()

Method Detail

main

public static void main(java.lang.String[] args)
                 throws java.lang.Exception

Runs the MapReduce job "anchor text extraction"

Parameters:

args - 0: path to web collection on HDFS; 1: (non-existing) path that will contain anchor texts

Throws:

java.lang.Exception

Example usage:

hadoop jar mirex-0.2.jar nl.utwente.mirex.AnchorExtract /user/hadoop/ClueWeb09_English/*/ /user/hadoop/ClueWeb09_Anchors