language: en

Improving Text Classification by Shrinkage in a Hierarchy of Classes

Andrew McCallum
mccallum@justresearch.com
Ronald Rosenfeld
roni@cs.cmu.edu
Tom Mitchell
mitchell+@cs.cmu.edu
Andrew Y. Ng
ayn@ai.mit.edu
Just Research
4616 Henry Street Pittsburgh, PA 15213
School of Computer Science Carnegie Mellon University
Pittsburgh, PA 15213
MIT AI Lab
545 Technology Square Cambridge, MA 02139

Abstract When documents are organized in a large number of topic categories, the categories are often arranged in a hierarchy. The U.S. patent database and Yahoo are two examples. This paper shows that the accuracy of a naive Bayes text classifier can be significantly improved by taking advantage of a hierarchy of classes. We adopt an established statistical technique called shrinkage that smoothes parameter estimates of a data-sparse child with its parent in order to obtain more robust parameter estimates. The approach is also employed in deleted interpolation, a technique for smoothing n-grams in language modeling for speech recognition. Our method scales well to large data sets, with numerous categories in large hierarchies. Experimental results on three real-world data sets from UseNet, Yahoo, and corporate web pages show improved performance, with a reduction in error up to 29% over the tradi tional flat classifier.

1 Introduction