General, Books
General, On-line and not in book form
Bayesian Modeling
Boosting (see "Ensemble Models")
Conditional Random Fields
Context-Free Grammars (CFG, PCFG) (see "Parsing")
Coreference
Dirichlet Processes (see "Nonparametric Bayes")
Discourse
Ensemble Models, Boosting
Expectation Maximization
Gaussian Processes
Gibbs Sampling (See "Sampling")
Grammar Learning/Inference/Induction
Graphical Models
Hidden Markov Models
Information Extraction
Information Retrieval, Question Answering
Information Theory
Lagrange Multipliers
Language Modeling, N-Grams
Lexical Acquisition
Logistic Regression (see "Maximum Entropy")
Kernels (see "Support Vector Machines")
Machine Learning (General)
Machine Translation
Markov Chain Monte Carlo (MCMC) (See "Sampling")
Markov Logic Networks
Maximum Entropy, Logistic Regression
Mixture Models
N-Grams (See "Language Modeling")
Named Entity Recognition
Neural Networks
Nonparametric Bayes, Dirichlet Processes
Parsing
Part-of-Speech Tagging
Principal Components Analysis (PCA)
Reranking
Sampling, Gibbs Sampling, Markov Chain Monte Carlo (MCMC)
Scalability
Selectional Preferences (see "Lexical Acquisition")
Semantic Role Labeling (see "Shallow Semantics")
Semantics
Semi-Supervised Learning
Shallow Semantics
Speech Recognition, Speech Synthesis
Summarization
Support Vector Machines, Kernels
Topic Modeling
Variational Methods
Word Sense Disambiguation (WSD) (see "Shallow Semantics")

Here are some useful links on various technical topics related to NLP. See also nlp_links for general NLP links of various sorts.

Format of the following links:

Each link contains a URL, a reference type, an anchor text containing author, year and title, and possibly an annotation further describing the work.
The reference type appears at the front and is one of:
- short overview: A short tutorial, introduction, survey or similar paper – less than 15 pages or so. These are good places to start if you are unfamiliar with a topic.
- long overview: A long tutorial, introduction, survey or similar paper – 15 pages or more. These are good places to start if you are unfamiliar with a topic, and give in-depth coverage of the topic.
- short paper: A conference paper (usually 8 pages) or a workshop paper (usually 4 pages). Intended for a technical audience already familiar with the subject at hand and typically quite terse. This is generally where new research results are announced.
- long paper: A journal paper (often 15 pages or more). Intended for a technical audience already familiar with the subject at hand. These papers generally have many background details and the distinction between “overview” and “long paper” is not always clear.
- thesis: A PhD thesis or dissertation (usually at least 100 pages) or a Master’s thesis (usually 50-100 pages). Generally contain a detailed overview of a subject in addition to a major piece of new research. Often gives an exhaustive description of current work.
- book: A book. May not be online.
- chapter: A chapter from a book. May not be online.
- slides: A set of slides from a lecture on a subject. Typically in the form of an overview, but may be harder to follow than a paper overview, because of missing text that would be supplied by the lecturer.
- video: A video lecture on a subject (often 30-60 minutes).
- class: A page containing information from a class, often a series of lecture notes.
- web page: A page in the form of a web page rather than a paper. Typically in the form of an overview, but usually shorter and often less carefully written than a paper. Note, however, that Wikipedia entries are often well-written and informative.
- web site: An entire site devoted to some topic.
- links: A page containing links to other references.

Work to do on this page/suggestions:

Much work needs to be done on the anchor text. All papers should have anchor text like this: “NAME OF ARTICLE - AUTHOR (WHERE, YEAR)” where WHERE is a short abbrev of the conference or magazine where it appeared.
Annotations should be added: Is this a tutorial (short or long? beginning or advanced?), classic paper, etc.?
If the number of papers goes over a small handful, separate into subsections, like currently exists for CRF’s and Maxent

General, Books

NOTE: An entry marked as [on-line] means that the whole book is available on-line. Even for the other books, there are often sample chapters and supplementary material of other sort available.

book [on-line] Jurafsky and Martin (2008) - Speech and Language Processing, draft 2nd edition: Most draft chapters are available online and are perhaps the best reference to start with for those who want a solid intro without excessive math. Beware: The first edition of Jurafsky and Martin is lacking in coverage of a large number of currently core NLP areas. Use the draft 2nd edition instead.
book Manning and Schuetze - Foundations of Statistical Natural Language Processing: Along with Jurafsky and Martin, this is the standard NLP reference. Beware: The text is often pitched too high to work well as a first introduction to the field.
book Chris Bishop (2007) - Pattern Recognition and Machine Learning: This book covers statistical machine learning rather than NLP. It is great as a more-advanced intro to many important modern-day topics (generalized linear models, generative vs. discriminative models, neural networks, kernel methods, graphical models, mixture models, EM, variational inference, sampling, HMM’s, ensembles, etc.). Beware: The book claims to be suitable as a first intro for those with little or no previous machine learning or probability background, but it in fact is pitched far, far above this. You should have a good understanding of probability and statistics and decent experience with machine learning before tackling this book.
book Russell and Norvig - Artificial Intelligence: A Modern Approach: This is the standard book on artificial intelligence, and is pitched towards those new to the field. It has chapters on probability (mostly Bayesian), probabilistic inference, directed graphical models (i.e. Bayes nets), probabilistic inference over time (e.g. HMM’s), learning, statistical learning, etc. These can serve as good overviews for those with no prior experience with these topics.
book [on-line] David MacKay - Information Theory, Inference, and Learning Algorithms: Covers much more than just information theory. Extensive coverage of many statistical topics.
book Tom Mitchell - Machine Learning: This is the classic book for machine learning. However, it covers very little in the way of statistical machine learning. It does have one chapter on Bayesian learning and one on computational learning theory.

The web site has a new chapter on Naive Bayes and logistic regression.

book Degroot and Schervish - Probability and Statistics (intermediate): There are a tremendous number of books on statistics. Most of them follow a pretty standardized set of topics, but differ somewhat in the level of difficulty. This book’s level is intermediate. It assumes at least passing familiarity with statistics and multi-variate calculus. The advantage of this book for an NLP‘er is that it takes a much more Bayesian outlook than most other books of this sort. It also has a good chapter on sampling (importance/rejection sampling, Gibbs sampling, MCMC, bootstrap, etc.).
book Casella and Berger - Statistical Inference (advanced): This is the classic book on statistics. It is well organized and extremely clearly written. It assumes previous statistics background. It has little Bayesian statistics, making it less than completely useful for NLP‘ers.
links [on-line] Books on machine translation, often online

General, On-line and not in book form

web page Wikipedia: Coverage is amazing. Quality is consistently high. Beware that some subjects will have excessively technical articles; in such a case, check the MathWorld or PlanetMath, alternative math-specific encyclopedias often with spottier coverage on statistics but typically less-technical articles.
slides Andrew Moore's Statistical Data Mining Tutorials: High-quality slide-style tutorials on a wide range of topics related to statistical machine learning.
web site SOCR: Statistics Online Computational Resource: Great statistics resource, filled with hands-on demonstrations letting you learn the practical behavior of algorithms, distributions, etc.
class Class based on Bishop's book, with slides for each chapter

Bayesian Modeling

links Sharon Goldwater's Bayesian language modeling reading list

Boosting (see "Ensemble Models")

Conditional Random Fields

Overviews and fundamental texts:

long overview Charles Sutton and Andrew McCallum (2006) - An Introduction to Conditional Random Fields for Relational Learning: The standard tutorial
short overview Hanna M. Wallach (2004) - Conditional Random Fields: An Introduction
long overview Rahul Gupta (2005) - Conditional Random Fields
short paper John Lafferty, Andrew McCallum, Fernando Pereira (ICML 01) - Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data: The fundamental paper

Other important texts:

short paper Sutton, McCallum (ICML 07) - Piecewise Pseudolikelihood for Efficient CRF Training
short paper Sutton, McCallum (Conf UAI 05) - Piecewise Training for Undirected Models
thesis Wallach (2002) - Efficient Training of Conditional Random Fields

Pages with links to related texts:

links Conditional Random Fields

Context-Free Grammars (CFG, PCFG) (see "Parsing")

Coreference

Dirichlet Processes (see "Nonparametric Bayes")

Discourse

chapter chapter 22 on Discourse from Jurafsky and Martin (2008) - Speech and Language Processing, draft 2nd edition

Ensemble Models, Boosting

Overviews and fundamental texts:

chapter Chapter 14 of Chris Bishop (2007) - Pattern Recognition and Machine Learning
short overview Yoav Freund, Robert E. Schapire (1999) - A Short Introduction to Boosting
long paper Yoav Freund, Robert E. Schapire (ECCLT 95) - A Decision-Theoretic Generalization of on-Line Learning and an Application to Boosting

Other important texts:

Expectation Maximization

web page Wikipedia page on expectation maximization
short overview Jeff Bilmes - A Gentle Tutorial of the EM Algorithm and its Application to Parameter Estimation for Gaussian Mixture and Hidden Markov Models:The classic text. Despite claiming to be “gentle”, however, it’s pretty math-heavy.
short overview Frank Dellaert - The Expectation Maximization Algorithm
short overview Sean Borman (2006) - The Expectation Maximization Algorithm: A short tutorial
chapter Chapter 10 of Chris Bishop (2007) - Pattern Recognition and Machine Learning

Gaussian Processes

web site The Gaussian Processes Web Site
book Rasmussen and Williams (2006) - Gaussian Processes for Machine Learning: Online book on Gaussian processes.

Gibbs Sampling (See "Sampling")

Grammar Learning/Inference/Induction

Lexicalized:

[Charniak, 1996]
[Collins, 1999]

Manual refinement:

[Johnson, 1998]
[Klein, Manning, 2003]

Automatic refinement:

[Matsuzaki, et al., 2005]
[Petrov, et al., 2006]

Misc:

Kenichi Kurihara and T. Sato. Variational Bayesian grammar induction for natural language. International Colloquium on Grammatical Inference, 2006.

Graphical Models

long overview Kevin P. Murphy (2001) - An Introduction to Graphical Models: great intro.
chapter Chapter 8 of Chris Bishop (2007) - Pattern Recognition and Machine Learning
video Sam Roweis (MLSS Taipei 06) - Machine Learning, Probability and Graphical Models
long overview Michael Jordan (Statistical Science 2004) - Graphical Models
long overview Michael Jordan and Y. Weiss (Handbook of Brain Theory and Neural Networks 2004) - Graphical Models: Probabilistic inference

Variational methods:

Hidden Markov Models

long overview Rabiner - A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition: The classic paper.
chapter Chapter 6 on Hidden Markov and Maximum Entropy Models from Jurafsky and Martin (2008) - Speech and Language Processing, draft 2nd edition
chapter Chapter 13 of Chris Bishop (2007) - Pattern Recognition and Machine Learning
long overview Zoubin Ghahramani and Michael Jordan (ML 1997) - Factorial hidden Markov models

Information Extraction

Information Retrieval, Question Answering

Information Theory

long paper Shannon (1948) - A Mathematical Theory of Communication
short paper Jaynes (1957) - Information Theory and Statistical Mechanics

Lagrange Multipliers

Language Modeling, N-Grams

Lexical Acquisition

Learning semantic classes from corpora:

short paper Lin (ACL 98) - Automatic Retrieval and Clustering of Similar Words

Learning semantic relations:

short paper Hearst (COLING 92) - Automatic Acquisition of Hyponyms from Large Text Corpora
short paper Snow et al (ACL 06) - Semantic Taxonomy Induction from Heterogenous Evidence
short paper Berland & Charniak (1999) - Finding Parts in Very Large Corpora
short paper Chklovski & Pantel (EMNLP 04) - VERBOCEAN: Mining the Web for Fine-Grained Semantic Verb Relations

Selectional preferences:

Learning paraphrases:

short paper Barzilay & McKeown (2001) - Extracting Paraphrases from a Parallel Corpus
long paper Lin & Pantel (JNLE 01) - Discovery of Inference Rules for Question Answering
short paper Bannard & Callison-Burch (ACL 05) - Paraphrasing with Bilingual Parallel Corpora

Logistic Regression (see "Maximum Entropy")

Kernels (see "Support Vector Machines")

Machine Learning (General)

link Nick Rizzolo's AI Qual Reading List

Machine Translation

Introduction:

chapter chapter 25 on Machine Translation from Jurafsky and Martin (2008) - Speech and Language Processing, draft 2nd edition
Kevin Knight (AI Magazine, 1997) - Automating Knowledge Acquisition for Machine Translation
Kevin Knight (1999) - A Statistical MT Tutorial Workbook

Word-for-Word Translation:

Phrase-based Systems:

short paper David Chiang (ACL 05) - A Hierarchical Phrase-Based Model for Statistical Machine Translation: Won best paper award. Sometimes considered a competitor to syntax-based methods.
Philipp Koehn, Franz Josef Och, and Daniel Marcu (NAACL, 2003) - Statistical Phrase-Based Translation: Another standard model.
Philipp Koehn and Kevin Knight (ACL, 2003) - Feature-Rich Statistical Translation of Noun Phrases
Daniel Marcu and William Wong (EMNLP, 2002) - A Phrase-Based, Joint Probability Model for Statistical Machine Translation

Syntax-Based Systems:

Language Modeling for MT:

Evaluation:

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu (ACL, 2002) - Bleu: a method for automatic evaluation of machine translation

Discriminative Training:

Synchronous Tree Adjoining Grammar:

Interlingua:

Example-Based MT:

Multilingual NLP:

Tree Transducers:

short paper Jonathan Graehl and Kevin Knight (NAACL, 2004) - Training Tree Transducers

Pages with links to related texts:

web site http://www.mt-archive.info/Machine Translation Archive: Electronic repository and bibliography of articles, books and papers on topics in machine translation and computer-based translation tools.
web site NAACL-HLT 2007 Workshop - Syntax and Structure in Statistical Translation (SSST)
web site Workshop Shared Task Statistical Machine Translation

Markov Chain Monte Carlo (MCMC) (See "Sampling")

Markov Logic Networks

long overview Matthew Richardson, Pedro Domingos - Markov Logic Networks: The standard reference.

Maximum Entropy, Logistic Regression

Overviews and fundamental texts:

long overview Berger, et al. (1996) - A Maximum Entropy Approach to Natural Language Processing: The standard reference. Start here.
chapter chapter 6 on Hidden Markov and Maximum Entropy Models from Jurafsky and Martin (2008) - Speech and Language Processing, draft 2nd edition: Another good place to start.
slides Dan Klein, Chris Manning - Maxent Models, Conditional Estimation, and Optimization, without the Magic: Another good place to start.
short overview Adwait Ratnaparkhi - A Simple Introduction to Maximum Entropy Models for Natural Language Processing: A good intro but somewhat technical.
thesis Adwait Ratnaparkhi (1998) - Maximum Entropy Models For Natural Language Ambiguity Resolution: Lots of details about maximum entropy NLP applications.
chapter Tom Mitchell - Generative and Discriminative Classifiers Naive Bayes and Logistic Regression: Discusses the larger context of logistic regression; a bit math-heavy.
class A course at Johns Hopkins - Introduction to Natural Language Processing: Maximum Entropy

Other important texts:

short paper Della Pietra, et al. (1997) - Inducing Features of Random Fields
thesis Rosenfeld (1994) - Adaptive Statistical Language Modeling: A Maximum Entropy Approach
short overview Adam Berger (1997) - The Improved Iterative Scaling Algorithm: A Gentle Introduction
short paper Malouf (2003) - A comparison of algorithms for maximum entropy parameter estimation
long paper Chen and Rosenfeld (1999) - A Gaussian Prior for Smoothing Maximum Entropy Models

Pages with links to related texts:

links Maximum Entropy Modeling
links Maximum Entropy Online Resources

Mixture Models

chapter Chapter 9 of Chris Bishop (2007) - Pattern Recognition and Machine Learning
slides Andrew Moore's Gaussian Mixture Models tutorial
short overview Petteri Nurmi - Mixture Models: Incomplete.
long overview Jean-Michel Marin, Kerrie Mengersen and Christian P. Robert - Bayesian Modelling and Inference on Mixtures of Distributions
slides Padhraic Smyth - A Guided Tour of Finite Mixture Models: From Pearson to the Web

N-Grams (See "Language Modeling")

Named Entity Recognition

thesis Andrew Borthwick (1999) - A Maximum Entropy Approach to Named Entity Recognition
links Toine Bogers - Named Entity Recognition

Neural Networks

chapter Chapter 5 of Chris Bishop (2007) - Pattern Recognition and Machine Learning

Nonparametric Bayes, Dirichlet Processes

Overviews:

web page Wikipedia page on Dirichlet processes
long overview Daniel J. Navarro, Thomas L. Griffiths, Mark Steyvers, Michael D. Lee - Modeling individual differences using Dirichlet processes: Start here.
slides Teg Grenager (2005) - Chinese Restaurants and Stick-Breaking: An Introduction to the Dirichlet Process
video Yee Whye Teh (MLSS07) - Dirichlet Processes: Tutorial and Practical Course
slides Yee Whye Teh (MLSS07) - Dirichlet Processes: Tutorial and Practical Course
short overview Yee Whye Teh - Dirichlet Processes
slides Khalid El-Arini - Dirichlet Process Mixtures
slides Michael Jordan (NIPS05) - Dirichlet Processes, Chinese Restaurant Processes and All That
short overview Ananth Ranganathan - The Dirichlet Process Mixture (DPM) Model
chapter David Aldous (Reversible Markov Chains and Random Walks on Graphs) - Markov Chain Monte Carlo

Basic theory:

T. S. Ferguson (Annals of Statistics, 1973) - A Bayesian analysis of some nonparametric problems: Defines the Dirichlet process using stochastic processes
C. E. Antoniak (Annals of Statistics, 1974) - Mixtures of Dirichlet processes with applications to Bayesian nonparametric problems
D. Blackwell and J. B. MacQueen (Annals of Statistics, 1973) - Ferguson distributions via Pólya urn schemes: Connection between DP and the Pólya urn.
J. Sethuraman (Statistica Sinica, 1994) - A constructive definition of Dirichlet priors
Michael Escobar and Mike West (Journal of the American Statistical Association, 1995) - Bayesian density estimation and inference using mixtures
R. Neal (Journal of Computational and Graphical Statistics, 2000) - Markov chain sampling methods for Dirichlet process mixture models
chapter Jim Pitman (Combinatorial Stochastic Processes) - Sequential Constructions of Random Partitions: About the Chinese Restaurant Process and generalizations.

More complex models:

long paper Yee Whye Teh, Michael Jordan, Matthew Beal and David Blei (JASA, 2006) - Hierarchical Dirichlet processes: Sets up the HDP, a mechanism of tying different DPs together (gives another treatment of HDP-HMMs).
Matthew Beal and Zoubin Ghahramani and Carl Rasmussen (NIPS 2002) - The infinite hidden Markov model: Introduces the (infinite) HDP-HMMs.
[Goldwater, et al., 2006]
Eric P. Xing, Kyung-Ah Sohn (Bayesian Analysis, 2004) - Hidden Markov Dirichlet Process: Modeling Genetic Inference in Open Ancestral Space
[Sohn, Xing, 2007]
long paper Abel Rodríguez, David B. Dunson, Alan E. Gelfand - The Nested Dirichlet Process: not NLP-specific.

Approximate inference: MCMC (Sampling):

long overview Neal (1998) - Markov Chain Sampling Methods for Dirichlet Process Mixture Models
long paper David M. Blei, Andrew Y. Ng and Michael I. Jordan (2003) - Latent Dirichlet Allocation

Approximate inference: Empirical Bayes:

short paper Jon D. McAuliffe, David M. Blei, Michael I. Jordan (2004) - Nonparametric empirical Bayes for the Dirichlet process mixture model

Approximate inference: Variational Bayes:

slides Percy Liang and Dan Klein (ACL 2007) - Structured Bayesian Nonparametric Models with Variational Inference
short paper David M. Blei, Michael I. Jordan - Variational methods for the Dirichlet process
David Blei and Andrew Ng and Michael Jordan (JMLR, 2003) - Latent Dirichlet allocation: Uses variational inference for LDA.
David Blei and Michael Jordan (Bayesian Analysis, 2006) - Variational inference for Dirichlet process mixtures: First mean-field algorithm for DP mixtures (stick-breaking representation).
Kenichi Kurihara and Max Welling and N. Vlassis (NIPS, 2007) - Accelerated variational Dirichlet mixture models: Use KD-trees to speed up inference for DP mixtures (stick-breaking representation).
Yee Whye Teh and Dave Newman and Max Welling (NIPS, 2007) - A collapsed variational Bayesian inference algorithm for Latent Dirichlet Allocation: Variational inference in the collapsed CRP representation for LDA.
Kenichi Kurihara and Max Welling and Yee Whye Teh (IJCAI, 2007) - Collapsed variational Dirichlet process mixture models: Compares several variational inference algorithms.

Language modeling:

long paper David J.C. MacKay, Linda C. Bauman Peto (1994) - A Hierarchical Dirichlet Language Model

Other applications:

Pages with links to related texts:

web site NIPS 2003 Workshop - Nonparametric Bayesian Methods and Infinite Models

Parsing

chapter chapter 14 on Statistical Parsing from Jurafsky and Martin (2008) - Speech and Language Processing, draft 2nd edition
long paper Michael Collins - Head-Driven Statistical Models for Natural Language Parsing
long paper Dan Bikel - Intricacies of Collins' Parsing Model
short paper Eugene Charniak (1997) - Statistical Parsing with a Context-free Grammar and Word Statistics
long overview Eugene Charniak (AI Magazine 1997) - Statistical Techniques for Natural Language Parsing

PCFG’s:

short paper Mark Johnson, Thomas L. Griffiths, and Sharon Goldwater (2007) - Bayesian Inference for PCFGs via Markov Chain Monte Carlo
Kenichi Kurihara and T. Sato. An application of the variational Bayesian approach to probabilistic context-free grammars. International Joint Conference on Natural Language Processing Workshop Beyond Shallow Analyses, 2004.