Super-European language translation corpus

The DGT-TM corpus takes sentences from 22 European languages and has translations (manually produced) for 21 other languages. That means there are about 3 million sentences per language.

With all pairs among these 22 languages:

• Bulgarian, Czech, Danish, Dutch, English, Estonian, German, Greek, Finnish, French, Hungarian, Italian, Latvian, Lithuanian, Maltese, Polish, Portuguese, Romanian, Slovak, Slovene, Spanish, and Swedish

Yes–you got that, right, it’s all European, but it’s not *just* Indo-European.

• http://langtech.jrc.ec.europa.eu/DGT-TM.html

It’s triple the size that it was in 2007.

The corpus is made up of all European legislation: treaties, regulations, directives, etc. (If you join the EU you have to accept the whole Acquis Communautaire, so you have to get it translated into your official languages.)

• Paper: Steinberger Ralf, Andreas Eisele, Szymon Klocek, Spyridon Pilos & Patrick Schlüter (2012).

Verb phrase ellipsis corpus

One of the hot topics in syntax dissertations in recent years has been ellipsis (I’m not kidding–when I was on a job search committee a few years ago, ellipsis was *everywhere*).

Bos and Spenader (2011) have an annotated corpus!

• Bos, Johan and Jennifer Spenader. 2011.  An annotated corpus for the analysis of VP ellipsis. Language Resources and Evaluation 45(4): 463–494. 

Names

Stephanie Shih has been doing really fun work on what makes a name (first and last) using a corpus of Facebook names. This helps her get recent trends–the Social Security Administration releases first names all the time, but it doesn’t release first+last until 100 years after the birth certificates come in.

• Stephanie’s talk at the LSA

Amaç Herdagdelen has compiled  Census data from 1990 and put it with the Social Security Administration’s statistics for popular baby names for every year between 1960 and 2010:

• Data, code: https://github.com/amacinho/Name-Gender-Guesser
  • Orwant and Daly’s older Perl module has fuzzy search capabilities (phonetic similarity of names): http://search.cpan.org/~edaly/Text-GenderFromName-0.32/GenderFromName.pm
• Paper: http://clic.cimec.unitn.it/amac/twitter_ngram/Herdagdelen2012-RTC-draft.pdf

Persian verb inflections

Mohammad Sadegh Rasooli has put together a rule-based verb-inflector for Persian as part of work for preprocessing the Persian dependency treebank and some other work, find out more:

• Project: http://dadegan.ir/en
• Code: https://github.com/rasoolims/PersianVerbAnalyzer
• Paper: http://dl.acm.org/citation.cfm?id=2178234

World Englishes

Elizabeth Traugott offers these suggestions for corpora on world Englishes:

• eWAVE = The electronic World Atlas of Varieties of English. 2011. Edited by Bernd Kortmann and Kerstin Lunkenheimer. Leipzig: Max Planck Institute for Evolutionary Anthropology. http://www.ewave-atlas.org/.
• ICE = The International Corpus of English, version 2. 2006. Coordinated by Gerald Nelson (University of Hong Kong). http://ice-corpora.net/ice/index.htm.
• ONZE = Origins of New Zealand English Corpus. In progress. Compiled by the ONZE project team. University of Canterbury. http://www.lacl.canterbury.ac.nz/onze/index.html.
• SAVE = The South Asian Varieties of English Corpus. 2011. Compiled by Joybrato Mukherjee, Tobias Bernaisch, Christopher Koch, and Marco Schilk. University of Giessen. http://www.uni-giessen.de/cms/faculties/f05/engl/ling/research/save.

Erorrs erorrs evrerywehere

Earlier I showed a neat resource that has grammatical and ungrammatical sentences from various linguistics papers over the years. But what if you want a whole bunch of English errors?

In that case, check out work that David Hale and Adam Kilgarriff are putting together: http://clt.mq.edu.au/research/projects/hoo/.

You might also look around for ESL corpora, for example:

• CK Jung’s lab has a one-million word written corpus of Korean learners of English (“YELC”): http://www.uclouvain.be/en-cecl-lcworld.html.
• Izumia, Uchimotoa, and Isaharaa (2004) have a Japanese Learner English Corpus, too (NICT JLE), but I think you have to email them.

Build your own corpus (well, for now)

BootCaT is meant to help folks build up their own corpora from the Internet. However, it uses the Bing API and may not be able to so for much longer, so it may go down temporarily. Go get your corpus started now!

• http://bootcat.sslmit.unibo.it/

Note that Google also has an API that you can use (but they limit you to 100 queries a day), as does Blekko and EntireWeb.

You may also want to check out this O’Reilly summary of data sources.