• split and organize uspanteko data
• clean up uspanteko data: multi-line clauses, morph-gloss alignments, untagged words

1. some loan (and other) words get no representation in the morph, gloss, and pos lines (\m, \g, \c)
2. some glossed texts include occasional unglossed words or even clauses
3. some clauses (i.e. single refID) carry over multiple sets of lines

This section is intended to be a log of the changes we make to the OKMA data in order to work with it in this project. These changes will include both general cleanup, which may be folded into the eventual distribution of these data, and EARL-specific preprocessing.

May 17 [AP] -- changed \nom to \_nom in glossed files. 
reason: NLTK toolbox reader takes first tag which does **not** begin with '_' 
to be the tag indicating the start of a new record

Data located in …/earlite/data/uspanteko
Fer each data set (train/dev/test) there are two directories: tb (shoebox/toolbox format) and igtxml (IGT-XML)

Information re: individual texts

Genre categories supplied by Telma Caan:

• story : kids' stories, folk stories, etc.
• oral history : usually have to do with history of the village and the community
• personal experience : stories from people's lives
• recipe/advice : recipe is self-explanatory, the story labeled 'advice' talks about how things should be done in order to take care of nature and the environment

New clause info