Round-tripping LIFT data through XLingpaper

Rationale

The LIFT specification allows for interchange between lexical databases we use, such as in FLEx and WeSay. As an XML specification, it is also subject to XSL transformation, and can be converted to XML documents that conform to other specifications, such as XLingPaper, an XML specification for writing linguistics papers. I described before a means to get data out of FLEx into XlingPaper, but that required a script generating regular expressions which were then put into a FLEx filter by hand (metaphorically speaking). Computers should be able to automate this, and so (following my “If computers can do a particular task, they should” motto) I developed a script to take that regular expression generator, and feed those expressions to an XSL stylesheet to produce XlingPaper XML from the LIFT XML automatically.
The other half of the rationale is that I hate exporting data from a database to a paper or report, seeing and error, and not being able to fix it once. Either I fix it in the paper and the database, or else in the database, then re-export to the paper. So a way to get data from LIFT to XlingPaper and back seemed helpful for drafting linguistics papers, even if one wasn’t dealing with the volume of reports I’m looking at generating.

Tools

One major caveat for this work is that these tools (FLEx, WeSay, and XLingpaper) are in active development, so functionality may vary over time. The tests in this post were run with the following:

FLEx 7.0.6.40863 (for Linux)
WeSay 1.1.9 (for Linux) –This doesn’t enter directly into these tests, but the LIFT files used often sync back and forth between these two programs.
xsltproc from a standard Ubuntu Linux install (i.e., compiled against libxml 20706, libxslt 10126 and libexslt 815)
GNU bash, also from standard Ubuntu Linux (i.e., version 4.1.5)
GNU diffutils, also from standard Ubuntu Linux (i.e., version 2.8.1)
XMLMind Xml Editor, version 5.1.0
XLingPaper, version 2.18.0_3

All of these tools are free (or have a free version) and available online from their respective sources, and most are open source.
The scripts I’ve written (to generate reports and call the XSL transforms) are not yet publicly available; I hope to have them cleaned up and more broadly tested before long.

Test Goals

I want to see if I can

Get data from LIFT to XLingPaper format,
Modify the XLingPaper document in XXE (which keeps it in conformity to the XLingPaper DTD),
Get it back into LIFT and imported to FLEx,
Show that the FLEx import made all and only the changes made by modifying the XLingPaper document (i.e., no other data loss)

To do this I will be using an output of diff between two versions of the XLingPaper document (original and modified), and another diff between two versions of the LIFT file (originally exported, and exported after input). To achieve #4, I will show that the two diffs show all and only the same changes to data entries (the modifications to the XLingPaper doc are the same as the changes to the FLEx database, as evidenced by its export to LIFT). Fyi, this LIFT file has 2033 entries, and takes up almost 2MB (plain text), so we’re not talking about a trivial amount of data.

Test procedure

Backup Wesay folder (this is real [ gey ] data I’m working with, after all…)
Export “Full Lexicon” from FLEx, and copy it to gey.ori.lift
Run report (vowel inventory) on exported gey.lift (This creates Report_VowelInventory.gey.xml)
Open created report in XXE
Modify and save (because XXE changes format –this helps diff see real changes, not those irrelevant to xml)
Save as Report_VowelInventory.gey.mod.xml, and modify one example of each field we’re interested in, including @root (at this point both files have been saved by XXE, for easier comparison).
Run `diff Report_VowelInventory.gey.{,mod.}xml` (results below)
Run `xlp-extract2lift Report_VowelInventory.gey.mod.xml .` (This creates Report_VowelInventory.gey.mod.compiledfromXLP.lift)
Backup FLEx project (just in case, as there’s real data here, too)
Import Report_VowelInventory.gey.mod.compiledfromXLP.lift to FLEx project, selecting “import the conflicting data and overwrite the current data (importing data overrules my work).” and unticking “Trust entry modification times” (This is important because if that box is selected entries won’t import unless you have also changed the ‘dateModified’ attribute on an entry –which I generally don’t).
Export again, producing a second LIFT file exported by FLEx (one before, and one after the import)
Run `diff gey{,.ori}.lift`
Compare diffs to see fidelity of the process.

Test results

Here is the diff showing the changes between the original report and the modifications:

$ diff Report_VowelInventory.gey.{,mod.}xml
11c11
<>Rapport de l’Inventaire des Voyelles de [gey]</title
—
>>Rapport de l’Inventaire des Voyelles de [gey]MOD</title
23c23
<>Kent Rasmussen</author
—
>>Kent RasmussenMOD</author
42c42
<>Voyelles</secTitle
—
>>VoyellesMOD</secTitle
65c65
<>mbata</langData
—
>>mbataMOD</langData
89c89
<>pl: mabata</langData
—
>>pl: mabataMOD</langData
113c113
<>fissure, fente</gloss
—
>>fissure, fenteMOD</gloss
137c137
<>mke / wake</gloss
—
>>mke / wakeMOD</gloss
155c155
< externalID=”ps=’Noun’|senseid=’hand_0d9c81ef-b052-4f61-bc6a-02840db4a49e’|senseorder=”|definition-swh=’mkono

/ mikono'”
—
> externalID=”ps=’Noun’|senseid=’hand_0d9c81ef-b052-4f61-bc6a-02840db4a49e’|senseorder=”|definition-swh=’mkono

/ mikonoMOD'”
171c171
< externalID=”ps=’Noun’|senseid=’orange_2924ca57-f722-44e1-b444-2a30d8674126’|senseorder=”|definition-fr=’orange'”
—
> externalID=”ps=’Noun’|senseid=’orange_2924ca57-f722-44e1-b444-2a30d8674126’|senseorder=”|definition-fr=’orangeMOD'”
180c180
< externalID=”root=’paka’|entrydateCreated=’2011-08-05T10:57:05Z’|entrydateModified=’2011-09-27T11:24:32Z’|entryguid=’44dcf55e-9cd7-47a9-ac66-1713a3769708’|entryid=’mopaka_44dcf55e-9cd7-47a9-ac66-1713a3769708′”
—
> externalID=”root=’pakaMOD’|entrydateCreated=’2011-08-05T10:57:05Z’|entrydateModified=’2011-09-27T11:24:32Z’|entryguid=’44dcf55e-9cd7-47a9-ac66-1713a3769708’|entryid=’mopaka_44dcf55e-9cd7-47a9-ac66-1713a3769708′”

As you can see from this diff output, I changed data in a number of different types of fields, including the report title, author, sectionTitle, langData (from citation), langData (from Plural), glosses in each of French and Swahili, and the last three are root and definitions, which are not visible in the printed report, but stored in an ExternalID attribute (recently added to XLingPaper to be able to store this kind of info, without having to put it elsewhere in the structure of the doc).

And here is the diff showing the changes between the original LIFT export and the one exported after importing the LIFT file with modifications:

$ diff gey{,.ori}.lift
2601c2601
<<form lang=”swh”><text>mkono / mikonoMOD</text></form>
—
><form lang=”swh”><text>mkono / mikono</text></form>
10776c10776
<<gloss lang=”swh”><text>mke / wakeMOD</text></gloss>
—
><gloss lang=”swh”><text>mke / wake</text></gloss>
15871c15871
<<form lang=”gey”><text>pakaMOD</text></form>
—
><form lang=”gey”><text>paka</text></form>
23529c23529
<<form lang=”gey”><text>mbataMOD</text></form>
—
><form lang=”gey”><text>mbata</text></form>
27587c27587
<<field type=”Plural”><form lang=”gey”><text>mabataMOD</text></form>
—
><field type=”Plural”><form lang=”gey”><text>mabata</text></form>
31657c31657
<<form lang=”fr”><text>orangeMOD</text></form>
—
><form lang=”fr”><text>orange</text></form>
32416c32416
<<gloss lang=”fr”><text>fissure, fenteMOD</text></gloss>
—
><gloss lang=”fr”><text>fissure, fente</text></gloss>

Summary

The first several MOD’s to the paper (to titles, etc.) are not in the second diff, since only example data is extracted into the LIFT file to import (this is what we want, right?).
The other mods –root, citation, plural, gloss-swahili, gloss-french, definition-french and definition-swahili– all survived.
No other changes existed between the exported LIFT files.

Discussion

Because FLEx exported essentially the same LIFT file (of 2033 entries and almost 2MB, remember), with all and only the changes made in XXE, I presume that there were no destructive changes to the underlying FLEx database, and this procedure is safe for further testing. I did not go so far as to diff the underlying fwdata file, as I probably wouldn’t understand its format anyway, and I wouldn’t know how to distinguish between differences in formatting and content (while it is also XML, I don’t understand its specification or how it is used in the program –which is not a bad thing).
Speaking of what I don’t know, I should be clear that my formal training is in Linguistics (M.A. Oregon 2002), not in IT. I’m doing this because there is a massive amount of linguistic data to collect, organize, analyze and verify, and I want to do that efficiently (the fact that this is fun is just a nice byproduct). In any case, I have certainly not followed best practices in my bash or XSL scripting. So if you read the attachments and think “this guy doesn’t know how to code efficiently or elegantly,” then we’re already in agreement on that. And you’d also be welcome to contribute on improvements. $:-\)$

Acknowledgements

I wouldn’t have gotten anywhere on this project without the work of many others, particularly including those that are giving of their own time and resources (which surely could have been spent elsewhere) on FLEx, WeSay, and the LIFT specification itself. Of particular note is Andy Black, who encouraged me to take another stab at XSLT (after telling him I’d tried and given up a few years ago), and who has provided invaluable and innumerable helps, both in the development of the XLingPaper specification, and in particular issues related to these transforms. Most of what is good here has roots in his work, though I hope no one holds him responsible for my errors and inelegance.

Round Tripping Data through XLingPaper