XML Pipelines and XProc 3.0: Report of the WG Meeting in Aachen

Last week (14th and 15th of September 2017) a meeting of the XProc 3.0 working group took place in Aachen, organized by Achim Berndzen of xml-project and Gerrit Imsieke of le-tex and hosted by LOGOI.

The meeting was extremely successful, consensus has been reached on many topics and important roadblocks have been overcome. I will tell you about what the WG accomplished in a second. Before that allow me to introduce XProc, XML pipelines and explain why they are useful. (If you already know all this stuff, skip directly to the XProc 3 section, that’s OK. :))

XML pipelines? What are you talking about?

Pipeline, by JuraHeep, CC0

Everybody who has worked with XML knows that real-world applications are always born as simple transformations (“I’ll just convert this XML to HTML with XSLT”) but quickly develop into a big tangled web of unreadable code as soon as you have to deal the inevitable…

  • small mistakes in the input (“wait, why is there a <p> inside a <em>?”),
  • flaws in the receiving applications (“let’s have a separate output for Internet Explorer 6, so that the poor students access this from the library”) or
  • requests from the project collaborators (“could you make a summary version with only one sentence per chapter?”).

Addressing all these needs can be done, but doing it by adding fixes on top of fixes on the original core transformation is a nightmare in terms of maintenance and readability.

Small steps and scripts

A better way to solve all these issues is splitting monolithic transformations into smaller pieces, or steps. (More about how our experience at the CCeH in splitting complicated transformations into focused steps in a future article.)

Now that you have all these steps, how do you transform the input into the output in practice?

Are you going to run each step manually, clicking around in your XML editor? I hope not.

A much better way to run this split transformation is to create a shell script that takes the input file, applies the first step (fix the small mistakes), then the second (transform into HTML) and then, if requested, the third (uglify HTML to make it IE6 compatible).

Such a script would work just fine but it has many problems:

  • Either you hardcode how to invoke the XSLT processor or you have to write an abstraction layer that allows you to call other XSLT processors.
  • Requires a working Bash shell environment (not that easy to get on Windows).
  • Does not provide any kind of validation of the intermediate results.
  • Requires a deserialization/serialization cycle for each step.
  • Gets rapidly very complex as soon as other steps, conditional steps and loops are added.
  • Works only on a single document.

We could address all these problems ourselves making a better script. Or we could avoid reinventing the wheel and make use of XProc and write a declarative XML pipeline.

Enter XML pipelines and XProc

XProc is a language for writing declarative XML pipelines.

An XML pipeline is a series of steps though which an XML documents flow, just as in the shell script in the previous example. However, in contrast with a shell script, XProc pipelines are:

  • Declarative: you state what you want and the XProc interpreter chooses the right tools. (A PDF transformation? Let’s use Apache FOP. An XSLT Transformation? Let’s use libxslt. Oh, are we running inside oXygen? Let’s use the internal Saxon-EE engine then.)
  • Portable: pipelines can run wherever there is a XProc interpreter: Linux, Windows, Mac OS, you name it.
  • Specialized for XML: documents are not deserialized and serialized in each step.
  • Can have more than one input and produce more than one output.
  • Easily extend to intricate pipelines with loops and parallel branches.

An example pipeline looks like the following

<p:pipeline xmlns:p="http://www.w3.org/ns/xproc" version="1.0">
        <p:input port="stylesheet">
            <p:document href="fix-mistakes.xsl"/>

        <p:input port="stylesheet">
            <p:document href="convert-doc.xsl"/>

    <p:xslt use-when="p:system-property('ie6-compatible') = 'true'">
        <p:input port="stylesheet">
            <p:document href="make-ie6-compatible.xsl"/>

XProc 3.0

XProc 3.0 is the upcoming version of XProc. The original XProc 1 specifications have been published in 2010 by the W3C and since then users and implementers have found small problems, inconsistencies as well as ergonomic issues that make writing XProc pipelines harder than it should.

The focus of XProc 3 is simplifying the language, making implementations behave in more sensible way by default and making it possible to process non-XML documents (think LaTeX or graphic files).

During last week’s working group meeting in Aachen plenty of progress has been done in this direction, with consensus reached on many key issues. I will summarize the main outcomes; the minutes are available at https://github.com/xproc/Workshop-2017-09/wiki/Agenda-and-Minutes.

Simplified and streamlined language

  • The actual unnecessary distinction between a pipeline and a step will be removed. (It turns out that the current definition of a pipeline makes it so strict that nobody is actually using it.)
  • The definition of a port and the use of a port will use different names. (This often confused beginners.)
  • Non XML documents will become first-class citizens in XProc 3.0 and treated exactly as XML documents.
  • The well known try/catch/finally construct will be introduced.

Run-time evaluation and extension

  • A new eval step will be introduced to run dynamically created pipelines.
  • User functions written in XPath, XSLT and XQuery will be usable in all fields where XPath can be used.

Diagnostic and debugging

  • Steps will be able to output side-information on the diagnostic, forwarded to stderr by default.
  • Implementation will provide a way to dump all the temporary documents produced by the intermediate steps.
  • A separate specification will standardize error reporting (so that XML editors like oXygen will be able to highlight where the problem occurred).

Plenty of interesting stuff, isn’t it? If you are interested in the development of XProc 3.0 or XProc in general, please participate in the discussions held on the XProc mailing list, join the W3C Community group, and suggest improvements on the XProc developement website.

See you at the next XProc meeting!

Open source at CCeH in 2016

Welcome to the annual CCeH report on our contributions to the open source world!

We at the Cologne Center for eHumanities always turn to open source software for our DH projects. Not only because they are free (as in beer) and free (as in speech) but also because of the marvelous communities that have formed around many of these free and open source projects. Our way to say thank you to these communities is to give back to their projects.

In 2014 we started a conscious effort to develop in the open and be good open source citizens. In 2015 we reported on our first contributions to the free software components that we use in our projects. We are happy to report in 2016 CCeH has contributed even more and to many more projects.

Our own DH projects

The number of projects that are available on our GitHub space https://github.com/cceh has grown a lot in 2016.

Some of the project repositories we published in the last year:

We are also setting up a GitLab installation to better integrate collaborators from other universities, private foundations and research partners. Stay tuned for more news on this front.

Improvements to other projects

Sometimes a program does 99% of what we need. Instead of just complaining about the missing 1%, we do our best to contribute the missing functionalities or fixing some incorrect behavior. Contributing with improvements to open source projects is also as a sign of gratitude towards the volunteers that work daily to improve and to maintain it.

During 2016 we contributed to the collation tool CollateX, fixing some subtle errors and making it work better in UNIX workflows.[1,2,3] We also provided patches to make the eXist XML database understand and process correctly the unconventional range of dates used in Archeology.[4,5]

We also contributed small improvement to the Rack web server[6,7] and the W3C Web Audio specifications.[8]

Bug reports

Nobody is perfect. Every software has a problem or two. In case we stumble upon one of these problems, we go at great lengths to report it in the best way possible, spending time to understand its root cause and to gather all the details that the maintainers of the project will need to fix the problem.

In 2016 we reported many conformance and performance issues to the widely used eXist XML database.[9,10,11,12,13,14] In another XML database we use, BaseX, we pointed out some possible improvement to the way it is run on servers.[15,16]

Big and well-known projects are not outside our radar. For example, we have found out and reported that Wikipedia was involuntarily producing pages that were not well formed XHTML and could not be analyzed using standard XML tools.[17] Thankfully that has been fixed in a couple of days.

It is nice to observe how our bug reports reflect the technologies predominately used at the CCeH: XML and TEI[18], web servers[19], browsers [20] and WordPress[21].

2016 has been a fulfilling year. We will strive to make 2017 even better.

Open source at CCeH in 2015

At the Cologne Center for eHumanities we rely on plenty of open source components: programs, libraries, frameworks and so on. Having so many useful components available is great, but we do not want to be just passive users. We are eager to contribute back to the open source community.

In 2014/2015 we started a conscious effort to develop in the open and to be good open source citizens. Here is a small selection of things we have done so far. This is just the beginning.

Our own DH projects

A selection of our most recent projects, including some that are still in progress, are publicly available on our GitHub space: https://github.com/cceh/.

Reusable libraries

Some of the code we produce could be useful to other researches, so we extracted them from their main project and released them as separate projects. Do you need to filter, extract and publish bibliographic records from your Zotero collection? pybibgen[1] may be what you need. Do you want to automate eXist-db tasks using Gulp? gulp-exist[2] does just that.

Improvements to other projects

Giving back is important. Contributing bug fixes and new features is the best way to say thank you to an open source project. In 2015 and 2014 we contributed to many established projects we used. Our aim is to make the project even better for all other users. For example we provided a new way to generate IDs to Artefactual’s AtoM that made ingestion of millions of records a matter of seconds instead of hours, while generating better URLs at the same time [3]. We also reported other small problems and features [4,5,6,7,8]. In Saxon we pointed out two problems that once fixed made some XSLT run an order of magnitude faster [9,10,11,12]. We also produced detailed problem reports with test cases to eXist-db [13,14,15,16], Chromium [17] and other development tools [18,19,20].

These were our contributions for 2015. Let’s see what awaits us in 2016.

  1. https://github.com/cceh/pybibgen
  2. https://github.com/olvidalo/gulp-exist
  3. https://github.com/artefactual/atom/pull/187
  4. https://github.com/artefactual/atom/pull/32
  5. https://github.com/artefactual/atom/pull/41
  6. https://github.com/artefactual/atom/pull/54
  7. https://projects.artefactual.com/issues/7026
  8. https://projects.artefactual.com/issues/6785
  9. https://saxonica.plan.io/boards/3/topics/6209
  10. https://saxonica.plan.io/issues/2489
  11. https://saxonica.plan.io/boards/3/topics/6256
  12. https://saxonica.plan.io/issues/2565
  13. https://github.com/eXist-db/exist/issues/362
  14. https://github.com/eXist-db/exist/issues/426
  15. https://github.com/eXist-db/exist/issues/712
  16. https://github.com/eXist-db/exist/issues/811
  17. https://code.google.com/p/chromium/issues/detail?id=565225
  18. https://mailman.uni-konstanz.de/pipermail/basex-talk/2015-February/008185.html
  19. https://github.com/toggl/toggldesktop/pull/1196
  20. https://github.com/davidswelt/zot_bib_web/issues/1