XML Pipelines and XProc 3.0: Report of the WG Meeting in Aachen

Last week (14th and 15th of September 2017) a meeting of the XProc 3.0 working group took place in Aachen, organized by Achim Berndzen of xml-project and Gerrit Imsieke of le-tex and hosted by LOGOI.

The meeting was extremely successful, consensus has been reached on many topics and important roadblocks have been overcome. I will tell you about what the WG accomplished in a second. Before that allow me to introduce XProc, XML pipelines and explain why they are useful. (If you already know all this stuff, skip directly to the XProc 3 section, that’s OK. :))

XML pipelines? What are you talking about?

Everybody who has worked with XML knows that real-world applications are always born as simple transformations (“I’ll just convert this XML to HTML with XSLT”) but quickly develop into a big tangled web of unreadable code as soon as you have to deal the inevitable…

small mistakes in the input (“wait, why is there a <p> inside a <em>?”),
flaws in the receiving applications (“let’s have a separate output for Internet Explorer 6, so that the poor students access this from the library”) or
requests from the project collaborators (“could you make a summary version with only one sentence per chapter?”).

Addressing all these needs can be done, but doing it by adding fixes on top of fixes on the original core transformation is a nightmare in terms of maintenance and readability.

Small steps and scripts

A better way to solve all these issues is splitting monolithic transformations into smaller pieces, or steps. (More about how our experience at the CCeH in splitting complicated transformations into focused steps in a future article.)

Now that you have all these steps, how do you transform the input into the output in practice?

Are you going to run each step manually, clicking around in your XML editor? I hope not.

A much better way to run this split transformation is to create a shell script that takes the input file, applies the first step (fix the small mistakes), then the second (transform into HTML) and then, if requested, the third (uglify HTML to make it IE6 compatible).

Such a script would work just fine but it has many problems:

Either you hardcode how to invoke the XSLT processor or you have to write an abstraction layer that allows you to call other XSLT processors.
Requires a working Bash shell environment (not that easy to get on Windows).
Does not provide any kind of validation of the intermediate results.
Requires a deserialization/serialization cycle for each step.
Gets rapidly very complex as soon as other steps, conditional steps and loops are added.
Works only on a single document.

We could address all these problems ourselves making a better script. Or we could avoid reinventing the wheel and make use of XProc and write a declarative XML pipeline.

Enter XML pipelines and XProc

XProc is a language for writing declarative XML pipelines.

An XML pipeline is a series of steps though which an XML documents flow, just as in the shell script in the previous example. However, in contrast with a shell script, XProc pipelines are:

Declarative: you state what you want and the XProc interpreter chooses the right tools. (A PDF transformation? Let’s use Apache FOP. An XSLT Transformation? Let’s use libxslt. Oh, are we running inside oXygen? Let’s use the internal Saxon-EE engine then.)
Portable: pipelines can run wherever there is a XProc interpreter: Linux, Windows, Mac OS, you name it.
Specialized for XML: documents are not deserialized and serialized in each step.
Can have more than one input and produce more than one output.
Easily extend to intricate pipelines with loops and parallel branches.

An example pipeline looks like the following

<p:pipeline xmlns:p="http://www.w3.org/ns/xproc" version="1.0">
    <p:xslt>
        <p:input port="stylesheet">
            <p:document href="fix-mistakes.xsl"/>
        </p:input>
    </p:xslt>

    <p:xslt>
        <p:input port="stylesheet">
            <p:document href="convert-doc.xsl"/>
        </p:input>
    </p:xslt>

    <p:xslt use-when="p:system-property('ie6-compatible') = 'true'">
        <p:input port="stylesheet">
            <p:document href="make-ie6-compatible.xsl"/>
        </p:input>
    </p:xslt>
</p:pipeline>

XProc 3.0

XProc 3.0 is the upcoming version of XProc. The original XProc 1 specifications have been published in 2010 by the W3C and since then users and implementers have found small problems, inconsistencies as well as ergonomic issues that make writing XProc pipelines harder than it should.

The focus of XProc 3 is simplifying the language, making implementations behave in more sensible way by default and making it possible to process non-XML documents (think LaTeX or graphic files).

During last week’s working group meeting in Aachen plenty of progress has been done in this direction, with consensus reached on many key issues. I will summarize the main outcomes; the minutes are available at https://github.com/xproc/Workshop-2017-09/wiki/Agenda-and-Minutes.

Simplified and streamlined language

The actual unnecessary distinction between a pipeline and a step will be removed. (It turns out that the current definition of a pipeline makes it so strict that nobody is actually using it.)
The definition of a port and the use of a port will use different names. (This often confused beginners.)
Non XML documents will become first-class citizens in XProc 3.0 and treated exactly as XML documents.
The well known try/catch/finally construct will be introduced.

Run-time evaluation and extension

A new eval step will be introduced to run dynamically created pipelines.
User functions written in XPath, XSLT and XQuery will be usable in all fields where XPath can be used.

Diagnostic and debugging

Steps will be able to output side-information on the diagnostic, forwarded to stderr by default.
Implementation will provide a way to dump all the temporary documents produced by the intermediate steps.
A separate specification will standardize error reporting (so that XML editors like oXygen will be able to highlight where the problem occurred).

Plenty of interesting stuff, isn’t it? If you are interested in the development of XProc 3.0 or XProc in general, please participate in the discussions held on the XProc mailing list, join the W3C Community group, and suggest improvements on the XProc developement website.

See you at the next XProc meeting!