---
title: "Linking Expat from another package"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Linking Expat from another package}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
```

## Who this is for

Read this if your package has C code that is written against **Expat itself** ---
it calls `XML_ParserCreate()`, `XML_GetBuffer()`, `XML_StopParser()` --- and you
want it to build without a system `libexpat`. Typically the code is vendored and
cannot be rewritten: `xlsxio`, which `zuxlsx` vendors, is that case.

If instead you are writing new C code that you control, **stop here and read
`vignette("streaming")`** first. The registered function table described there
is the better path for new code: it needs no `configure`, no linker flags, and
picks up a zuxml fix when zuxml is upgraded. This vignette is for the case where
that is not an option.

## The two modes

|                        | table (`zuxml_api`)              | archive (`libzuxml.a`)          |
|------------------------|----------------------------------|---------------------------------|
| `DESCRIPTION`          | `Imports:` **and** `LinkingTo:`  | `LinkingTo:` only               |
| `NAMESPACE`            | needs an `importFrom()`          | nothing                         |
| Header                 | `zuxml.h`, no Expat type in it   | `expat.h`, Expat's own API      |
| API style              | callbacks through a table        | everything Expat offers         |
| Symbols resolved       | at run time, `R_GetCCallable()`  | at link time, into your `.so`   |
| zuxml at run time      | installed **and** loadable       | not needed at all               |
| A zuxml fix reaches you| when zuxml is upgraded           | when *you* are reinstalled      |

The archive exists because the table cannot express a *pull*-style reader. A
reader that returns a row to its caller and resumes where it left off is built
on `XML_StopParser()`/`XML_ResumeParser()`, and there is no equivalent in
`zuxml.h`.

## What an installed zuxml gives you

```{r}
# system.file() returns "" outside an installed layout, and dir("") would then
# list the working directory rather than nothing.
installed <- function(what) {
  path <- system.file(what, package = "zuxml")
  if (nzchar(path)) dir(path) else "(not an installed layout)"
}

installed("include")
arch <- .Platform$r_arch
installed(if (nzchar(arch)) file.path("lib", arch) else "lib")
```

The archive installs under `lib${R_ARCH}`, as the shared object does under
`libs${R_ARCH}`. `R_ARCH` is empty on most platforms, which makes that plain
`lib`.

`LinkingTo: zuxml` puts that `include` directory on your compiler's path
(R appends it to `CLINK_CPPFLAGS`), so `#include <expat.h>` resolves with no
further work.

The headers are not kept under `inst/include/` in zuxml's sources. They are
copied out of the vendored tree at install time by `src/install.libs.R`, so the
header you compile against is by construction the one the objects in the
archive were compiled from.

## There is no `LinkingTo` for a library

`LinkingTo` handles headers and nothing else. For the archive you have to find
its directory yourself, and the only portable place to do that is a
`configure` script that writes `src/Makevars`.

The two shortcuts are both worse. `$(shell ...)` in `Makevars` forces
`SystemRequirements: GNU make`. An `Imports: zuxml` entry gets you nothing:
`Imports` is about loading a namespace at run time, which a statically linked
consumer never does.

### `configure`

```sh
#!/bin/sh
set -eu

: "${R_HOME:?configure must be run by R CMD INSTALL, which sets R_HOME}"
RSCRIPT="${R_HOME}/bin/Rscript"

# lib/<arch> first, then lib: right whether R sets an architecture or not,
# and whichever layout the installed zuxml uses.
lib_dir() {
  "${RSCRIPT}" --vanilla -e "arch <- .Platform\$r_arch; d <- if (nzchar(arch)) system.file('lib', arch, package = '$1') else ''; if (!nzchar(d)) d <- system.file('lib', package = '$1'); cat(d)"
}

ZUXML_LIB=$(lib_dir zuxml)

if [ -z "${ZUXML_LIB}" ] || [ ! -f "${ZUXML_LIB}/libzuxml.a" ]; then
  echo "configure: libzuxml.a was found in neither zuxml/lib/<arch> nor zuxml/lib." >&2
  exit 1
fi

sed -e "s|@ZUXML_LIB@|${ZUXML_LIB}|g" src/Makevars.in > src/Makevars
```

Check for the **file**, not for a version. An installed zuxml from before the
archive existed reports the same version as one that carries it, so its presence
on disk is the only honest test.

`configure.win` is the same script ending in `src/Makevars.win`, and a `cleanup`
script should remove both, since they are generated:

```sh
#!/bin/sh
rm -f src/Makevars src/Makevars.win
```

### `src/Makevars.in`

```make
PKG_CPPFLAGS = -DXML_STATIC
PKG_LIBS = '@ZUXML_LIB@/libzuxml.a'
```

Two details that are easy to get wrong and fail only on one platform:

* **`-DXML_STATIC` is recommended, not required.** It says what is true ---
  the library is linked statically --- and it keeps the headers from marking
  declarations `__declspec(dllimport)`. Stock `expat_external.h` does that
  only for Microsoft's compiler (`_MSC_VER`), which Rtools is not, so a build
  without the define links today. Keep it anyway: it costs nothing and
  survives a change of toolchain.
* **Quote the path.** It comes from `system.file()`, so it lives under the R
  library, and on Windows the user library sits under the profile directory ---
  a path with a space in it reaches the linker as two nonexistent arguments. R
  quotes its own `LinkingTo` include paths for the same reason.

### `DESCRIPTION`

```
LinkingTo: zuxml
```

That is all. No `Imports: zuxml`, and nothing in `NAMESPACE`.

## Which Expat you are getting

The archive holds the Expat objects zuxml's own shared object was linked from,
and **nothing else** --- no `zux_*` wrapper, no R glue, since either would be
dead weight or a duplicate symbol inside your shared object.

It is compiled with zuxml's feature policy, which is stricter than a distro
build. Your code inherits the compiled-in half of that policy:

* **`XML_GE 0`, and `XML_DTD` never defined.** General entities, parameter
  entities, external subsets and the external-entity machinery are compiled
  out, so XXE and entity amplification are impossible rather than disabled. In
  a document with no `DOCTYPE`, any entity reference other than the five
  built-ins and numeric character references is a parse error. With one,
  it may not be: see the next section.
* **Do not define `XML_GE=1` on your own command line.** `expat.h` would then
  declare `XML_SetBillionLaughsAttackProtection*()`, which the archive does not
  define, and you would get a link error. Those limiters are pointless here
  anyway --- the attack they bound is already compiled out.
* **`XML_NS 1`.** Namespace-aware parsing via `XML_ParserCreateNS()` works.
* **`XML_CONTEXT_BYTES 1024`**, so `XML_GetInputContext()` is available.
* **`XML_UNICODE` is not defined**, so `XML_Char` is plain `char` and input is
  UTF-8. `XML_LARGE_SIZE` and `XML_ATTR_INFO` are not defined either.

## What you do not inherit

The rest of zuxml's policy is not compiled into Expat. It lives in zuxml's own
parser, above Expat, which is exactly the code an archive consumer does not
call. So `XML_ParserCreate()` on the archive gives you none of it:

* **No `DOCTYPE` rejection.** zuxml refuses a `DOCTYPE` by default, and an
  internal subset always. That matters more than it looks. With `XML_GE 0`,
  Expat does not record entity declarations, so in a document that declares
  `<!ENTITY e "x">` in an internal subset, a reference `&e;` does not fail: it
  arrives in your character-data handler as the four characters `&e;`. The
  document is silently wrong. A `DOCTYPE` that names an external subset makes
  Expat skip undeclared references instead of failing on them.
* **No resource limits.** zuxml's `max_depth`, `max_nodes`, `max_attrs` and
  `max_text` are counted in its own handlers. Expat bounds none of them.

If your input can be hostile, install the same guards yourself. Rejecting a
`DOCTYPE` is one handler, which is what zuxml does in `src/zux_parser.c`:

```c
static void XMLCALL
reject_doctype(void *user, const XML_Char *name, const XML_Char *sysid,
               const XML_Char *pubid, int has_internal_subset) {
  XML_Parser p = (XML_Parser)user;
  (void)name; (void)sysid; (void)pubid; (void)has_internal_subset;
  /* Record why, then stop: XML_Parse() returns XML_STATUS_ERROR. */
  XML_StopParser(p, XML_FALSE);
}

/* after XML_ParserCreate(): */
XML_SetUserData(p, p);   /* or your own struct holding p */
XML_SetStartDoctypeDeclHandler(p, reject_doctype);
```

To accept a `DOCTYPE` but refuse an internal subset, as zuxml's
`allow_doctype = TRUE` does, stop only when `has_internal_subset` is non-zero.
Bound depth by counting in your start- and end-element handlers, and text by
counting in your character-data handler, stopping the parser the same way.

**Security fixes reach you only when you rebuild.** Your copy of Expat is
linked into your shared object. When zuxml updates its bundled Expat, for a
security fix or otherwise, your users get the fix only once your package is
rebuilt and re-released against the new zuxml. R does not rebuild `LinkingTo`
dependents on upgrade.

## Check that you actually linked it

On Linux and Windows a wrong `PKG_LIBS` fails at link time. **On macOS it does
not.** R links a package shared object with `-undefined dynamic_lookup`, so a
consumer that never linked the archive still builds, still loads, and still
parses --- against whatever Expat happens to be in the process, usually the
system one. Every behavioural test passes. The only reliable check is the
symbol table:

```sh
# Nothing may be left for the loader to resolve.
nm -u path/to/yourpkg.so | grep XML_

# And Expat has to actually be in there.
nm path/to/yourpkg.so | grep ' [TtDd] _*XML_' | wc -l
```

## A worked example

[`tools/zuxmltest/`][fixture] in zuxml's development sources is a complete,
minimal package in exactly this shape --- `configure`, `configure.win`,
`cleanup`, `src/Makevars.in`, and a `src/consume.c` that drives
`XML_GetBuffer`/`XML_ParseBuffer` with a suspend/resume round trip.
[`tools/run-downstream-check`][gate] installs it against a freshly built zuxml
and asserts all of the above, including that it still parses after zuxml has
been removed from the library path.

Both live in the git repository rather than in the installed package, since
`tools/` is not shipped.

[fixture]: https://github.com/pedrobtz/zuxml/tree/main/tools/zuxmltest
[gate]: https://github.com/pedrobtz/zuxml/blob/main/tools/run-downstream-check