The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.

zubin

R-CMD-check coverage Lifecycle: experimental

Every binary format, from an image header to a market data feed to a C struct dumped to disk, is specified the same way: a table of fields, each with a name, a type, a width and an offset. Base R reads such data one readBin() call at a time, loops over the records itself, has no unsigned 32-bit integer and no half-precision float, and reads an 8-byte integer silently wrong once it outgrows 32 bits.

zubin lets you write the table down once, as a layout, and then read or write any number of records in one call.

Installation

zubin is not on CRAN yet. Once it is, install it with:

install.packages("zubin")

Until then, and for the development version, install it from GitHub:

# install.packages("pak")
pak::pak("pedrobtz/zubin")

Usage

library(zubin)

Read a real header. A PNG file opens with an 8-byte signature and then an IHDR chunk, big-endian, as its specification lays it out. That table, written down, reads the image’s size from the file R just wrote:

png_file <- tempfile(fileext = ".png")
png(png_file, width = 640, height = 480)
plot(pressure)
invisible(dev.off())

ihdr <- bin_layout(">x8 length:u32 type:s4 width:u32 height:u32 depth:u8 colour:u8")
bin_unpack(readBin(png_file, "raw", 64), ihdr, n = 1)
#>   length type width height depth colour
#> 1     13 IHDR   640    480     8      6

The types base R lacks. readBin() has no unsigned 32-bit integers, so it warns and reads four ff bytes as -1, and it has no half floats. zubin decodes both, in either byte order:

x <- as.raw(c(0xff, 0xff, 0xff, 0xff))
readBin(x, "integer", size = 4, signed = FALSE)
#> Warning in readBin(x, "integer", size = 4, signed = FALSE): 'signed = FALSE' is
#> only valid for integers of sizes 1 and 2
#> [1] -1
bin_decode(x, "u32")
#> [1] 4294967295

# half-precision floats, as machine-learning weights are stored
bin_decode(as.raw(c(0x00, 0x3c, 0x00, 0xc0, 0x00, 0x7c)), "f16")
#> [1]   1  -2 Inf

Exact, or an error. readBin() reads the 8-byte integer 2^32 as 0, without a word. zubin reads it exactly, and refuses a value above 2^53, which has no exact double, naming the record rather than handing back a neighbouring number (int64 = "integer64" returns the exact bits instead):

counts <- bin_encode(c(2^32, 2^53 + 2), "u64")
readBin(counts, "integer", size = 8, n = 2)
#> [1] 0 2
bin_decode(counts[1:8], "u64")
#> [1] 4294967296
bin_decode(counts, "u64")
#> Error in `bin_decode()`:
#> ! The value of record 1 has no exact R value.

Take a number apart. A double is a sign bit, an 11-bit exponent and a 52-bit fraction. Encoded and read back as two unsigned words, its fields come out exactly, and 0.1 turns out to be 7205759403792794 / 2^56, a little more than a tenth:

w <- bin_unpack(bin_encode(c(1, 0.1, -2.5), "f64", endian = "big"), ">hi:u32 lo:u32")
bits <- function(v, from, width) (v %/% 2^from) %% 2^width
parts <- data.frame(sign = bits(w$hi, 31, 1), exponent = bits(w$hi, 20, 11) - 1023,
                    fraction = bits(w$hi, 0, 20) * 2^32 + w$lo)
format(parts, scientific = FALSE)
#>   sign exponent         fraction
#> 1    0        0                0
#> 2    0       -4 2702159776422298
#> 3    1        1 1125899906842624
(2^52 + parts$fraction[2]) / 2^56 == 0.1
#> [1] TRUE

A million records in one call. Packing a data frame and unpacking it again is one call each way, and the round trip is exact:

set.seed(1)
rec <- bin_layout("<id:i32 price:f64 qty:i16 x2")
trades <- data.frame(id = 1:1e6, price = round(runif(1e6, 90, 110), 2),
                     qty = sample(1:500, 1e6, replace = TRUE))
bytes <- bin_pack(rec, trades)
length(bytes)
#> [1] 16000000
identical(bin_unpack(bytes, rec), trades)
#> [1] TRUE

Mixed byte orders: a shapefile. The .shp part of a shapefile mixes byte orders inside one record: its file code and length are big-endian, everything after them little-endian. A .be suffix overrides the layout’s order for one field, so each record is still a single layout. Here the quakes data become a point shapefile, written with a builder, as the ESRI specification lays it out:

shp_header <- bin_layout("<code:i32.be x20 words:i32.be version:i32 type:i32
                          xmin:f64 ymin:f64 xmax:f64 ymax:f64 x32")
point <- bin_layout("<id:i32.be words:i32.be type:i32 x:f64 y:f64")

n <- nrow(quakes)
b <- bin_builder()
bin_put(b, bin_pack(shp_header, code = 9994, words = (100 + 28 * n) / 2,
                    version = 1000, type = 1,
                    xmin = min(quakes$long), ymin = min(quakes$lat),
                    xmax = max(quakes$long), ymax = max(quakes$lat)))
bin_put(b, bin_pack(point, id = seq_len(n), words = 10, type = 1,
                    x = quakes$long, y = quakes$lat))
shp <- bin_take(b)
bin_hexdump(shp, n = 48)
#> 00000000: 0000 270a 0000 0000 0000 0000 0000 0000  ..'.............
#> 00000010: 0000 0000 0000 0000 0000 36e2 e803 0000  ..........6.....
#> 00000020: 0100 0000 3d0a d7a3 70b5 6440 ec51 b81e  ....=...p.d@.Q..

The file code 9994 opens the file big-endian (0000 270a), and the version 1000 at offset 28 is little-endian (e803 0000). Reading it back is one call for the header and one for every point after it:

bin_unpack(shp, shp_header, n = 1)
#>   code words version type   xmin   ymin   xmax   ymax
#> 1 9994 14050    1000    1 165.67 -38.59 188.13 -10.72
pts <- bin_unpack(shp, point, offset = 100)
head(pts, 3)
#>   id words type      x      y
#> 1  1    10    1 181.62 -20.42
#> 2  2    10    1 181.03 -20.62
#> 3  3    10    1 184.10 -26.00
identical(pts$x, quakes$long)
#> [1] TRUE

Every function is prefixed bin_, and offsets are 0-based everywhere, as in every format specification.

Learn more

zubin’s integer loads and stores, half-float conversions and UTF-8 validation come from zufast. Python’s struct module and the structured dtypes of NumPy are the nearest equivalents in other languages.

These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.