The hardware and bandwidth for this mirror is donated by METANET, the Webhosting and Full Service-Cloud Provider.
If you wish to report a bug, or if you are interested in having us mirror your free-software or open-source project, please feel free to contact us at mirror[@]metanet.ch.
Every binary format, from an image header to a market data feed to a
C struct dumped to disk, is specified the same way: a table of fields,
each with a name, a type, a width and an offset. Base R reads such data
one readBin() call at a time, loops over the records
itself, has no unsigned 32-bit integer and no half-precision float, and
reads an 8-byte integer silently wrong once it outgrows 32 bits.
zubin lets you write the table down once, as a layout, and then read or write any number of records in one call.
"<id:u32 ts:i64 price:f64 side:u8", and
bin_unpack() turns a raw vector into a data frame of every
record in it, one tight loop per field.f16, bf16,
f32 and f64; complex numbers; booleans; fixed
bytes and strings; arrays; padding; either byte order, per field; and C
struct alignment.bin_pack() and
bin_encode() are the inverses, and a byte builder appends
without quadratic copying, up to a hard cap.
bin_serialize() writes an R object into one without first
making a raw vector, and bin_hash_object() fingerprints an
object without making its serialisation at all.LinkingTo alone.zubin is not on CRAN yet. Once it is, install it with:
install.packages("zubin")Until then, and for the development version, install it from GitHub:
# install.packages("pak")
pak::pak("pedrobtz/zubin")library(zubin)Read a real header. A PNG file opens with an 8-byte
signature and then an IHDR chunk, big-endian, as its
specification lays it out. That table, written down, reads the image’s
size from the file R just wrote:
png_file <- tempfile(fileext = ".png")
png(png_file, width = 640, height = 480)
plot(pressure)
invisible(dev.off())
ihdr <- bin_layout(">x8 length:u32 type:s4 width:u32 height:u32 depth:u8 colour:u8")
bin_unpack(readBin(png_file, "raw", 64), ihdr, n = 1)
#> length type width height depth colour
#> 1 13 IHDR 640 480 8 6The types base R lacks. readBin() has
no unsigned 32-bit integers, so it warns and reads four ff
bytes as -1, and it has no half floats. zubin decodes both, in either
byte order:
x <- as.raw(c(0xff, 0xff, 0xff, 0xff))
readBin(x, "integer", size = 4, signed = FALSE)
#> Warning in readBin(x, "integer", size = 4, signed = FALSE): 'signed = FALSE' is
#> only valid for integers of sizes 1 and 2
#> [1] -1
bin_decode(x, "u32")
#> [1] 4294967295
# half-precision floats, as machine-learning weights are stored
bin_decode(as.raw(c(0x00, 0x3c, 0x00, 0xc0, 0x00, 0x7c)), "f16")
#> [1] 1 -2 InfExact, or an error. readBin() reads the
8-byte integer 2^32 as 0, without a word. zubin reads it exactly, and
refuses a value above 2^53, which has no exact double, naming the record
rather than handing back a neighbouring number
(int64 = "integer64" returns the exact bits instead):
counts <- bin_encode(c(2^32, 2^53 + 2), "u64")
readBin(counts, "integer", size = 8, n = 2)
#> [1] 0 2
bin_decode(counts[1:8], "u64")
#> [1] 4294967296
bin_decode(counts, "u64")
#> Error in `bin_decode()`:
#> ! The value of record 1 has no exact R value.Take a number apart. A double is a sign bit, an
11-bit exponent and a 52-bit fraction. Encoded and read back as two
unsigned words, its fields come out exactly, and 0.1 turns
out to be 7205759403792794 / 2^56, a little more than a
tenth:
w <- bin_unpack(bin_encode(c(1, 0.1, -2.5), "f64", endian = "big"), ">hi:u32 lo:u32")
bits <- function(v, from, width) (v %/% 2^from) %% 2^width
parts <- data.frame(sign = bits(w$hi, 31, 1), exponent = bits(w$hi, 20, 11) - 1023,
fraction = bits(w$hi, 0, 20) * 2^32 + w$lo)
format(parts, scientific = FALSE)
#> sign exponent fraction
#> 1 0 0 0
#> 2 0 -4 2702159776422298
#> 3 1 1 1125899906842624
(2^52 + parts$fraction[2]) / 2^56 == 0.1
#> [1] TRUEA million records in one call. Packing a data frame and unpacking it again is one call each way, and the round trip is exact:
set.seed(1)
rec <- bin_layout("<id:i32 price:f64 qty:i16 x2")
trades <- data.frame(id = 1:1e6, price = round(runif(1e6, 90, 110), 2),
qty = sample(1:500, 1e6, replace = TRUE))
bytes <- bin_pack(rec, trades)
length(bytes)
#> [1] 16000000
identical(bin_unpack(bytes, rec), trades)
#> [1] TRUEMixed byte orders: a shapefile. The
.shp part of a shapefile mixes byte orders inside one
record: its file code and length are big-endian, everything after them
little-endian. A .be suffix overrides the layout’s order
for one field, so each record is still a single layout. Here the
quakes data become a point shapefile, written with a
builder, as the ESRI specification lays it out:
shp_header <- bin_layout("<code:i32.be x20 words:i32.be version:i32 type:i32
xmin:f64 ymin:f64 xmax:f64 ymax:f64 x32")
point <- bin_layout("<id:i32.be words:i32.be type:i32 x:f64 y:f64")
n <- nrow(quakes)
b <- bin_builder()
bin_put(b, bin_pack(shp_header, code = 9994, words = (100 + 28 * n) / 2,
version = 1000, type = 1,
xmin = min(quakes$long), ymin = min(quakes$lat),
xmax = max(quakes$long), ymax = max(quakes$lat)))
bin_put(b, bin_pack(point, id = seq_len(n), words = 10, type = 1,
x = quakes$long, y = quakes$lat))
shp <- bin_take(b)
bin_hexdump(shp, n = 48)
#> 00000000: 0000 270a 0000 0000 0000 0000 0000 0000 ..'.............
#> 00000010: 0000 0000 0000 0000 0000 36e2 e803 0000 ..........6.....
#> 00000020: 0100 0000 3d0a d7a3 70b5 6440 ec51 b81e ....=...p.d@.Q..The file code 9994 opens the file big-endian
(0000 270a), and the version 1000 at offset 28 is
little-endian (e803 0000). Reading it back is one call for
the header and one for every point after it:
bin_unpack(shp, shp_header, n = 1)
#> code words version type xmin ymin xmax ymax
#> 1 9994 14050 1000 1 165.67 -38.59 188.13 -10.72
pts <- bin_unpack(shp, point, offset = 100)
head(pts, 3)
#> id words type x y
#> 1 1 10 1 181.62 -20.42
#> 2 2 10 1 181.03 -20.62
#> 3 3 10 1 184.10 -26.00
identical(pts$x, quakes$long)
#> [1] TRUEEvery function is prefixed bin_, and offsets are 0-based
everywhere, as in every format specification.
LinkingTo recipe, the
cursor, the record kernels, and buffers owned by R.?bin_layout and
the error classes under ?zubin-conditions.zubin’s integer loads and stores, half-float conversions and UTF-8
validation come from zufast. Python’s struct
module and the structured dtypes of NumPy are the nearest equivalents in
other languages.
These binaries (installable software) and packages are in development.
They may not be fully stable and should be used with caution. We make no claims about them.