The first release.
bin_:
bin_layout() describes a fixed-size binary record in a
one-line specification (integers of 8 to 128 bits, f16,
bf16, f32, f64, complex
c64 and c128, bool, fixed bytes
and strings, padding, arrays, either byte order, C struct alignment);
bin_unpack() and bin_pack() read and write any
number of records in one call; bin_decode() and
bin_encode() convert whole vectors to and from bytes of one
type; bin_builder(), bin_put(),
bin_reserve(), bin_reset() and
bin_take() append to a growable byte buffer with a hard
cap; bin_size(); bin_hexdump() and
bin_diff(); and bin_info().bin_serialize() appends R’s serialization of an object
to a builder, without first allocating it as a raw vector;
bin_unserialize() reads one from a raw vector at a 0-based
offset; bin_hash_object() gives an XXH3 fingerprint of an
object’s serialization, without the header that records the R version,
streamed so that the serialization is never allocated (#25).NA with no bytes, a 64-bit integer that a double cannot
hold exactly, and a string that is not UTF-8 are each a classed error
naming the field and the 0-based record
(?zubin-conditions). That holds on write too: a string must
be valid UTF-8, unless it is marked "bytes", which is
written as its bytes, so that what
bin_unpack(encoding = "bytes") reads packs back (#44, #50).
A layout may not name a field a.1 beside an array field
a, whose data-frame columns those are (#73), and a
refhook that changes the builder
bin_serialize() writes into is an error rather than a
corrupt stream (#72).bin_encode() and a typed bin_put() write
every element of a matrix or array, as writeBin() does
(#79); a typed bin_put() of complex numbers or a byte list
checks for an interrupt every 64 MiB, as one of numbers does (#77).inst/include, used
through LinkingTo: zubin, zufast alone:
zubin/rw.h (typed reads and writes at any alignment and
byte order), zubin/cursor.h (checked sequential reads),
zubin/buf.h (a buffer with ownership flags, a hard cap and
checked growth, adopting and detaching malloc blocks, and
the public checked size arithmetic zb_size_add() and
zb_size_mul()), zubin/layout.h (the
specification parser and the field-major unpack and pack kernels), and
zubin-r.h (a buffer owned by an R external pointer, and
foreign memory handed to R with the function that releases it).
zubin/column.h holds what columnar formats need: bitmaps,
bit-packed booleans, validity bitmaps to and from R’s missing values,
and byte swapping; the layout kernels also read and write values in
their own width and the host’s order, and rw.h adds 128-bit
integers and reads at a width chosen at run time. Every header but
zubin-r.h may be used from any thread, as long as no two
threads share a buffer or cursor (#26). Source compatibility within
major version 1; see the c-api article on the package
website.zubin-r.h also points R’s serialization at these
containers: zb_serialize() into a buffer,
zb_unserialize() from a cursor, and
zb_serialize_to_sink() through any sink, as a block
pipeline consumes it (#25). R’s own errors reach the caller unchanged; a
stream that runs out and a builder that cannot grow are statuses.Variable-length fields, a cursor-style reader from R, byte search and
splitting, zero-copy views, memory-mapped files, connections, bitfields
and the integers of zuint as an int64 mode are planned,
each with the condition that admits it, in .agents/next.md
in the source repository.