Content-defined chunking, off by default and priced in the interface.
FastCDC picks boundaries from a rolling hash of the content, so inserting bytes shifts one chunk instead of every chunk after it. That is the one thing fixed-size framing cannot do, and it is why this exists.
It is 120 to 160 times slower than the fixed splitter and that is not a bug to optimise away.
Measured in pure PHP on the reference host: 4.29 MB/s reading byte by byte with ord(), 5.63 MB/s
over 64 KiB windows unpacked with unpack, against 683 MB/s for fixed framing plus a BLAKE2b per
frame. An 80 GB initial file capture would spend 5.9 hours of pure CPU here. PHP has no primitive
for a per-byte rolling hash, and the array form that would vectorise it costs roughly 16x the
string in memory, which exhausts a 128 MB limit at 8 MB of input.
So this is reached only from file.shift_detected, only when an administrator has turned it on for
a file type, only with a byte budget per cron run, and never for media or the operation stream.
Two normalisations matter and are easy to get wrong. The mask is widened for chunks below the target and narrowed above it, which is what makes the size distribution tight rather than exponential - that is the difference between FastCDC and the older Rabin approach, and it is where most of its speed comes from. And the minimum size is skipped without hashing, because hashing bytes that cannot produce a boundary is pure cost.
$gear : list|null
The gear table the rolling hash indexes.
Derived deterministically rather than shipped as a literal, so two hosts produce identical boundaries for identical content. A random table would make a chunk's identity depend on which machine wrote it, which would defeat deduplication across a restore.
chunk(mixed $stream, int $budget = 0) : \Traversable<int,array{offset: int, bytes: string}>
Splits a stream into chunks.
| mixed | $stream | An open, readable stream positioned where splitting should start. |
| int | $budget | Most bytes to read before stopping, so one cron run cannot be consumed by one file. Zero means no limit and is for a command an operator is watching. |
Each chunk's offset in the stream and its content, in order.
boundary(string $buffer, list$gear, int $minimum, int $maximum, int $hash) : int|null
Where the next chunk ends.
| string | $buffer | The bytes available. |
| list |
$gear | The gear table. |
| int | $minimum | Smallest chunk. |
| int | $maximum | Largest chunk. |
| int | $hash | The rolling hash, carried across calls. |
The chunk length, or NULL when the buffer holds no boundary and is not yet at the maximum, so the caller should read more before deciding.