\Drupal\strata\File MediaStore

Stores file blocks and the maps that reassemble them.

Blocks are content-addressed and shared across every file and every version that contains them, so the same image uploaded twice under two names costs one copy, and the second version of a video costs the blocks that changed.

Blocks are stored raw: not compressed, not delta coded, not framed. Media is already compressed

  • a JPEG, an MP4, a PDF - so a second pass spends CPU to add bytes, and the measured ratio on the incompressible part of the corpus is 1.00x. They ARE encrypted when a cipher is configured, because a file's contents are exactly what encryption is for.

Blocks live outside the site namespace. A block is addressed by the digest of its content, so two sites sharing a bucket share their identical media, which is the point of sharing a bucket. The maps that describe which blocks make up which file are per-site, because a path belongs to a site.

Summary

Methods
Properties
Constants
__construct
store
put
writeMap
has
block
map
reassemble
missing
key
flushCache
No public properties found
PREFIX
PROBE_BATCH
No protected methods found
No protected properties found
No protected constants found
No private methods found
present
provider
splitter
classes
No private constants found

Constant

PREFIX

PREFIX = 'media'

Key prefix blocks are stored under.

PROBE_BATCH

PROBE_BATCH = 1000

Blocks whose presence is checked in one listing rather than one head each.

A first capture of a large file asks about thousands of blocks, and one HEAD per block is one request per block. A listing answers for a whole shard at once.

Properties

$present

$present : array

Block addresses known to be present, so a run does not ask twice.

Type

array<string, true> —

$splitter

$splitter : \Drupal\strata\File\BlockSplitter

Type

BlockSplitter

Methods

__construct()

__construct(\Drupal\strata\Storage\StorageProviderInterface  $provider, \Drupal\strata\File\BlockSplitter  $splitter, \Drupal\strata\File\StorageClassPolicy  $classes = new \Drupal\strata\File\StorageClassPolicy()) : mixed

Constructs a store.

Parameters

\Drupal\strata\Storage\StorageProviderInterface $provider

Where blocks and maps are written.

\Drupal\strata\File\BlockSplitter $splitter

Splits a file into blocks.

\Drupal\strata\File\StorageClassPolicy $classes

Decides which storage class a block is written to.

Returns

mixed —

store()

store(string  $path, string|null  $storedAs = null) : array{map: \Drupal\strata\File\FileMap, key: string, uploaded: int, skipped: int, bytes: int}

Stores a file, uploading only the blocks that are not already there.

Parameters

string $path

Path to the file on disk.

string|null $storedAs

The path to record in the map, when it differs from where the file sits on disk - a managed file's URI rather than its resolved location. NULL records the path as given.

Throws

\RuntimeException

When the file cannot be read or a block cannot be written.

Returns

array{map: \Drupal\strata\File\FileMap, key: string, uploaded: int, skipped: int, bytes: int} —

The map, the key it was written to, how many blocks were uploaded, how many were already present, and how many bytes went out.

put()

put(string  $hash, string  $bytes) : void

Writes one block.

Parameters

string $hash

Its content address.

string $bytes

Its content.

Throws

\RuntimeException

When the write fails.

Returns

void —

writeMap()

writeMap(\Drupal\strata\File\FileMap  $map) : string

Writes a map.

Parameters

\Drupal\strata\File\FileMap $map

The map.

Throws

\RuntimeException

When the write fails.

Returns

string —

The key it was written to.

has()

has(string  $hash) : bool

Whether a block is already stored.

Parameters

string $hash

Its content address.

Returns

bool —

TRUE when the object is present.

block()

block(string  $hash) : string

Reads one block.

Parameters

string $hash

Its content address.

Throws

\RuntimeException

When the block is absent, or its bytes do not hash to the address it is filed under. A block that does not match its address is corruption, and reassembling a file out of it would produce a file that looks intact and is not.

Returns

string —

The block's content.

map()

map(string  $key) : \Drupal\strata\File\FileMap

Reads a map back.

Parameters

string $key

Its object key.

Throws

\RuntimeException

When the map is absent or unreadable.

Returns

\Drupal\strata\File\FileMap —

The map.

reassemble()

reassemble(\Drupal\strata\File\FileMap  $map, string  $destination) : int

Writes a file back out from its map.

Parameters

\Drupal\strata\File\FileMap $map

The version to reassemble.

string $destination

Where to write it.

Throws

\RuntimeException

When a block is missing or the destination cannot be written. A partly reassembled file is deleted rather than left behind, because a truncated video looks like a file and is not one.

Returns

int —

Bytes written.

missing()

missing(\Drupal\strata\File\FileMap  $map) : list<string>

Which of a map's blocks are missing from the store.

What a verify pass over the file realm reports, and what tells a map that cannot be restored from one that can.

Parameters

\Drupal\strata\File\FileMap $map

The version to check.

Returns

list

Addresses of the absent blocks, deduplicated.

key()

key(string  $hash) : string

The object key a block lives at.

Sharded two levels deep like every other content-addressed object, so no directory holds hundreds of thousands of entries.

Parameters

string $hash

The block's content address.

Returns

string —

The key.

flushCache()

flushCache() : void

Forgets which blocks are known to be present.

Called by a long-running command between files, so a capture of a large tree does not hold a digest per block it has seen.

Returns

void —