\Drupal\strata\Capture Reconciler

Proves capture is complete, and closes the gap when it is not.

Every other part of the capture path records a change it was told about. This one goes looking for changes nobody told anyone about: a module writing straight to its own table, a query that ran before the statement tap was enabled, a migration executed outside Drupal entirely. From inside the capture path all three are invisible, and a backup that cannot detect its own gaps is a backup nobody should trust.

The method is a bounded fingerprint per table, compared against the last one stored. A table whose fingerprint moved while the journal and the commit log hold no operation for it has drifted, which raises watermark.drift and, when asked, captures the table's current rows so the gap closes.

What this covers and what it does not. Row count, highest key and highest changed-timestamp are exact. The sampled digest is a sample: a change confined to rows outside the sample is not seen. So a clean pass means "no drift detected over what was measured", which is why the finding is a warning that names the table rather than a silent repair, and why the sample size is a setting. A table with no single-column primary key gets count and digest only.

Summary

Methods
Properties
Constants
__construct
reconcile
tables
observe
stored
store
forget
No public properties found
TABLE
DEFAULT_SAMPLE
CHANGED_COLUMNS
CAPTURE_LIMIT
JOURNAL_SCAN
No protected methods found
No protected properties found
No protected constants found
highest
digest
columns
orderColumn
identityColumn
changedColumn
wasCaptured
captureRows
appendRow
raise
database
journal
scope
tripwires
ledger
logger
sampleSize
No private constants found

Constant

TABLE

TABLE = 'strata_watermark'

The table watermarks are stored in.

DEFAULT_SAMPLE

DEFAULT_SAMPLE = 200

How many rows the sampled digest covers by default.

A bound on the cost of a pass rather than a statistical choice. At 200 rows over a few hundred tables a pass reads tens of thousands of rows, which is a fraction of one page render.

CHANGED_COLUMNS

CHANGED_COLUMNS = ['changed', 'updated', 'timestamp', 'created']

Column names treated as a changed timestamp, in order of preference.

CAPTURE_LIMIT

CAPTURE_LIMIT = 500

Most rows to capture from one drifted table in one pass.

Closing a gap must not become the thing that blows the cron window. A table with more drifted rows than this keeps its finding open and is picked up again next pass.

JOURNAL_SCAN

JOURNAL_SCAN = 10000

How much of the journal is scanned when asking whether a table was captured.

A pass runs after a flush, so the journal holds at most one window. This is a bound in case it does not: a backlog deeper than this makes the answer "not captured", which errs toward reporting drift that turns out to be accounted for rather than missing drift that is real.

Properties

$database

$database : \Drupal\Core\Database\Connection

Type

Connection

$journal

$journal : \Drupal\strata\Journal\JournalInterface

Type

JournalInterface

$scope

$scope : \Drupal\strata\Capture\CaptureScope

Type

CaptureScope

$tripwires

$tripwires : \Drupal\strata\Health\TripwireRegistry

Type

TripwireRegistry

$logger

$logger : \Psr\Log\LoggerInterface

Type

LoggerInterface

$sampleSize

$sampleSize : int

Type

int

Methods

__construct()

__construct(\Drupal\Core\Database\Connection  $database, \Drupal\strata\Journal\JournalInterface  $journal, \Drupal\strata\Capture\CaptureScope  $scope, \Drupal\strata\Health\TripwireRegistry  $tripwires, \Drupal\strata\Health\HealthLedgerInterface  $ledger, \Psr\Log\LoggerInterface  $logger, int  $sampleSize = \self::DEFAULT_SAMPLE) : mixed

Constructs a reconciler.

Parameters

\Drupal\Core\Database\Connection $database

The connection to inspect.

\Drupal\strata\Journal\JournalInterface $journal

Where captured rows are appended.

\Drupal\strata\Capture\CaptureScope $scope

Decides which tables are in scope.

\Drupal\strata\Health\TripwireRegistry $tripwires

Runs the drift check over each observation.

\Drupal\strata\Health\HealthLedgerInterface $ledger

Where drift is recorded.

\Psr\Log\LoggerInterface $logger

Records what a pass did.

int $sampleSize

How many rows the digest covers.

Returns

mixed —

reconcile()

reconcile(list  $tables = [], int  $limit = 100, bool  $capture = true) : \Drupal\strata\Capture\ReconcileReport

Compares every captured table against its stored watermark.

Parameters

list $tables

Tables to examine, or an empty list for every captured table.

int $limit

Most tables to examine in this pass.

bool $capture

TRUE to capture the rows of a drifted table, closing the gap. FALSE reports the drift and leaves it, which is what a dry run and a status page want.

Returns

\Drupal\strata\Capture\ReconcileReport —

What the pass found.

tables()

tables() : list<string>

Tables the reconciler watches.

Every table the connection holds that capture covers. Strata's own tables are excluded by the scope, and so is anything the site has told it to leave alone.

Returns

list

Table names, in a stable order so a bounded pass covers them evenly over time.

observe()

observe(string  $table) : \Drupal\strata\Capture\Watermark

Takes a fresh reading of one table.

Parameters

string $table

The table name.

Throws

\Drupal\Core\Database\DatabaseExceptionWrapper

When the table cannot be read, which the caller reports rather than swallows.

Returns

\Drupal\strata\Capture\Watermark —

The reading.

stored()

stored(string  $table) : \Drupal\strata\Capture\Watermark|null

The stored watermark for a table.

Parameters

string $table

The table name.

Returns

\Drupal\strata\Capture\Watermark|null —

The stored reading, or NULL when the table has never been read.

store()

store(\Drupal\strata\Capture\Watermark  $watermark) : void

Records a reading, replacing any earlier one.

Parameters

\Drupal\strata\Capture\Watermark $watermark

The reading.

Returns

void —

forget()

forget() : int

Forgets every stored reading.

Used after a restore, which moves every table at once and would otherwise report the whole schema as drifted on the next pass.

Returns

int —

How many readings were removed.

highest()

highest(string  $table, string  $column) : string|null

The highest value in a column.

Parameters

string $table

The table name.

string $column

The column.

Returns

string|null —

The value as a string, or NULL when the table is empty.

digest()

digest(string  $table, string|null  $key, list  $columns) : string

A digest over a bounded sample of rows.

Ordered by the primary key when there is one, so two readings sample the same rows and a difference means the contents moved rather than the order did. Without a key the order is whatever the database gives, so the digest is only meaningful for a table small enough that the sample covers all of it.

Parameters

string $table

The table name.

string|null $key

The primary key column, or NULL.

list $columns

The table's columns.

Returns

string —

The digest, or an empty string when no sample could be taken.

columns()

columns(string  $table) : list<string>

A table's columns, in a stable order.

Parameters

string $table

The table name.

Returns

list

Column names, sorted so the digest does not change with the schema's reported order.

orderColumn()

orderColumn(list  $columns) : string|null

The column a sample is ordered by, so two readings cover the same rows.

A heuristic, and only ever used for ordering: any column that sorts deterministically makes two readings comparable, so being wrong about which one is the key costs nothing here. Drupal names its keys predictably - id, or nid, uid, fid, tid - which is what the preference order follows.

Parameters

list $columns

The table's columns.

Returns

string|null —

The column, or NULL when the table has none.

identityColumn()

identityColumn(string  $table, list  $columns) : string|null

A single column whose values are unique, when one exists.

Used only when capturing the rows of a table that has already drifted, because a row needs an identity before it can be a restorable subject and a guess would give two different rows the same subject. Verified rather than assumed: one aggregate query per candidate, on a table the pass has already decided is worth the work.

Parameters

string $table

The table name.

list $columns

The table's columns.

Returns

string|null —

The column, or NULL when no single column identifies a row.

changedColumn()

changedColumn(list  $columns) : string|null

The column that records when a row last changed, when there is one.

Parameters

list $columns

The table's columns.

Returns

string|null —

The column, or NULL.

wasCaptured()

wasCaptured(string  $table, int  $since) : bool

Whether an operation was captured for a table since a given time.

Read from the journal, which holds what has been captured and not yet sealed. A window already sealed into a commit is not consulted: the reconciler runs on cron after a flush, so anything sealed was seen, and re-reading the commit log per table would cost a request per table.

Parameters

string $table

The table name.

int $since

Unix timestamp of the previous reading.

Returns

bool —

TRUE when something was captured for this table.

captureRows()

captureRows(string  $table, \Drupal\strata\Capture\Watermark  $stored, list  $problems) : int

Captures the rows of a drifted table, so the gap closes.

Only rows the reading can identify are captured: the table needs a single-column primary key, because a row with no identity has no subject to be restored into. A table without one keeps its finding open, which is the honest outcome - the drift is real and this cannot fix it.

Parameters

string $table

The table name.

\Drupal\strata\Capture\Watermark $stored

The previous reading, used to capture only what moved past it where possible.

list $problems

Collects a line when the capture itself fails.

Returns

int —

How many rows were captured.

appendRow()

appendRow(string  $table, string  $key, array  $row, list  $problems) : bool

Appends one row as a captured operation.

A row holding bytes that are not valid UTF-8 - a BLOB column, a binary primary key - cannot be described in JSON, and the reading's fresh watermark is already stored by the time this runs. So a row that cannot be encoded is reported rather than appended: appending (string) json_encode($row) would file an operation whose payload is empty, whose address is Hash::of('') and whose length is zero, and the drift would never be looked at again.

Parameters

string $table

The table name.

string $key

The primary key column.

array $row

The row.

list $problems

Collects a line when the row cannot be captured.

Returns

bool —

TRUE when the row was appended.

raise()

raise(string  $table, string  $description) : void

Records drift.

Parameters

string $table

The table that drifted.

string $description

What moved.

Returns

void —