Persistent mmap-backed representation of the provenance circuit. More...

#include "MMappedCircuit.h"

Collaboration diagram for MMappedCircuit:

[legend]

Classes
struct	Check
	What `provsql.check_store()` reports about a store. More...
struct	Counts
	The size of a store, in the three units that matter. More...

Public Types
enum class	SetProbResult { NotProbGate , Written , Unchanged , AlreadySet }
	Outcome of `setProb`. More...
enum class	SetAnnotationResult { NoSuchGate , Written , Unchanged , AlreadySet }
	Outcome of `setInfos` or `setExtra`. More...

Public Member Functions
	MMappedCircuit (Oid db_oid, Oid db_tablespace, bool read_only=false)
	Open all four mmap backing files for the given database.
	~MMappedCircuit ()
	Sync all backing files before destruction.
void	createGate (pg_uuid_t token, gate_type type, const std::vector< pg_uuid_t > &children)
	Persist a new gate to the mmap store.
SetAnnotationResult	setInfos (pg_uuid_t token, unsigned info1, unsigned info2, std::pair< unsigned, unsigned > *existing=nullptr)
	Write the `info1` / `info2` annotations of a gate, once.
SetAnnotationResult	setExtra (pg_uuid_t token, const std::string &s, std::string *existing=nullptr)
	Attach a variable-length string annotation to a gate, once.
SetProbResult	setProb (pg_uuid_t token, double prob, double *existing=nullptr)
	Write a gate's probability, once.
bool	hasProb (pg_uuid_t token, double *prob) const
	Report whether `token` names a gate that holds a probability.
void	sync ()
	Flush all backing files to disk with `msync()`.
void	flush ()
	Force every backing file to stable storage.
bool	uncleanShutdown () const
	Whether any backing file was found still marked open-for-writing when it was opened – the previous writer died mid-write.
Check	check () const
	Walk the store and report what does not add up.
unsigned long	mark (const std::vector< pg_uuid_t > &roots, std::vector< bool > &live) const
	Mark every gate reachable from `roots`.
Counts	counts () const
	This store's current size.
Counts	sweepInto (const std::vector< bool > &live, const std::string &mp, const std::string &gp, const std::string &wp, const std::string &ep) const
	Copy the gates flagged in `live` into a fresh set of files beside the current ones (suffix `".new"`).
bool	legacyProbabilities () const
	Whether this store's `gates` file predates the `NaN` unset-probability convention (see `GATES_VERSION`).
gate_type	getGateType (pg_uuid_t token) const
	Return the type of the gate identified by `token`.
std::vector< pg_uuid_t >	getChildren (pg_uuid_t token) const
	Return the child UUIDs of the gate identified by `token`.
double	getProb (pg_uuid_t token) const
	Return the probability an evaluation would use for `token`.
std::pair< unsigned, unsigned >	getInfos (pg_uuid_t token) const
	Return the `info1` / `info2` pair for the gate `token`.
std::string	getExtra (pg_uuid_t token) const
	Return the variable-length string annotation for gate `token`.
unsigned long	getNbGates () const
	Return the total number of gates stored in the circuit.
GenericCircuit	createGenericCircuit (pg_uuid_t token) const
	Build an in-memory `GenericCircuit` rooted at `token`.
GenericCircuit	createGenericCircuit (const std::vector< pg_uuid_t > &roots) const
	Build an in-memory `GenericCircuit` reachable from any of `roots`.

Static Public Member Functions
static std::string	storePath (Oid db_oid, Oid db_tablespace, const char *filename)
	Build the full path of a file in a database's store directory, the one `GetDatabasePath` resolves for `db_oid` in `db_tablespace`.

Static Public Attributes
static constexpr uint16_t	GATES_VERSION = 2
	Format version of the `gates` file this build writes.
static constexpr uint64_t	MAGIC_GATES
	8-byte magic constants identifying each mmap file type.
static constexpr uint64_t	MAGIC_WIRES
static constexpr uint64_t	MAGIC_MAPPING
static constexpr uint64_t	MAGIC_EXTRA

Private Member Functions
void	appendGate (pg_uuid_t token, gate_type type, const std::vector< pg_uuid_t > &children)
	Append a complete gate record, then publish `token` for it.
bool	hasProbAt (unsigned long idx) const
	Whether the gate record at index `idx` holds a written probability (see `GATES_VERSION` for the version-1 leniency).
	MMappedCircuit (const std::string &mp, const std::string &gp, const std::string &wp, const std::string &ep, bool read_only)
	Delegating constructor that accepts pre-built paths.

Private Attributes
MMappedUUIDHashTable	mapping
	UUID → gate-index hash table.
MMappedVector< GateInformation >	gates
	Gate metadata array.
MMappedVector< pg_uuid_t >	wires
	Flattened child UUID array.
MMappedVector< char >	extra
	Variable-length string data.

Static Private Attributes
static constexpr const char *	GATES_FILENAME ="provsql_gates.mmap"
	Backing file for `gates`.
static constexpr const char *	WIRES_FILENAME ="provsql_wires.mmap"
	Backing file for `wires`.
static constexpr const char *	MAPPING_FILENAME ="provsql_mapping.mmap"
	Backing file for `mapping`.
static constexpr const char *	EXTRA_FILENAME ="provsql_extra.mmap"
	Backing file for `extra`.

Detailed Description

Persistent mmap-backed representation of the provenance circuit.

MMappedCircuit is the single writer for circuit data; only the background worker should call its mutating methods. Reading methods may be called from any process that has mapped the files read-only.

Definition at line 91 of file MMappedCircuit.h.

Member Enumeration Documentation

◆ SetAnnotationResult

enum class MMappedCircuit::SetAnnotationResult

strong

Outcome of setInfos or setExtra.

Same discipline as setProb: an annotation is a fact appended to the gate, written once and idempotent on the same value.

Enumerator
NoSuchGate	The token names no gate.
Written	The gate had none and now holds the given annotation.
Unchanged	The gate already held exactly this annotation.
AlreadySet	The gate holds a different annotation; refused.

Definition at line 164 of file MMappedCircuit.h.

◆ SetProbResult

enum class MMappedCircuit::SetProbResult

strong

Outcome of setProb.

Written and Unchanged are both successes; only Written needs undoing if the writing transaction rolls back.

Enumerator
NotProbGate	The token names a gate that carries no probability.
Written	The gate had no probability and now holds the given one.
Unchanged	The gate already held exactly this probability.
AlreadySet	The gate holds a different probability; refused.

Definition at line 145 of file MMappedCircuit.h.

Constructor & Destructor Documentation

◆ MMappedCircuit() [1/2]

MMappedCircuit::MMappedCircuit	(	const std::string &	mp,
		const std::string &	gp,
		const std::string &	wp,
		const std::string &	ep,
		bool	read_only )

inlineprivate

Delegating constructor that accepts pre-built paths.

Definition at line 114 of file MMappedCircuit.h.

Here is the caller graph for this function:

◆ MMappedCircuit() [2/2]

MMappedCircuit::MMappedCircuit	(	Oid	db_oid,
		Oid	db_tablespace,
		bool	read_only = false )

Open all four mmap backing files for the given database.

Parameters

db_oid	OID of the target database.
db_tablespace	OID of that database's default tablespace; the files go in the directory `GetDatabasePath` resolves for the pair.
read_only	If `true`, all files are mapped read-only.

Definition at line 73 of file MMappedCircuit.cpp.

Here is the call graph for this function:

◆ ~MMappedCircuit()

MMappedCircuit::~MMappedCircuit ( )

inline

Sync all backing files before destruction.

Definition at line 196 of file MMappedCircuit.h.

Here is the call graph for this function:

Member Function Documentation

◆ appendGate()

void MMappedCircuit::appendGate	(	pg_uuid_t	token,
		gate_type	type,
		const std::vector< pg_uuid_t > &	children )

private

Append a complete gate record, then publish token for it.

Add a gate record and only then make it reachable under token.

The record has to be complete before the mapping entry that points at it exists: killed between the two, the store keeps an unreferenced record, which costs nothing, whereas the other order leaves a mapping entry indexing past the end of the record vector and every gate created afterwards resolves one slot off, for the rest of the store's life.

Definition at line 101 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ check()

MMappedCircuit::Check MMappedCircuit::check ( ) const

Walk the store and report what does not add up.

Definition at line 928 of file MMappedCircuit.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ counts()

Counts MMappedCircuit::counts ( ) const

inline

This store's current size.

Definition at line 343 of file MMappedCircuit.h.

Here is the caller graph for this function:

◆ createGate()

void MMappedCircuit::createGate	(	pg_uuid_t	token,
		gate_type	type,
		const std::vector< pg_uuid_t > &	children )

Persist a new gate to the mmap store.

Allocates a GateInformation record, appends the children to the wires vector, and records the UUID→index mapping. Existing gates with the same token are silently skipped.

Parameters

token	UUID identifying the new gate.
type	Gate type.
children	Ordered list of child gate UUIDs.

Definition at line 112 of file MMappedCircuit.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ createGenericCircuit() [1/2]

GenericCircuit MMappedCircuit::createGenericCircuit ( const std::vector< pg_uuid_t > & roots ) const

Build an in-memory GenericCircuit reachable from any of roots.

Multi-root variant of createGenericCircuit. Seeds the BFS with every UUID in roots so a shared subgraph reachable from more than one root is represented by a single gate_t (the GenericCircuit::setGate / getGate pair is idempotent on the UUID key). Used by getJointCircuit to load an RV's sub-DAG together with a conditioning gate that sits above it in the persisted DAG.

Parameters

roots UUIDs whose reachable closure to load. Order is irrelevant; identical UUIDs collapse via the std::set deduplication of the work list.

Returns: An in-memory GenericCircuit containing every gate reachable from any root.

Definition at line 1098 of file MMappedCircuit.cpp.

Here is the call graph for this function:

◆ createGenericCircuit() [2/2]

GenericCircuit MMappedCircuit::createGenericCircuit ( pg_uuid_t token ) const

Build an in-memory GenericCircuit rooted at token.

Performs a depth-first traversal of the mmap-backed circuit starting from token and copies all reachable gates and wires into a newly constructed GenericCircuit.

Parameters

token UUID of the root gate.

Returns: An in-memory GenericCircuit containing the sub-circuit.

Definition at line 1093 of file MMappedCircuit.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ flush()

void MMappedCircuit::flush ( )

Force every backing file to stable storage.

sync() pushes the regions' bytes into the files; this pushes the files out of the kernel's page cache, which is what a crash of the machine can otherwise lose.

Definition at line 910 of file MMappedCircuit.cpp.

◆ getChildren()

std::vector< pg_uuid_t > MMappedCircuit::getChildren ( pg_uuid_t token ) const

Return the child UUIDs of the gate identified by token.

Parameters

token UUID of the gate.

Returns: Ordered vector of child UUIDs.

Definition at line 160 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ getExtra()

std::string MMappedCircuit::getExtra ( pg_uuid_t token ) const

Return the variable-length string annotation for gate token.

Parameters

token UUID of the gate.

Returns: The stored string, or empty if none.

Definition at line 333 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ getGateType()

gate_type MMappedCircuit::getGateType ( pg_uuid_t token ) const

Return the type of the gate identified by token.

Parameters

token UUID of the gate.

Returns: The gate's type, or gate_input if not found (lazy default).

Definition at line 151 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ getInfos()

std::pair< unsigned, unsigned > MMappedCircuit::getInfos ( pg_uuid_t token ) const

Return the info1 / info2 pair for the gate token.

Parameters

token UUID of the gate.

Returns: Pair {info1, info2}, or {0,0} if not found.

Definition at line 322 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ getNbGates()

unsigned long MMappedCircuit::getNbGates ( ) const

inline

Return the total number of gates stored in the circuit.

Returns: Total gate count.

Definition at line 418 of file MMappedCircuit.h.

Here is the caller graph for this function:

◆ getProb()

double MMappedCircuit::getProb ( pg_uuid_t token ) const

Return the probability an evaluation would use for token.

A gate nobody has given a probability evaluates as certain, and a repaired row nobody has given one evaluates at the uniform weight of its repair_key block; either way this answers a number, not "unset". Use hasProb to tell the two apart.

Parameters

token UUID of the gate.

Returns: The probability; NaN when token names no gate, or one that carries no probability at all.

Definition at line 223 of file MMappedCircuit.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ hasProb()

bool MMappedCircuit::hasProb	(	pg_uuid_t	token,
		double *	prob ) const

Report whether token names a gate that holds a probability.

Parameters

token	UUID of the gate.
prob	On `true` return, the stored probability.

Returns: true when the gate exists, carries probabilities, and one has been written to it.

Definition at line 239 of file MMappedCircuit.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ hasProbAt()

bool MMappedCircuit::hasProbAt ( unsigned long idx ) const

private

Whether the gate record at index idx holds a written probability (see GATES_VERSION for the version-1 leniency).

Whether the record at idx holds a probability someone wrote.

A version-1 gates file cannot distinguish "written as 1.0" from "never written" – both are stored as 1.0 – so on such a file a 1.0 counts as unset and stays writable. See GATES_VERSION.

Definition at line 213 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ legacyProbabilities()

bool MMappedCircuit::legacyProbabilities ( ) const

inline

Whether this store's gates file predates the NaN unset-probability convention (see GATES_VERSION).

Definition at line 366 of file MMappedCircuit.h.

Here is the caller graph for this function:

◆ mark()

unsigned long MMappedCircuit::mark	(	const std::vector< pg_uuid_t > &	roots,
		std::vector< bool > &	live ) const

Mark every gate reachable from roots.

Parameters

roots	Token UUIDs to start from; unknown ones are ignored.
live	Filled with one flag per gate record.

Returns: The number of live gates.

Definition at line 959 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ setExtra()

MMappedCircuit::SetAnnotationResult MMappedCircuit::setExtra	(	pg_uuid_t	token,
		const std::string &	s,
		std::string *	existing = nullptr )

Attach a variable-length string annotation to a gate, once.

A gate with no annotation has an empty one, and an empty string is not an annotation, so "nothing recorded" and "recorded as nothing" coincide and the question of an unset marker does not arise. Offering the bytes the gate already holds is a no-op rather than a fresh append, which is what keeps the extra file from accumulating abandoned copies of the same string.

Parameters

token	UUID of the gate.
s	String to store.
existing	On `AlreadySet`, the string the gate holds.

Definition at line 285 of file MMappedCircuit.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ setInfos()

MMappedCircuit::SetAnnotationResult MMappedCircuit::setInfos	(	pg_uuid_t	token,
		unsigned	info1,
		unsigned	info2,
		std::pair< unsigned, unsigned > *	existing = nullptr )

Write the info1 / info2 annotations of a gate, once.

The two fields are written once each, with 0 meaning "nothing recorded": a field goes from 0 to a value once, accepts that value again, and refuses a different one; writing 0 over a value records nothing and leaves it alone. Per field rather than per pair because the two are written by different parties – a certified plus gate is marked as certified when it is built and tagged with the route that made it a query's root afterwards.

Parameters

token	UUID of the gate to annotate.
info1	Value for `info1`, or `0` to leave it alone.
info2	Value for `info2`, or `0` to leave it alone.
existing	On `AlreadySet`, the pair the gate holds.

Definition at line 258 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ setProb()

MMappedCircuit::SetProbResult MMappedCircuit::setProb	(	pg_uuid_t	token,
		double	prob,
		double *	existing = nullptr )

Write a gate's probability, once.

A probability is a fact appended to the circuit, like the gate itself: it can be written when the gate has none, re-written with the identical value (so setup scripts and notebook cells stay re-runnable), and otherwise refused. Changing one means minting a fresh input gate and rewriting the rows that carry the old token – see provsql.replace_input.

Passing NaN clears the probability unconditionally; that is how a transaction that rolls back drops the probabilities it wrote, and it is not reachable from SQL.

If the token is not yet in the circuit, an input gate is created lazily.

Parameters

token	UUID of the gate.
prob	Probability value in [0, 1], or `NaN` to clear.
existing	On `AlreadySet`, the probability the gate holds.

Returns: Which of the four cases applied.

Definition at line 178 of file MMappedCircuit.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ storePath()

std::string MMappedCircuit::storePath	(	Oid	db_oid,
		Oid	db_tablespace,
		const char *	filename )

static

Build the full path of a file in a database's store directory, the one GetDatabasePath resolves for db_oid in db_tablespace.

Definition at line 58 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ sweepInto()

MMappedCircuit::Counts MMappedCircuit::sweepInto	(	const std::vector< bool > &	live,
		const std::string &	mp,
		const std::string &	gp,
		const std::string &	wp,
		const std::string &	ep ) const

Copy the gates flagged in live into a fresh set of files beside the current ones (suffix ".new").

Child indices, extra offsets and the token table are all rebuilt, so the result is a compact, freshly hashed store. A version-1 gates file also gets its probabilities normalised on the way through: the 1.0 that used to mean both "written as certain" and "never written" becomes NaN on gates the file cannot prove were written, and the new file is stamped version 2, after which the ambiguity is gone.

Returns: The gate, wire and extra-byte counts of the new files.

Definition at line 995 of file MMappedCircuit.cpp.

Here is the call graph for this function:

Here is the caller graph for this function:

◆ sync()

void MMappedCircuit::sync ( )

Flush all backing files to disk with msync().

Definition at line 902 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ uncleanShutdown()

bool MMappedCircuit::uncleanShutdown ( ) const

Whether any backing file was found still marked open-for-writing when it was opened – the previous writer died mid-write.

Definition at line 922 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

Member Data Documentation

◆ extra

MMappedVector<char> MMappedCircuit::extra

private

Variable-length string data.

Definition at line 96 of file MMappedCircuit.h.

◆ EXTRA_FILENAME

const char* MMappedCircuit::EXTRA_FILENAME ="provsql_extra.mmap"

staticconstexprprivate

Backing file for extra.

Definition at line 101 of file MMappedCircuit.h.

◆ gates

MMappedVector<GateInformation> MMappedCircuit::gates

private

Gate metadata array.

Definition at line 94 of file MMappedCircuit.h.

◆ GATES_FILENAME

const char* MMappedCircuit::GATES_FILENAME ="provsql_gates.mmap"

staticconstexprprivate

Backing file for gates.

Definition at line 98 of file MMappedCircuit.h.

◆ GATES_VERSION

uint16_t MMappedCircuit::GATES_VERSION = 2

staticconstexpr

Format version of the gates file this build writes.

Version 1 stored 1.0 in GateInformation::prob both for a gate whose probability had been set to 1 and for one that had never been given a probability at all. Version 2 writes NaN for the latter, which is what lets a probability be written once and refused afterwards. A version-1 file is still read: 1.0 on a probability-bearing gate is then treated as unset, so a store carried across an upgrade keeps accepting the probabilities its owner has always been allowed to write. The ambiguity lasts until circuit_cleanup rewrites the file, which normalises those 1.0 values and stamps the file version 2.

Definition at line 137 of file MMappedCircuit.h.

◆ MAGIC_EXTRA

uint64_t MMappedCircuit::MAGIC_EXTRA

staticconstexpr

Initial value:

                                    =
uint64_t('P')       | uint64_t('v') <<  8 | uint64_t('S') << 16 | uint64_t('E') << 24 |
uint64_t('x') << 32 | uint64_t('t') << 40 | uint64_t('r') << 48 | uint64_t('a') << 56

Definition at line 181 of file MMappedCircuit.h.

◆ MAGIC_GATES

uint64_t MMappedCircuit::MAGIC_GATES

staticconstexpr

Initial value:

                                    =
uint64_t('P')       | uint64_t('v') <<  8 | uint64_t('S') << 16 | uint64_t('G') << 24 |
uint64_t('a') << 32 | uint64_t('t') << 40 | uint64_t('e') << 48 | uint64_t('s') << 56

8-byte magic constants identifying each mmap file type.

Definition at line 172 of file MMappedCircuit.h.

◆ MAGIC_MAPPING

uint64_t MMappedCircuit::MAGIC_MAPPING

staticconstexpr

Initial value:

                                      =
uint64_t('P')       | uint64_t('v') <<  8 | uint64_t('S') << 16 | uint64_t('M') << 24 |
uint64_t('a') << 32 | uint64_t('p') << 40 | uint64_t('n') << 48 | uint64_t('g') << 56

Definition at line 178 of file MMappedCircuit.h.

◆ MAGIC_WIRES

uint64_t MMappedCircuit::MAGIC_WIRES

staticconstexpr

Initial value:

                                    =
uint64_t('P')       | uint64_t('v') <<  8 | uint64_t('S') << 16 | uint64_t('W') << 24 |
uint64_t('i') << 32 | uint64_t('r') << 40 | uint64_t('e') << 48 | uint64_t('s') << 56

Definition at line 175 of file MMappedCircuit.h.

◆ mapping

MMappedUUIDHashTable MMappedCircuit::mapping

private

UUID → gate-index hash table.

Definition at line 93 of file MMappedCircuit.h.

◆ MAPPING_FILENAME

const char* MMappedCircuit::MAPPING_FILENAME ="provsql_mapping.mmap"

staticconstexprprivate

Backing file for mapping.

Definition at line 100 of file MMappedCircuit.h.

◆ wires

MMappedVector<pg_uuid_t> MMappedCircuit::wires

private

Flattened child UUID array.

Definition at line 95 of file MMappedCircuit.h.

◆ WIRES_FILENAME

const char* MMappedCircuit::WIRES_FILENAME ="provsql_wires.mmap"

staticconstexprprivate

Backing file for wires.

Definition at line 99 of file MMappedCircuit.h.

The documentation for this class was generated from the following files:

/home/senellar/git/software/provsql/src/MMappedCircuit.h
/home/senellar/git/software/provsql/src/MMappedCircuit.cpp

Classes

Public Types

Public Member Functions

Static Public Member Functions

Static Public Attributes

Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Member Enumeration Documentation

◆ SetAnnotationResult

◆ SetProbResult

Constructor & Destructor Documentation

◆ MMappedCircuit() [1/2]

◆ MMappedCircuit() [2/2]

◆ ~MMappedCircuit()

Member Function Documentation

◆ appendGate()

◆ check()

◆ counts()

◆ createGate()

◆ createGenericCircuit() [1/2]

◆ createGenericCircuit() [2/2]

◆ flush()

◆ getChildren()

◆ getExtra()

◆ getGateType()

◆ getInfos()

◆ getNbGates()

◆ getProb()

◆ hasProb()

◆ hasProbAt()

◆ legacyProbabilities()

◆ mark()

◆ setExtra()

◆ setInfos()

◆ setProb()

◆ storePath()

◆ sweepInto()

◆ sync()

◆ uncleanShutdown()

Member Data Documentation

◆ extra

◆ EXTRA_FILENAME

◆ gates

◆ GATES_FILENAME

◆ GATES_VERSION

◆ MAGIC_EXTRA

◆ MAGIC_GATES

◆ MAGIC_MAPPING

◆ MAGIC_WIRES

◆ mapping

◆ MAPPING_FILENAME

◆ wires

◆ WIRES_FILENAME