ProvSQL C/C++ API
Adding support for provenance and uncertainty management to PostgreSQL databases
Loading...
Searching...
No Matches
MMappedCircuit.cpp File Reference

Persistent mmap-backed circuit: implementation and background-worker entry points. More...

#include <cerrno>
#include <cmath>
#include <map>
#include <queue>
#include <unordered_map>
#include <unordered_set>
#include <sstream>
#include <string>
#include "MMappedCircuit.h"
#include "process_memory.h"
#include "GenericCircuit.h"
#include "Circuit.hpp"
#include "provsql_utils_cpp.h"
#include <poll.h>
#include <boost/iostreams/device/back_inserter.hpp>
#include <boost/iostreams/stream.hpp>
#include <fcntl.h>
#include <unistd.h>
#include <cstring>
#include "miscadmin.h"
#include "common/relpath.h"
#include "utils/palloc.h"
#include "provsql_mmap.h"
#include "provsql_shmem.h"
Include dependency graph for MMappedCircuit.cpp:

Go to the source code of this file.

Macros

#define PROVSQL_FROM_SPOOL   provsql_worker_from_spool

Functions

void initialize_provsql_mmap ()
 Initialise the circuit store.
void destroy_provsql_mmap ()
 Unmap and close the mmap files.
static bool carriesProb (gate_type type)
 Whether type is one of the gate kinds that carry a probability.
static void finishInterruptedCleanup (Oid db_oid, Oid db_tablespace)
 Bring a database's store to a definite state before opening it.
static void closeDroppedCircuits ()
 Close the circuits of the databases dropped since they were opened.
static MMappedCircuit * getCircuit (Oid db_oid, Oid db_tablespace)
 Return (creating lazily if needed) the circuit for db_oid.
static void cleanupStore (Oid db_oid, Oid db_tablespace, const std::vector< pg_uuid_t > &roots, bool dry_run, provsql_cleanup_result *out)
 Rebuild a database's store, keeping only what roots reach.
void provsql_store_flush (void)
 Force every open circuit to stable storage.
static void send_circuit (const MMappedCircuit *circuit, const std::vector< pg_uuid_t > &roots, unsigned long budget, char message_char)
 Answer a 'g' / 'j' request: the circuit reachable from roots, serialised, preceded by its size.
static void dispatch (char c, Oid db_oid, Oid db_tablespace)
 The body of provsql_mmap_dispatch, which may throw.
void provsql_mmap_dispatch (char c, Oid db_oid, Oid db_tablespace)
 Serve one message of the store's protocol.
void provsql_mmap_main_loop ()
 Main processing loop of the mmap background worker.
bool operator< (const pg_uuid_t a, const pg_uuid_t b)
 Lexicographic less-than comparison for pg_uuid_t.

Variables

static std::map< Oid, MMappedCircuit * > circuits
 Per-database mmap-backed provenance circuits, keyed by database OID.
static bool store_dirty = false
 Whether any circuit has been written to since the last flush.
static constexpr const char * CLEANUP_SUFFIX = ".new"
 Suffix of the files a clean-up builds beside the live ones.
static constexpr const char * CLEANUP_MARKER = "provsql_cleanup.commit"
 Marker file present only while a clean-up's rename sequence is in flight; see finishInterruptedCleanup.
static std::map< Oid, Oid > circuit_tablespaces
 The tablespace of each open circuit, to find its files again.

Detailed Description

Persistent mmap-backed circuit: implementation and background-worker entry points.

Implements the MMappedCircuit methods declared in MMappedCircuit.h, the createGenericCircuit() free function, and the background-worker entry points declared in provsql_mmap.h:

The createGenericCircuit() function performs a BFS from a root UUID, reading gates from the mmap store and building an in-memory GenericCircuit.

Definition in file MMappedCircuit.cpp.

Macro Definition Documentation

◆ PROVSQL_FROM_SPOOL

#define PROVSQL_FROM_SPOOL   provsql_worker_from_spool

Definition at line 59 of file MMappedCircuit.cpp.

Function Documentation

◆ carriesProb()

bool carriesProb ( gate_type type)
static

Whether type is one of the gate kinds that carry a probability.

Definition at line 198 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ cleanupStore()

void cleanupStore ( Oid db_oid,
Oid db_tablespace,
const std::vector< pg_uuid_t > & roots,
bool dry_run,
provsql_cleanup_result * out )
static

Rebuild a database's store, keeping only what roots reach.

The store only ever grows: a gate is never removed, because removing one is unsafe while anything might still reference it, and because a content-addressed gate can be re-adopted by a query at any moment. This is the one operation allowed to remove gates, and it is safe only because the caller holds the database exclusively – no other session is connected, so nothing can adopt an orphan while the sweep runs.

On dry_run the mark phase runs and the counts are reported, but nothing is written.

Definition at line 476 of file MMappedCircuit.cpp.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ closeDroppedCircuits()

void closeDroppedCircuits ( )
static

Close the circuits of the databases dropped since they were opened.

Dropping a database removes its directory, the circuit files with it, but not the worker's descriptors and mappings of them: kept, they would hold the disk space of every dropped database, and four descriptors each, until opening one more failed for lack of descriptors – which a test run that creates and drops a database per test reached after a few hundred. A circuit whose gates file is gone belongs to a dropped database (one being cleaned up keeps it until the swap, and is not open meanwhile).

Definition at line 432 of file MMappedCircuit.cpp.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ destroy_provsql_mmap()

void destroy_provsql_mmap ( void )

Unmap and close the mmap files.

Called by the background worker on shutdown to release resources and ensure all dirty pages are synced to disk via msync().

Definition at line 111 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ dispatch()

void dispatch ( char c,
Oid db_oid,
Oid db_tablespace )
static

The body of provsql_mmap_dispatch, which may throw.

Definition at line 638 of file MMappedCircuit.cpp.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ finishInterruptedCleanup()

void finishInterruptedCleanup ( Oid db_oid,
Oid db_tablespace )
static

Bring a database's store to a definite state before opening it.

A clean-up writes four new files, then renames them over the live ones one at a time. A crash inside that sequence would leave a mixture, so the sequence is bracketed by a marker file: present, it says the new files are complete and the renames must be finished; absent, it says any ".new" files are the debris of a clean-up that never got that far and are to be discarded.

Definition at line 395 of file MMappedCircuit.cpp.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ getCircuit()

MMappedCircuit * getCircuit ( Oid db_oid,
Oid db_tablespace )
static

Return (creating lazily if needed) the circuit for db_oid.

Definition at line 448 of file MMappedCircuit.cpp.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ initialize_provsql_mmap()

void initialize_provsql_mmap ( void )

Initialise the circuit store.

Called once by the background worker at startup. The per-database files themselves are opened lazily, on the first message for their database.

Definition at line 106 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ operator<()

bool operator< ( const pg_uuid_t a,
const pg_uuid_t b )

Lexicographic less-than comparison for pg_uuid_t.

Parameters
aLeft UUID.
bRight UUID.
Returns
true if a is lexicographically less than b.

Definition at line 1276 of file MMappedCircuit.cpp.

◆ provsql_mmap_dispatch()

void provsql_mmap_dispatch ( char c,
Oid db_oid,
Oid db_tablespace )

Serve one message of the store's protocol.

Handle a single IPC message: read its payload and write its reply.

A C++ exception escaping here – a file that cannot be opened or mapped, an allocation that fails – would reach the C frames of the worker and terminate it with an abort, which the postmaster answers by resetting every backend of the cluster. It is reported as the error it is instead, as the worker's other failures are: the worker exits and is restarted, and the other databases keep their sessions.

Definition at line 1031 of file MMappedCircuit.cpp.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ provsql_mmap_main_loop()

void provsql_mmap_main_loop ( void )

Main processing loop of the mmap background worker.

Waits for gate-creation requests from backend processes, processes them by writing to the mmap files, and handles SIGTERM for graceful shutdown.

Definition at line 1047 of file MMappedCircuit.cpp.

Here is the call graph for this function:
Here is the caller graph for this function:

◆ provsql_store_flush()

void provsql_store_flush ( void )

Force every open circuit to stable storage.

Definition at line 568 of file MMappedCircuit.cpp.

Here is the caller graph for this function:

◆ send_circuit()

void send_circuit ( const MMappedCircuit * circuit,
const std::vector< pg_uuid_t > & roots,
unsigned long budget,
char message_char )
static

Answer a 'g' / 'j' request: the circuit reachable from roots, serialised, preceded by its size.

The circuit is serialised straight into the buffer that is sent, without the copy a std::stringstream makes of its content, and is freed before it is sent. With a budget (bytes, 0 for none: what the requesting statement has left of provsql.max_memory), building and serialising it may not make this process grow by more; past it, the answer is PROVSQL_CIRCUIT_OVER_BUDGET and no circuit; if the requesting statement is cancelled meanwhile, it is PROVSQL_CIRCUIT_CANCELLED. The memory is given back afterwards, so that one large request does not stay resident in the worker.

Definition at line 590 of file MMappedCircuit.cpp.

Here is the call graph for this function:
Here is the caller graph for this function:

Variable Documentation

◆ circuit_tablespaces

std::map<Oid, Oid> circuit_tablespaces
static

The tablespace of each open circuit, to find its files again.

Definition at line 419 of file MMappedCircuit.cpp.

◆ circuits

std::map<Oid, MMappedCircuit*> circuits
static

Per-database mmap-backed provenance circuits, keyed by database OID.

Definition at line 65 of file MMappedCircuit.cpp.

◆ CLEANUP_MARKER

const char* CLEANUP_MARKER = "provsql_cleanup.commit"
staticconstexpr

Marker file present only while a clean-up's rename sequence is in flight; see finishInterruptedCleanup.

Definition at line 383 of file MMappedCircuit.cpp.

◆ CLEANUP_SUFFIX

const char* CLEANUP_SUFFIX = ".new"
staticconstexpr

Suffix of the files a clean-up builds beside the live ones.

Definition at line 380 of file MMappedCircuit.cpp.

◆ store_dirty

bool store_dirty = false
static

Whether any circuit has been written to since the last flush.

The store's bytes reach the kernel as soon as a message is applied, but they reach the disk only when someone forces them; a machine that loses power in between loses whatever the kernel had not written back. The worker forces them a short while after the last write (see the main loop below), which bounds that loss the way synchronous_commit = off bounds the heap's.

Definition at line 75 of file MMappedCircuit.cpp.