![]() |
ProvSQL C/C++ API
Adding support for provenance and uncertainty management to PostgreSQL databases
|
Persistent mmap-backed circuit: implementation and background-worker entry points. More...
#include <cerrno>#include <cmath>#include <map>#include <queue>#include <unordered_map>#include <unordered_set>#include <sstream>#include <string>#include "MMappedCircuit.h"#include "process_memory.h"#include "GenericCircuit.h"#include "Circuit.hpp"#include "provsql_utils_cpp.h"#include <poll.h>#include <boost/iostreams/device/back_inserter.hpp>#include <boost/iostreams/stream.hpp>#include <fcntl.h>#include <unistd.h>#include <cstring>#include "miscadmin.h"#include "common/relpath.h"#include "utils/palloc.h"#include "provsql_mmap.h"#include "provsql_shmem.h"
Go to the source code of this file.
Macros | |
| #define | PROVSQL_FROM_SPOOL provsql_worker_from_spool |
Functions | |
| void | initialize_provsql_mmap () |
| Initialise the circuit store. | |
| void | destroy_provsql_mmap () |
| Unmap and close the mmap files. | |
| static bool | carriesProb (gate_type type) |
Whether type is one of the gate kinds that carry a probability. | |
| static void | finishInterruptedCleanup (Oid db_oid, Oid db_tablespace) |
| Bring a database's store to a definite state before opening it. | |
| static void | closeDroppedCircuits () |
| Close the circuits of the databases dropped since they were opened. | |
| static MMappedCircuit * | getCircuit (Oid db_oid, Oid db_tablespace) |
Return (creating lazily if needed) the circuit for db_oid. | |
| static void | cleanupStore (Oid db_oid, Oid db_tablespace, const std::vector< pg_uuid_t > &roots, bool dry_run, provsql_cleanup_result *out) |
Rebuild a database's store, keeping only what roots reach. | |
| void | provsql_store_flush (void) |
| Force every open circuit to stable storage. | |
| static void | send_circuit (const MMappedCircuit *circuit, const std::vector< pg_uuid_t > &roots, unsigned long budget, char message_char) |
Answer a 'g' / 'j' request: the circuit reachable from roots, serialised, preceded by its size. | |
| static void | dispatch (char c, Oid db_oid, Oid db_tablespace) |
The body of provsql_mmap_dispatch, which may throw. | |
| void | provsql_mmap_dispatch (char c, Oid db_oid, Oid db_tablespace) |
| Serve one message of the store's protocol. | |
| void | provsql_mmap_main_loop () |
| Main processing loop of the mmap background worker. | |
| bool | operator< (const pg_uuid_t a, const pg_uuid_t b) |
Lexicographic less-than comparison for pg_uuid_t. | |
Variables | |
| static std::map< Oid, MMappedCircuit * > | circuits |
| Per-database mmap-backed provenance circuits, keyed by database OID. | |
| static bool | store_dirty = false |
| Whether any circuit has been written to since the last flush. | |
| static constexpr const char * | CLEANUP_SUFFIX = ".new" |
| Suffix of the files a clean-up builds beside the live ones. | |
| static constexpr const char * | CLEANUP_MARKER = "provsql_cleanup.commit" |
Marker file present only while a clean-up's rename sequence is in flight; see finishInterruptedCleanup. | |
| static std::map< Oid, Oid > | circuit_tablespaces |
| The tablespace of each open circuit, to find its files again. | |
Persistent mmap-backed circuit: implementation and background-worker entry points.
Implements the MMappedCircuit methods declared in MMappedCircuit.h, the createGenericCircuit() free function, and the background-worker entry points declared in provsql_mmap.h:
initialize_provsql_mmap(): called by the background worker at startup; the per-database MMappedCircuit instances themselves are opened lazily, on the first message for their database.destroy_provsql_mmap(): called on shutdown; syncs and deletes the singleton.provsql_mmap_main_loop(): the worker's main loop; receives gate- creation messages from backends over the IPC pipe and writes them to the mmap store.The createGenericCircuit() function performs a BFS from a root UUID, reading gates from the mmap store and building an in-memory GenericCircuit.
Definition in file MMappedCircuit.cpp.
| #define PROVSQL_FROM_SPOOL provsql_worker_from_spool |
Definition at line 59 of file MMappedCircuit.cpp.
|
static |
Whether type is one of the gate kinds that carry a probability.
Definition at line 198 of file MMappedCircuit.cpp.

|
static |
Rebuild a database's store, keeping only what roots reach.
The store only ever grows: a gate is never removed, because removing one is unsafe while anything might still reference it, and because a content-addressed gate can be re-adopted by a query at any moment. This is the one operation allowed to remove gates, and it is safe only because the caller holds the database exclusively – no other session is connected, so nothing can adopt an orphan while the sweep runs.
On dry_run the mark phase runs and the counts are reported, but nothing is written.
Definition at line 476 of file MMappedCircuit.cpp.


|
static |
Close the circuits of the databases dropped since they were opened.
Dropping a database removes its directory, the circuit files with it, but not the worker's descriptors and mappings of them: kept, they would hold the disk space of every dropped database, and four descriptors each, until opening one more failed for lack of descriptors – which a test run that creates and drops a database per test reached after a few hundred. A circuit whose gates file is gone belongs to a dropped database (one being cleaned up keeps it until the swap, and is not open meanwhile).
Definition at line 432 of file MMappedCircuit.cpp.


| void destroy_provsql_mmap | ( | void | ) |
Unmap and close the mmap files.
Called by the background worker on shutdown to release resources and ensure all dirty pages are synced to disk via msync().
Definition at line 111 of file MMappedCircuit.cpp.

|
static |
The body of provsql_mmap_dispatch, which may throw.
Definition at line 638 of file MMappedCircuit.cpp.


|
static |
Bring a database's store to a definite state before opening it.
A clean-up writes four new files, then renames them over the live ones one at a time. A crash inside that sequence would leave a mixture, so the sequence is bracketed by a marker file: present, it says the new files are complete and the renames must be finished; absent, it says any ".new" files are the debris of a clean-up that never got that far and are to be discarded.
Definition at line 395 of file MMappedCircuit.cpp.


|
static |
Return (creating lazily if needed) the circuit for db_oid.
Definition at line 448 of file MMappedCircuit.cpp.


| void initialize_provsql_mmap | ( | void | ) |
Initialise the circuit store.
Called once by the background worker at startup. The per-database files themselves are opened lazily, on the first message for their database.
Definition at line 106 of file MMappedCircuit.cpp.

Lexicographic less-than comparison for pg_uuid_t.
| a | Left UUID. |
| b | Right UUID. |
true if a is lexicographically less than b. Definition at line 1276 of file MMappedCircuit.cpp.
| void provsql_mmap_dispatch | ( | char | c, |
| Oid | db_oid, | ||
| Oid | db_tablespace ) |
Serve one message of the store's protocol.
Handle a single IPC message: read its payload and write its reply.
A C++ exception escaping here – a file that cannot be opened or mapped, an allocation that fails – would reach the C frames of the worker and terminate it with an abort, which the postmaster answers by resetting every backend of the cluster. It is reported as the error it is instead, as the worker's other failures are: the worker exits and is restarted, and the other databases keep their sessions.
Definition at line 1031 of file MMappedCircuit.cpp.


| void provsql_mmap_main_loop | ( | void | ) |
Main processing loop of the mmap background worker.
Waits for gate-creation requests from backend processes, processes them by writing to the mmap files, and handles SIGTERM for graceful shutdown.
Definition at line 1047 of file MMappedCircuit.cpp.


| void provsql_store_flush | ( | void | ) |
Force every open circuit to stable storage.
Definition at line 568 of file MMappedCircuit.cpp.

|
static |
Answer a 'g' / 'j' request: the circuit reachable from roots, serialised, preceded by its size.
The circuit is serialised straight into the buffer that is sent, without the copy a std::stringstream makes of its content, and is freed before it is sent. With a budget (bytes, 0 for none: what the requesting statement has left of provsql.max_memory), building and serialising it may not make this process grow by more; past it, the answer is PROVSQL_CIRCUIT_OVER_BUDGET and no circuit; if the requesting statement is cancelled meanwhile, it is PROVSQL_CIRCUIT_CANCELLED. The memory is given back afterwards, so that one large request does not stay resident in the worker.
Definition at line 590 of file MMappedCircuit.cpp.


|
static |
The tablespace of each open circuit, to find its files again.
Definition at line 419 of file MMappedCircuit.cpp.
|
static |
Per-database mmap-backed provenance circuits, keyed by database OID.
Definition at line 65 of file MMappedCircuit.cpp.
|
staticconstexpr |
Marker file present only while a clean-up's rename sequence is in flight; see finishInterruptedCleanup.
Definition at line 383 of file MMappedCircuit.cpp.
|
staticconstexpr |
Suffix of the files a clean-up builds beside the live ones.
Definition at line 380 of file MMappedCircuit.cpp.
|
static |
Whether any circuit has been written to since the last flush.
The store's bytes reach the kernel as soon as a message is applied, but they reach the disk only when someone forces them; a machine that loses power in between loses whatever the kernel had not written back. The worker forces them a short while after the last write (see the main loop below), which bounds that loss the way synchronous_commit = off bounds the heap's.
Definition at line 75 of file MMappedCircuit.cpp.