Welcome to ProvSQL’s documentation!
ProvSQL is a PostgreSQL extension that adds semiring provenance and uncertainty management to SQL queries. It transparently rewrites queries to track which input tuples contribute to each result, then evaluates the provenance in any user-chosen semiring – Boolean reachability, counting, probability, Shapley values, and more.
A companion Python web UI, ProvSQL Studio, provides interactive provenance inspection on top of any ProvSQL-enabled database.
Both are free, open-source software distributed under the MIT License.
This documentation is organized into four parts:
The User Guide explains how to install, configure, and use ProvSQL from the SQL level and through ProvSQL Studio. Start here if you are new to ProvSQL.
The Case Studies present extended worked examples of ProvSQL applied to realistic scenarios, covering a broader range of features than the introductory tutorial.
The Developer Guide describes ProvSQL’s internal architecture and is aimed at contributors. It covers the PostgreSQL extension concepts ProvSQL relies on, the architecture and component map, the query rewriting pipeline, memory management, the where-provenance and data-modification subsystems, aggregation and semiring evaluation, probability computation, coding conventions, testing, debugging, the build system, and ProvSQL Studio’s architecture.
The API Reference provides auto-generated reference documentation for the SQL and C/C++ APIs (via Doxygen).
User Guide
- Introduction
- Getting ProvSQL
- Tutorial: Who Killed Daphine?
- The Scenario
- Setup
- Step 1: Explore the Database
- Step 2: Build a Sightings Table
- Step 3: Enable Provenance
- Step 4: Find Contradictions
- Step 5: Display Provenance Formulas
- Step 6: Build a Consistent Sightings Table
- Step 7: Identify Suspects
- Step 8: Count Confirming Sightings
- Step 9: Assign Reliability Probabilities
- Step 10: Find the Murderer
- Adding Provenance to a Table
- Querying with Provenance
- NULL Values
- Aggregation and Grouping
- Semiring Evaluation
- Boolean Semiring
- Boolean-Expression Semiring
- Symbolic Representation (as a Formula)
- Counting Semiring (m-semiring)
- Why-Provenance
- How-Provenance
- Which-Provenance (Lineage)
- Tropical Semiring (m-semiring)
- Viterbi Semiring (m-semiring)
- Łukasiewicz Fuzzy Semiring (m-semiring)
- Temporal (Interval-Union) Semiring (m-semiring)
- Numeric and Integer Interval-Union Semirings (m-semirings)
- Min-Max and Max-Min Semirings (m-semirings)
- Custom Semirings with
provenance_evaluate - Provenance Mappings
- Compatibility with Boolean-Provenance Rewriting
- Probabilities
- Setting input probabilities
- Correlated and block-independent inputs
- Computing query probabilities
- Aggregates: expected values and HAVING
- Continuous random variables
- When is exact evaluation tractable?
- Specialized routes for hard queries
- Forcing a specific method
- Performance optimizations under the hood
- Knowledge Compilation
- Continuous Distributions
- Introduction
- Distribution Constructors
- Arithmetic on Random Variables
- Order Statistics: greatest / least, min / max
- CASE Over Random Variables
- Probabilistic Queries
- Configuration of the Monte Carlo Sampler
- Exact vs. Sampled Answers
- Moments, Quantiles, and Support
- Conditional Inference
- Sampling and Histograms
- Mixtures and Categorical Random Variables
- Aggregation Over Random Variables
- Latent variables and posterior inference
- Studio Integration
- Out of Scope / Open Follow-ups
- Conditioning
- Shapley and Banzhaf Values
- Where-Provenance
- Data Modification Tracking
- Temporal Features
- Provenance Export
- ProvSQL Studio
- Configuration Reference
- External-Tool Registry
Case Studies
- Case Studies: Overview
- Case Study: Intelligence Agency
- The Scenario
- Setup
- Step 1: Explore the Database
- Step 2: Enable Provenance and Create a Name Mapping
- Step 3: Cities Shared by Multiple Agents
- Step 4: Minimum Security Clearance (sr_minmax)
- Step 5: Cities with Exactly One Agent (EXCEPT / Monus)
- Step 6: Where-Provenance
- Step 7: Assign Probabilities
- Step 8: Probability – Exact
- Step 9: Probability – Monte Carlo
- Step 10: Probability – Knowledge Compiler
- Step 11: Visualise a Provenance Circuit
- Step 12: Export to XML
- Step 13: Large Circuit Benchmark
- Step 14: The Boolean Expression Behind a Token
- Step 15: Programmatic Circuit Inspection
- Case Study: Open Science Database
- The Scenario
- Setup
- Step 1: Explore the Database
- Step 2: Enable Provenance and Join with Lookup Tables
- Step 3: Create a Provenance Mapping
- Step 4: Identify Single-Source Claims
- Step 5: Why-Provenance for Replicated Findings
- Step 6: Evidence Grade Semiring
- Step 7: Where-Provenance
- Step 8: Where-Provenance on the Base Table
- Step 9: Assign Probabilities
- Step 10: Build the Replication View
- Step 11: Inspect Replication with
sr_counting - Step 12: Probability of Replication
- Step 13: Shapley Values
- Step 14: Banzhaf Values
- Step 15: Bulk Shapley/Banzhaf with
shapley_all_varsandbanzhaf_all_vars - Step 16: Arithmetic on Aggregate Results
- Step 17: Richer Aggregates –
DISTINCT,string_agg,FILTER - Step 18: A Signed-Effect View with
UNION ALL
- Case Study: Île-de-France Public Transit
- Case Study: Government Ministers Over Time
- Case Study: Wildlife Photo Archive
- The Scenario
- Setup
- Step 1: Explore the Database
- Step 2: Enable Provenance and Create a Name Mapping
- Step 3: Inline Lookup with
VALUES - Step 4: Conjunctive Query (Naive)
- Step 5: Mutually Exclusive Candidates with
repair_key - Step 6: Assign Probabilities and Verify Mutual Exclusion
- Step 7: Probabilistic Ranking vs. Threshold Filtering
- Step 8: Absence Constraint with
EXCEPT - Step 9: Multi-Condition Query via a CTE
- Step 10: Expected Species Counts with
expected - Step 11: Materialising a Subset with
INSERT … SELECT - Step 12: Unidentified Detections – Three Kinds of “Not”
- Case Study: City Air-Quality Sensor Network
- The Scenario
- Setup
- Step 1: Inspect a Noisy Reading
- Step 2: A First Probabilistic Threshold
- Step 3: The Simplifier in Action
- Step 4: Calibration via Mixtures
- Step 5: Aggregation Over Random Variables
- Step 6: Conditional Inference
- Step 7: Diagnostic Sampling
- Step 8: Combining Batches via UNION
- Step 9: Filtering Grouped Random Variables by Expected Value
- Step 10: Independent vs Monte Carlo
- Step 11: The Worst Reading in a District
- Step 12: Percentiles and Exceedance Margins
- Step 13: Expected Excess via CASE
- Step 14: Correlated Sensors
- Step 15: Drift, Lifetime, and Spikes: More Families
- Step 16: Fleet Statistics Under Sensor Dropout
- Step 17: A Maintenance-Triage Headline via CASE over Aggregates
- Step 18: Shared Information in Nats
- Step 19: An Offline Sensor Reports NULL
- Case Study: Peer-Review Assignment and Knowledge Compilation
- Case Study: ProvSQL as a Probability Calculator
- The Scenario
- Problem 1: The Base-Rate Fallacy
- Problem 2: Correlation That Matters
- Problem 3: The Right Method, Chosen for You
- Problem 4: A Continuous Quantity
- Problem 5: A Probabilistic Total
- Problem 6: Ruling Worlds Out
- Problem 7: A Skewed Waiting Time
- Problem 8: How Many, and How Often
- Problem 9: How Much Did the Data Teach Us?
- Problem 10: Two Subpopulations, One Measurement
- Problem 11: Borrowed Posteriors and Forecast Tables
- Problem 12: A Parameter That Is Itself Uncertain
- Problem 13: Learning That Parameter From Data
- Recap
Developer Guide
- Developer Guide
- PostgreSQL Extension Primer
- Architecture Overview
- Query Rewriting Pipeline
- Memory Management
- Where-Provenance
- Data-Modification Tracking
- Aggregation Provenance
- Semiring Evaluation
- Probability Evaluation
- Architecture
- Background: d-DNNF, Tseytin, Knowledge Compilation
- HAVING Query Complexity: the Ré–Suciu Trichotomy
- Block-Independent Databases and Multivalued Inputs
- Shapley and Banzhaf Values
- Hybrid Evaluation for Continuous Distributions
- Conditional Evaluation
- The Inversion-Free
UCQ(OBDD)Path - Step-by-Step: Adding a New Probability Evaluation Method
- Knowledge-Compiler / Model-Counter Server Protocol
- Continuous Distributions
- Coding Conventions
- Testing
- Debugging
- Build System
- ProvSQL Studio
- ProvSQL Playground (browser build)
API Reference
References