Documentation

Provenance.Query

Relational algebra #

This file defines the abstract syntax and semantics of relational algebra queries over plain (unannotated) databases. The language is the extended relational algebra described in Section III of Sen, Maniu & Senellart, ProvSQL: A General System for Keeping Track of the Provenance and Probability of Data, with multiset semantics, explicit duplicate elimination, multiset difference, and aggregation.

Main definitions #

References #

inductive Term (T : Sort u_1) (n : ) :
Sort (max 1 u_1)
Instances For
    Dependency graph
    def Term.repr {T : Type} {n : } [Repr T] :
    Term T nStd.Format
    Equations
    Instances For
      Dependency graph
      @[instance_reducible]
      instance instReprTerm {α : Type} {n : } [Repr α] :
      Repr (Term α n)
      Equations
      Dependency graph
      Dependency graph
      def Term.eval {T : Type} [ValueType T] {n : } (term : Term T n) (tuple : Tuple T n) :
      T
      Equations
      Instances For
        Dependency graph
        theorem Term.castToAnnotatedTuple_eval {T : Type} [ValueType T] {K : Type} {n : } [HasAltLinearOrder K] [SemiringWithMonus K] (t : Term T n) (tuple : Tuple T n) (α : K) :
        t.castToAnnotatedTuple.eval (Fin.append (fun (k : Fin n) => Sum.inl (tuple k)) ![Sum.inr α]) = Sum.inl (t.eval tuple)
        Dependency graph
        @[instance_reducible]
        instance instCoeTerm {T : Type} {n : } :
        Coe T (Term T n)
        Equations
        Dependency graph
        @[instance_reducible]
        instance instOfNatTermNat {n a : } :
        OfNat (Term n) a
        Equations
        Dependency graph
        Dependency graph
        inductive BoolTerm (T : Sort u_1) (n : ) :
        Sort (max 1 u_1)
        Instances For
          Dependency graph
          def BoolTerm.repr {T : Type} {n : } [Repr T] :
          BoolTerm T nStd.Format
          Equations
          Instances For
            Dependency graph
            @[instance_reducible]
            instance instReprBoolTerm {α : Type} {n : } [Repr α] :
            Repr (BoolTerm α n)
            Equations
            Dependency graph
            Dependency graph
            Dependency graph
            Dependency graph
            def BoolTerm.eval {T : Type} [ValueType T] {n : } (φ : BoolTerm T n) (tuple : Tuple T n) :
            Equations
            Instances For
              Dependency graph
              theorem BoolTerm.castToAnnotatedTuple_eval {T : Type} [ValueType T] {K : Type} {n : } [HasAltLinearOrder K] [SemiringWithMonus K] (t : BoolTerm T n) (tuple : Tuple T n) (α : K) :
              t.castToAnnotatedTuple.eval (Fin.append (fun (k : Fin n) => Sum.inl (tuple k)) ![Sum.inr α]) = t.eval tuple
              Dependency graph
              @[reducible]
              Equations
              • One or more equations did not get rendered due to their size.
              Instances For
                Dependency graph
                inductive Selection (T : Sort u_1) (n : ) :
                Sort (max 1 u_1)
                Instances For
                  Dependency graph
                  def Selection.repr {T : Type} {n : } [Repr T] :
                  Equations
                  Instances For
                    Dependency graph
                    @[instance_reducible]
                    instance instReprSelection {α : Type} {n : } [Repr α] :
                    Equations
                    Dependency graph
                    Dependency graph
                    def Selection.eval {T : Type} [ValueType T] {n : } (φ : Selection T n) (tuple : Tuple T n) :
                    Equations
                    Instances For
                      Dependency graph
                      theorem Selection.castToAnnotatedTuple_eval {T : Type} [ValueType T] {K : Type} {n : } [HasAltLinearOrder K] [SemiringWithMonus K] (φ : Selection T n) (tuple : Tuple T n) (α : K) :
                      φ.castToAnnotatedTuple.eval (Fin.append (fun (k : Fin n) => Sum.inl (tuple k)) ![Sum.inr α]) = φ.eval tuple
                      Dependency graph
                      @[reducible]
                      Equations
                      Instances For
                        Dependency graph
                        @[instance_reducible]
                        instance instCoeBoolTermSelection {T : Type} {n : } :
                        Coe (BoolTerm T n) (Selection T n)
                        Equations
                        Dependency graph
                        def addFn {T : Type} [ValueType T] (a b : T) :
                        T

                        Addition as a binary function, the fold of the -sum performed by the rewriting-target operator Query.ProvSum (and by its general-syntax counterpart AggQuery.ProvSum).

                        Equations
                        Instances For
                          Dependency graph
                          Dependency graph
                          Dependency graph
                          def SeqAggFunc (T : Type) :

                          An aggregate function on sequences of values: an arbitrary function from finite sequences over T to T. Beyond the monoid-shaped -sum of Query.ProvSum, this interface covers non-commutative aggregates – such as PICKFIRST, whose result depends on the order of its input – and non-associative ones. It is the aggregate interface of the fused Having operator, whose possible-world semantics does not need any algebraic structure on the aggregate.

                          Equations
                          Instances For
                            Dependency graph

                            SUM as a sequence aggregate.

                            Equations
                            Instances For
                              Dependency graph

                              COUNT(*) as a sequence aggregate (over an -valued domain).

                              Equations
                              Instances For
                                Dependency graph

                                MIN, with value 0 on the empty sequence (the possible-world semantics only ever applies it to non-empty sequences).

                                Equations
                                Instances For
                                  Dependency graph

                                  MAX, with value 0 on the empty sequence.

                                  Equations
                                  Instances For
                                    Dependency graph

                                    PICKFIRST: the first value of the sequence, 0 if empty.

                                    Equations
                                    Instances For
                                      Dependency graph
                                      inductive Query (T : Type) :
                                      Type
                                      • Rel {T : Type} (n : ) : StringQuery T n
                                      • Proj {T : Type} {n m : } : Tuple (Term T n) mQuery T nQuery T m
                                      • Sel {T : Type} {n : } : Selection T nQuery T nQuery T n
                                      • Prod {T : Type} {n₁ n₂ n : } {hn : n₁ + n₂ = n} : Query T n₁Query T n₂Query T n
                                      • Sum {T : Type} {n : } : Query T nQuery T nQuery T n
                                      • Dedup {T : Type} {n : } : Query T nQuery T n
                                      • Diff {T : Type} {n : } : Query T nQuery T nQuery T n
                                      • ProvSum {T : Type} {m n₁ : } : Tuple (Fin m) n₁Term T mQuery T mQuery T (n₁ + 1)

                                        Provenance aggregation: group by the key columns of the first argument and -sum the term of the second over each group into a single trailing output column. It is not a source operator – aggregation as such lives on the general syntax (AggQuery.Gamma) – but the target of the (R1)–(R4) rewriting: the ⊕-gate creation of the ε and rules of Query.rewriting. Its general-syntax counterpart is AggQuery.ProvSum.

                                      • Having {T : Type} {m n₁ n₂ : } : Tuple (Fin m) n₁Tuple (Term T m) n₂Tuple (SeqAggFunc T) n₂CompOpFin n₂Term T n₁Query T mQuery T (n₁ + n₂)

                                        The fused HAVING operator: grouping by the indices of the first argument, computing the sequence aggregates of the third argument applied to the terms of the second (each group read in the canonical tuple order, which plays the role of the ordering of non-commutative aggregates), and keeping only the groups whose aggregate value in column l (the Fin n₂ argument) compares, via the comparison operator, with the value of the regular term (the Term T n₁ argument, evaluated on the group key – this covers both query constants and group-key attributes). The output has the group key followed by the aggregate values.

                                      Instances For
                                        Dependency graph
                                        def Query.repr {T : Type} {n : } [Repr T] :
                                        Query T nStd.Format
                                        Equations
                                        Instances For
                                          Dependency graph
                                          @[instance_reducible]
                                          instance instReprQuery {α : Type} {n : } [Repr α] :
                                          Repr (Query α n)
                                          Equations
                                          Dependency graph
                                          def Query.source {T : Type} {n : } (q : Query T n) :

                                          The source fragment of the classical syntax: the operators a query is written with, RA⁺(∖). It excludes the two operators that are not source operators – ProvSum, which the (R1)–(R4) rewriting emits, and the fused Having, whose semantics lives on the general syntax – and is exactly the fragment carrying an annotated semantics (Query.evaluateAnnotated) and accepted by Query.rewriting. Its general-syntax counterpart is AggQuery.classical.

                                          Equations
                                          Instances For
                                            Dependency graph
                                            @[reducible]
                                            Equations
                                            Instances For
                                              Dependency graph
                                              Dependency graph
                                              @[simp]
                                              theorem Query.sourceProd {T : Type} [ValueType T] {n n₂ : } {q : Query T n} :
                                              q.source∀ {n₁ : } {q₁ : Query T n₁} {q₂ : Query T n₂} {hn : n₁ + n₂ = n}, q = q₁ × q₂q₁.source q₂.source
                                              Dependency graph
                                              @[simp]
                                              theorem Query.sourceSum {T : Type} [ValueType T] {n : } {q : Query T n} :
                                              q.source∀ {q₁ q₂ : Query T n}, q = (q₁ q₂) → q₁.source q₂.source
                                              Dependency graph
                                              @[simp]
                                              theorem Query.sourceDiff {T : Type} [ValueType T] {n : } {q : Query T n} :
                                              q.source∀ {q₁ q₂ : Query T n}, q = (q₁ - q₂) → q₁.source q₂.source
                                              Dependency graph
                                              @[simp]
                                              theorem Query.sourceProj {T : Type} [ValueType T] {n : } {q : Query T n} :
                                              q.source∀ {m : } {t : Tuple (Term T m) n} {q' : Query T m}, q = Π t q'q'.source
                                              Dependency graph
                                              @[simp]
                                              theorem Query.sourceSel {T : Type} [ValueType T] {n : } {q : Query T n} :
                                              q.source∀ {φ : Selection T n} {q' : Query T n}, q = σ φ q'q'.source
                                              Dependency graph
                                              @[simp]
                                              theorem Query.sourceDedup {T : Type} [ValueType T] {n : } {q : Query T n} :
                                              q.source∀ {q' : Query T n}, q = ε q'q'.source
                                              Dependency graph
                                              Dependency graph
                                              Dependency graph
                                              Dependency graph
                                              Dependency graph
                                              Dependency graph
                                              Dependency graph
                                              Dependency graph
                                              Dependency graph
                                              def Query.arity {T : Type} {n : } :
                                              Query T n
                                              Equations
                                              Instances For
                                                Dependency graph
                                                Dependency graph
                                                def Relation.groupSeq {T : Type} [ValueType T] {m n₁ : } (is : Tuple (Fin m) n₁) (r : Relation T m) (g : Tuple T n₁) :
                                                List (Tuple T m)

                                                The occurrences of the group of key g in relation r: the multiset of matching tuples, as a list sorted by the canonical linear order on tuples. The sort order plays the role of the ordering along which non-commutative sequence aggregates read the occurrences of a group; for commutative aggregates it is irrelevant.

                                                Equations
                                                Instances For
                                                  Dependency graph
                                                  @[irreducible]
                                                  def Query.evaluate {T : Type} [ValueType T] {n : } (q : Query T n) (d : Database T) :

                                                  Standard multiset semantics of a query over a plain database.

                                                  The Diff case is all-or-nothing difference: every copy of a tuple that occurs at all in r₂ is removed from r₁ (deliberately not Multiset.sub, which would subtract multiplicities as in SQL's EXCEPT ALL). This matches the monus-based annotated semantics of difference (Query.evaluateAnnotated) exactly on 0/1-annotated inputs; on general annotations the two disagree over (see Nat.counterexample_diff_adequacy and Provenance.QueryAdequacy).

                                                  Equations
                                                  Instances For
                                                    Dependency graph