Documentation

Provenance.HavingMinMax

Scan-computable HAVING provenance for MIN, MAX and PICKFIRST #

For a HAVING predicate comparing an aggregate to a constant, the possible-world semantics sums the world annotations T_U(W) over the non-empty valid worlds W ⊆ U of a group, of which there are in general exponentially many. This file shows that for the aggregates MIN, MAX and PICKFIRST – those whose validity is decided occurrence by occurrence – that sum collapses, in an absorptive commutative m-semiring, to a closed form using O(|U|) semiring operations: the provenance is obtained by a single scan over the occurrences of the group, hence in polynomial time in data complexity.

The collapse #

Everything rests on one identity, meet_family_eq: for H ⊆ G ⊆ U,

⊕_{W ⊆ G, W ∩ H ≠ ∅} T_U(W) = (𝟙 ⊖ ⊕_{x ∈ U \ G} α x) ⊗ (⊕_{i ∈ H} α i),

i.e., the provenance of “the world stays inside G and meets H” is the product of two running sums. Both hypotheses of the absorptive setting are used: follows from A_W ≤ α i ≤ ⊕_H α (absorptivity makes A decreasing under inclusion), and from upward_expansion in the universe G, transported to U by T_eq_T_of_subset and sum_monus.

Where the hypotheses are used #

Absorptivity makes A decreasing under inclusion (A_le_of_subset_absorptive) and is used in both halves. mul_sub_left_distributive is used exactly twice: in the half, to pass from T_U(W) – defined in Provenance.Having as A_W ⊖ ⊕_x A_{W ∪ {x}} – to the factored form A_W ⊗ (𝟙 ⊖ ⊕_{U∖W} α) in which the possible-world semantics is stated; and in the half, to rewrite the right-hand side as ⊕_H α ⊖ (⊕_H α ⊗ ⊕_{U∖G} α), the shape sum_monus needs. The first use is not a proof convenience: in MaxMin TVL (absorptive but not mul_sub_left_distributive) the identity is false for T as defined here – U = {1,2}, G = H = {1} and α₁ = α₂ = unknown give 𝟘 on the left and unknown on the right – because that is precisely where the two forms of the world annotation part company.

The six comparison operators then instantiate this with the right pair (G, H); e.g., MIN(t) ≥ c keeps the worlds inside G = {i | t i ≥ c}, while MIN(t) ≤ c keeps the worlds meeting H = {i | t i ≤ c}, and MIN(t) = c uses both. PICKFIRST splits the worlds according to their first occurrence and applies the identity to each fiber.

Main results #

Unlike the COUNT/SUM cases, no world enumeration is involved: the scans are closed forms, and the absorptivity hypothesis is what makes them exist (in ℕ[X], for instance, the same provenance is a product rather than a sum of annotations).

@[simp]
theorem CompOp.eval_coe_withTop {V : Type} [LinearOrder V] (op : CompOp) (a b : V) :
op.eval a b op.eval a b

Comparisons are unchanged by the embedding of the value domain into WithTop (used to give the empty world an aggregate value).

Dependency graph
@[simp]
theorem CompOp.eval_coe_withBot {V : Type} [LinearOrder V] (op : CompOp) (a b : V) :
op.eval a b op.eval a b

Comparisons are unchanged by the embedding of the value domain into WithBot.

Dependency graph
theorem Having.monus_mono_left {K : Type} [CommSemiringWithMonus K] {a a' c : K} (h : a a') :
a - c a' - c

Monus is monotone in its first argument.

Dependency graph
theorem Having.A_le_sum_of_meet {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (h_abs : absorptive K) (α : ιK) {W H : Finset ι} {i : ι} (hiW : i W) (hiH : i H) :
A α W jH, α j

The annotation of a world meeting H is bounded by ⊕_{i ∈ H} α i.

Dependency graph
theorem Having.meet_family_le {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) {U G H : Finset ι} :
WG.powerset with (W H).Nonempty, T α U W (1 - xU \ G, α x) * iH, α i

Upper bound half of meet_family_eq.

Dependency graph
theorem Having.meet_family_self_ge {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (h_abs : absorptive K) (α : ιK) {G H : Finset ι} (hHG : HG) :
iH, α i WG.powerset with (W H).Nonempty, T α G W

Lower bound half in the special case G = U: every α i, i ∈ H, is already reached by the worlds containing i.

Dependency graph
theorem Having.T_eq_T_of_subset {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (α : ιK) {U G W : Finset ι} (hGU : GU) (hWG : WG) :
T α U W = T α G W - A α W * xU \ G, α x

For a world W ⊆ G ⊆ U, passing from the universe G to the larger universe U subtracts the annotations of the occurrences of U \ G. Only monus_add and ordinary distributivity are used, not mul_sub_left_distributive.

Dependency graph
theorem Having.meet_family_eq {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) {U G H : Finset ι} (hGU : GU) (hHG : HG) :
WG.powerset with (W H).Nonempty, T α U W = (1 - xU \ G, α x) * iH, α i

Core collapse. In an absorptive m-semiring, the possible-world provenance of the family of worlds that stay inside G and meet H is (𝟙 ⊖ ⊕_{x ∈ U \ G} α x) ⊗ (⊕_{i ∈ H} α i): two running sums, hence a single scan over the occurrences of the group.

Dependency graph

Possible-world provenance of a selection predicate #

def Having.prov {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (α : ιK) (U : Finset ι) (P : Finset ιProp) [DecidablePred P] :
K

The possible-world provenance, at one group, of a selection predicate P on worlds: the -sum of the world annotations T_U(W) over the non-empty worlds satisfying P, in the possible-world semantics of HAVING predicates.

Equations
Instances For
    Dependency graph
    theorem Having.prov_congr {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (α : ιK) (U : Finset ι) {P Q : Finset ιProp} [DecidablePred P] [DecidablePred Q] (h : WU, P W Q W) :
    prov α U P = prov α U Q

    Two predicates that agree on the worlds of U have the same provenance.

    Dependency graph
    theorem Having.prov_or {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (α : ιK) (U : Finset ι) {P Q : Finset ιProp} [DecidablePred P] [DecidablePred Q] (hdisj : ∀ (W : Finset ι), P WQ WFalse) :
    (prov α U fun (W : Finset ι) => P W Q W) = prov α U P + prov α U Q

    The provenance of a disjunction of two mutually exclusive predicates is the -sum of the two provenances.

    Dependency graph
    theorem Having.prov_fiber {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (α : ιK) (U P : Finset ι) (Q : ιFinset ιProp) [(i : ι) → DecidablePred (Q i)] {R : Finset ιProp} [DecidablePred R] (hR : WU, W.Nonempty R W iP, W.Nonempty Q i W) (hdisj : ∀ (W : Finset ι), iP, jP, Q i WQ j Wi = j) :
    prov α U R = iP, prov α U (Q i)

    Provenance of a predicate that splits into fibers indexed by a finite set: if every non-empty world satisfying R satisfies Q i for exactly one i ∈ P, the provenance of R is the -sum of the provenances of the Q i.

    Dependency graph
    theorem Having.prov_of_forall_not {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (α : ιK) (U : Finset ι) {P : Finset ιProp} [DecidablePred P] (hP : WU, W.Nonempty¬P W) :
    prov α U P = 0

    Unsatisfiable predicates have provenance 𝟘. This is the algebraic content of the range-check short-circuit of the enumeration algorithms: when no world can satisfy the predicate, the possible-world -sum is empty. No hypothesis on the m-semiring is needed.

    Dependency graph
    theorem Having.prov_of_forall {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (h_abs : absorptive K) (α : ιK) (U : Finset ι) {P : Finset ιProp} [DecidablePred P] (hP : WU, W.NonemptyP W) :
    prov α U P = iU, α i

    Necessarily-true predicates have provenance ⊕_{i ∈ U} α i. This is the algebraic content of the complementary range-check short-circuit: when every non-empty world satisfies the predicate, the possible-world provenance is F_1(U), which in an absorptive m-semiring collapses (by F_eq_S at C = 1) to the -sum of the annotations of the group.

    Dependency graph
    theorem Having.prov_eq_of_pointwise {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (p q r : ιProp) [DecidablePred p] [DecidablePred q] [DecidablePred r] (hqp : iU, q ip i) (hr : iU, r i ¬p i) {P : Finset ιProp} [DecidablePred P] (hP : WU, W.Nonempty P W (∀ iW, p i) iW, q i) :
    prov α U P = (1 - xU with r x, α x) * iU with q i, α i

    Occurrence-wise selection predicates are scan-computable. If a non-empty world satisfies P exactly when all its occurrences satisfy p and at least one of them satisfies q, then the provenance of P is (𝟙 ⊖ ⊕_{¬p} α) ⊗ (⊕_{q} α).

    This is the algebraic content of the tractability of the MIN, MAX and PICKFIRST comparisons: all of them have this shape, so their provenance is obtained from two running sums over the occurrences of the group.

    Dependency graph
    theorem Having.prov_eq_of_exists {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (q : ιProp) [DecidablePred q] {P : Finset ιProp} [DecidablePred P] (hP : WU, W.Nonempty P W iW, q i) :
    prov α U P = iU with q i, α i

    Special case of prov_eq_of_pointwise with no constraint on the excluded occurrences: the provenance of “some occurrence of the world satisfies q” is ⊕_{q} α.

    Dependency graph

    The MIN and MAX aggregates #

    def Having.minAgg {ι V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) :

    MIN(t) over a world, as an element of WithTop V: the empty world has aggregate value (it is excluded from the possible-world sum anyway).

    Equations
    Instances For
      Dependency graph
      def Having.maxAgg {ι V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) :

      MAX(t) over a world, as an element of WithBot V.

      Equations
      Instances For
        Dependency graph
        def Having.minScan {ι K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (α : ιK) (U : Finset ι) (t : ιV) (op : CompOp) (c : V) :
        K

        The MIN scan. Closed form for the provenance of MIN(t) op c: for each operator, at most two running -sums over the occurrences of the group, combined by one and one . Computing it takes O(|U|) semiring operations, hence polynomial time in data complexity; minScan_correct proves it correct.

        Equations
        Instances For
          Dependency graph
          def Having.maxScan {ι K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (α : ιK) (U : Finset ι) (t : ιV) (op : CompOp) (c : V) :
          K

          The MAX scan, the mirror image of minScan; see maxScan_correct.

          Equations
          Instances For
            Dependency graph
            @[simp]
            theorem Having.le_minAgg_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            c minAgg t W iW, c t i
            Dependency graph
            @[simp]
            theorem Having.lt_minAgg_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            c < minAgg t W iW, c < t i
            Dependency graph
            @[simp]
            theorem Having.minAgg_lt_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            minAgg t W < c iW, t i < c
            Dependency graph
            @[simp]
            theorem Having.minAgg_le_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            minAgg t W c iW, t i c
            Dependency graph
            theorem Having.minAgg_eq_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            minAgg t W = c (∀ iW, c t i) iW, t i = c
            Dependency graph
            @[simp]
            theorem Having.maxAgg_le_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            maxAgg t W c iW, t i c
            Dependency graph
            @[simp]
            theorem Having.maxAgg_lt_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            maxAgg t W < c iW, t i < c
            Dependency graph
            @[simp]
            theorem Having.lt_maxAgg_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            c < maxAgg t W iW, c < t i
            Dependency graph
            @[simp]
            theorem Having.le_maxAgg_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            c maxAgg t W iW, c t i
            Dependency graph
            theorem Having.maxAgg_eq_iff {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] (t : ιV) (W : Finset ι) (c : V) :
            maxAgg t W = c (∀ iW, t i c) iW, t i = c
            Dependency graph

            Provenance of the six MIN(t) op c predicates #

            theorem Having.prov_min_lt {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => minAgg t W < c) = iU with t i < c, α i

            MIN(t) < c: some occurrence of the world has value < c.

            Dependency graph
            theorem Having.prov_min_le {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => minAgg t W c) = iU with t i c, α i

            MIN(t) ≤ c: some occurrence of the world has value ≤ c.

            Dependency graph
            theorem Having.prov_min_ge {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => c minAgg t W) = (1 - xU with t x < c, α x) * iU with c t i, α i

            MIN(t) ≥ c: the world avoids every occurrence of value < c and is non-empty.

            Dependency graph
            theorem Having.prov_min_gt {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => c < minAgg t W) = (1 - xU with t x c, α x) * iU with c < t i, α i

            MIN(t) > c: the world avoids every occurrence of value ≤ c and is non-empty.

            Dependency graph
            theorem Having.prov_min_eq {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => minAgg t W = c) = (1 - xU with t x < c, α x) * iU with t i = c, α i

            MIN(t) = c: the world avoids every occurrence of value < c and contains one of value c.

            Dependency graph
            theorem Having.prov_min_ne {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => minAgg t W c) = iU with t i < c, α i + (1 - xU with t x c, α x) * iU with c < t i, α i

            MIN(t) ≠ c: the two disjoint cases MIN(t) < c and MIN(t) > c.

            Dependency graph

            Provenance of the six MAX(t) op c predicates #

            theorem Having.prov_max_gt {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => c < maxAgg t W) = iU with c < t i, α i

            MAX(t) > c: some occurrence of the world has value > c.

            Dependency graph
            theorem Having.prov_max_ge {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => c maxAgg t W) = iU with c t i, α i

            MAX(t) ≥ c: some occurrence of the world has value ≥ c.

            Dependency graph
            theorem Having.prov_max_le {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => maxAgg t W c) = (1 - xU with c < t x, α x) * iU with t i c, α i

            MAX(t) ≤ c: the world avoids every occurrence of value > c and is non-empty.

            Dependency graph
            theorem Having.prov_max_lt {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => maxAgg t W < c) = (1 - xU with c t x, α x) * iU with t i < c, α i

            MAX(t) < c: the world avoids every occurrence of value ≥ c and is non-empty.

            Dependency graph
            theorem Having.prov_max_eq {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => maxAgg t W = c) = (1 - xU with c < t x, α x) * iU with t i = c, α i

            MAX(t) = c: the world avoids every occurrence of value > c and contains one of value c.

            Dependency graph
            theorem Having.prov_max_ne {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) :
            (prov α U fun (W : Finset ι) => maxAgg t W c) = iU with c < t i, α i + (1 - xU with c t x, α x) * iU with t i < c, α i

            MAX(t) ≠ c: the two disjoint cases MAX(t) > c and MAX(t) < c.

            Dependency graph

            The two scans are correct #

            theorem Having.minScan_correct {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) (op : CompOp) :
            (prov α U fun (W : Finset ι) => op.eval (minAgg t W) c) = minScan α U t op c

            Correctness of the MIN scan. For every comparison operator, the possible-world provenance of the HAVING MIN(t) op c predicate is computed by the scan minScan.

            Dependency graph
            theorem Having.maxScan_correct {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) (op : CompOp) :
            (prov α U fun (W : Finset ι) => op.eval (maxAgg t W) c) = maxScan α U t op c

            Correctness of the MAX scan.

            Dependency graph

            The PICKFIRST aggregate #

            PICKFIRST is the non-commutative aggregate returning the value of the first occurrence of its input sequence. Here the occurrences are ordered by a linear order on ι (the order along which the group is enumerated), so the first occurrence of a world is its -minimum.

            def Having.firstAgg {ι V : Type} [LinearOrder ι] (t : ιV) (W : Finset ι) :

            PICKFIRST(t) over a world: the value of its -first occurrence, with for the empty world.

            Equations
            Instances For
              Dependency graph
              theorem Having.firstAgg_eq {ι : Type} [DecidableEq ι] {V : Type} [LinearOrder V] [LinearOrder ι] (t : ιV) {W : Finset ι} {i : ι} (hiW : i W) (hmin : jW, i j) :
              firstAgg t W = (t i)

              The value of PICKFIRST on a world whose -minimum is i.

              Dependency graph
              def Having.firstScan {ι K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] [LinearOrder ι] (α : ιK) (U : Finset ι) (t : ιV) (op : CompOp) (c : V) :
              K

              The PICKFIRST scan. Closed form for the provenance of PICKFIRST(t) op c: one term per satisfying occurrence, each combining the annotation of that occurrence with the running sum of the annotations of the occurrences that precede it.

              Equations
              Instances For
                Dependency graph
                theorem Having.prov_first_fiber {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] [LinearOrder ι] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) {i : ι} (hiU : i U) :
                (prov α U fun (W : Finset ι) => i W jW, i j) = (1 - xU with x < i, α x) * α i

                Provenance of the worlds whose -first occurrence is a given i: the world contains i and none of the occurrences preceding i.

                Dependency graph
                theorem Having.firstScan_correct {ι : Type} [DecidableEq ι] {K : Type} [CommSemiringWithMonus K] {V : Type} [LinearOrder V] [LinearOrder ι] (h_abs : absorptive K) (h_distrib : mul_sub_left_distributive K) (α : ιK) (U : Finset ι) (t : ιV) (c : V) (op : CompOp) :
                (prov α U fun (W : Finset ι) => op.eval (firstAgg t W) c) = firstScan α U t op c

                Correctness of the PICKFIRST scan. The possible-world provenance of the HAVING PICKFIRST(t) op c predicate is computed by the scan firstScan: the worlds are partitioned according to their first occurrence.

                Dependency graph