Sole.jl – Long live transparent modeling!

A framework for symbolic, transparent, and interpretable machine learning!

Manifesto

Symbolic learning is machine learning based on symbolic logic. Its peculiarity lies in the fact that the learned models enclose an explicit knowledge representation, which offers many opportunities: - Verifying that the model's thought process is adequate for a given task; - Learning of new insights by simple inspection of the model; - Manual refinement of the model at a later time.

These levels of transparency (or interpretability) are generally not available with standard machine learning methods, thus, as AI permeates more and more aspects of our lives, symbolic learning is becoming increasingly popular. In spite of this, implementations of symbolic algorithms (e.g., extraction of decision trees or rules) are mostly scattered across different languages and machine learning frameworks.

Enough with this! The lesser and lesser minoritarian theory of symbolic learning deserves a programming framework of its own!

JuliaCon2024 8-minute talk

JuliaCon2023 30-minute talk

The Sole.jl framework

```mermaid graph TD SX[SoleExplorer.jl]

subgraph Group1[ ]
    MAR[<font color="black">ModalAssociationRules.jl</font>]
    MDT[<font color="black">ModalDecisionTrees.jl</font>]
    MDL[<font color="black">ModalDecisionLists.jl</font>]
end
S[<font color="black">Sole.jl</font>]

subgraph Group2[ ]
    SF[<font color="black">SoleFeatures.jl</font>]
    SD[<font color="black">SoleData.jl</font>]
    MD[<font color="black">MultiData.jl</font>]
end

subgraph Group3[ ]
    PHC[<font color="black">SolePostHoc.jl</font>]
    SM[<font color="black">SoleModels.jl</font>]
end

subgraph Group4[ ]
    SL[<font color="black">SoleLogics.jl</font>]
    SR[<font color="black">SoleReasoners.jl</font>]
end
SB[<font color="black">SoleBase.jl</font>]

SX --> MDL 
SX --> MDT
SX --> MAR
SX --> S
SX --> PHC

SL --> SB
SD --> SL
SD --> MD
SM --> SL
S --> SD
PHC --> SM
S --> SM
SF --> SD
MDL --> S
MDT --> S
MAR --> S
SR --> SL

style SX fill:#FFFFFF,stroke:#000000
style SB fill:#FFFFFF,stroke:#000000
style SL fill:#9558B2,stroke:#000000
style SD fill:#4063D8,stroke:#000000
style SM fill:#389824,stroke:#000000
style SF fill:#4063D8,stroke:#000000
style S fill:#FFFFFF,stroke:#000000
style MDL fill:#D56B3D,stroke:#000000
style MDT fill:#D56B3D,stroke:#000000
style MAR fill:#D56B3D,stroke:#000000
style PHC fill:#389824,stroke:#000000
style SR fill:#9558B2,stroke:#000000
style MD fill:#4063D8,stroke:#000000

```

Sole is a collection of Julia packages for symbolic learning and reasoning. Although at an embryonic stage, Sole.jl covers a range of functionality that is of interest for the symbolic community, but it also fills some gaps with a few functionalities for standard machine learning pipelines. At the time of writing, the framework comprehends the three core packages: + SoleLogics.jl provides the logical layer for symbolic learning. It provides a useful codebase for computational logic, which features easy manipulation of: + Propositional and (multi)modal logics (atoms, logical constants, alphabet, grammars, fuzzy algebras); + Logical formulas (random generation, parsing, minimization); + Logical interpretations (or models, e.g., Kripke structures); + Algorithms for model checking (that is, checking that a formula is satisfied by an interpretation). + SoleData.jl provides the data layer for representing logisets, that is, the logical counterpart to machine learning datasets: + Optimized data structures, useful when learning models from datasets; + Support for multimodal data. + SoleModels.jl defines the building blocks of symbolic modeling, featuring: + Definitions for (logic-agnostic) symbolic models (mainly, decision rules/lists/trees/forests); + Basic rule extraction, evaluation and ispection algorithms; + Conversion from DecisionTree.jl and XGBoost.jl; + Support for mixed, neuro-symbolic computation.

Additional packages include: + ModalAssociationRules.jl which implements various strategies to mine association rules from data; + ModalDecisionTrees.jl which allows you to learn decision trees based on temporal logics on time-series datasets, and spatial logics on (small) image datasets; + ModalDecisionLists.jl which implements a sequential covering algorithm to learn decision lists;

A first use case: inspecting decision trees

Ever wondered what to do with a trained DecisionTree? Convert it to Sole, and inspect its knowledge in terms of logical formulas! - Evaluate them in terms of + accuracy (e.g., confidence, lift), + relevance (e.g., support), + interpretability (e.g., syntax height, number of atoms); - Modify them; - Merge them.

Converting to a Sole model

```julia using MLJ using MLJDecisionTreeInterface using DataFrames using Sole

X, y = @load_iris X = DataFrame(X)

train, test = partition(eachindex(y), 0.8, shuffle=true); Xtrain, ytrain = X[train, :], y[train]; Xtest, ytest = X[test, :], y[test];

Train a model

learneddttree = begin Tree = MLJ.@load DecisionTreeClassifier pkg=DecisionTree model = Tree(maxdepth=-1, ) mach = machine(model, Xtrain, ytrain) fit!(mach) fittedparams(mach).tree end

Convert to Sole model

soledt = solemodel(learneddt_tree) ```

Model inspection & rule study

```julia-repl julia> # Make test instances flow into the model, so that test metrics can, then, be computed. apply!(soledt, Xtest, y_test);

julia> # Print Sole model printmodel(soledt; showmetrics = true); ▣ V4 < 0.8 ├✔ setosa : (ninstances = 7, ncovered = 7, confidence = 1.0, lift = 1.0) └✘ V3 < 4.95 ├✔ V4 < 1.65 │├✔ versicolor : (ninstances = 10, ncovered = 10, confidence = 1.0, lift = 1.0) │└✘ V2 < 3.1 │ ├✔ virginica : (ninstances = 2, ncovered = 2, confidence = 1.0, lift = 1.0) │ └✘ versicolor : (ninstances = 0, ncovered = 0, confidence = NaN, lift = NaN) └✘ V3 < 5.05 ├✔ V1 < 6.5 │├✔ virginica : (ninstances = 0, ncovered = 0, confidence = NaN, lift = NaN) │└✘ versicolor : (ninstances = 0, ncovered = 0, confidence = NaN, lift = NaN) └✘ virginica : (ninstances = 11, ncovered = 11, confidence = 0.91, lift = 1.0)

julia> # Extract rules that are at least as good as a random baseline model interestingrules = listrules(soledt, minlift = 1.0, minninstances = 0);

julia> printmodel.(interestingrules; showmetrics = true); ▣ (V4 < 0.8) ∧ (⊤) ↣ setosa : (ninstances = 30, ncovered = 7, coverage = 0.23, confidence = 1.0, natoms = 1, lift = 4.29) ▣ (¬(V4 < 0.8)) ∧ (V3 < 4.95) ∧ (V4 < 1.65) ∧ (⊤) ↣ versicolor : (ninstances = 30, ncovered = 10, coverage = 0.33, confidence = 1.0, natoms = 3, lift = 2.73) ▣ (¬(V4 < 0.8)) ∧ (V3 < 4.95) ∧ (¬(V4 < 1.65)) ∧ (V2 < 3.1) ∧ (⊤) ↣ virginica : (ninstances = 30, ncovered = 2, coverage = 0.07, confidence = 1.0, natoms = 4, lift = 2.5) ▣ (¬(V4 < 0.8)) ∧ (¬(V3 < 4.95)) ∧ (¬(V3 < 5.05)) ∧ (⊤) ↣ virginica : (ninstances = 30, ncovered = 11, coverage = 0.37, confidence = 0.91, natoms = 3, lift = 2.27)

julia> # Simplify rules while extracting and prettify result interestingrules = listrules(soledt, minlift = 1.0, minninstances = 0, normalize = true);

julia> printmodel.(interestingrules; showmetrics = true, syntaxstringkwargs = (; thresholddigits = 2)); ▣ V4 < 0.8 ↣ setosa : (ninstances = 30, ncovered = 7, coverage = 0.23, confidence = 1.0, natoms = 1, lift = 4.29) ▣ (V4 ∈ [0.8,1.65)) ∧ (V3 < 4.95) ↣ versicolor : (ninstances = 30, ncovered = 10, coverage = 0.33, confidence = 1.0, natoms = 2, lift = 2.73) ▣ (V4 ≥ 1.65) ∧ (V3 < 4.95) ∧ (V2 < 3.1) ↣ virginica : (ninstances = 30, ncovered = 2, coverage = 0.07, confidence = 1.0, natoms = 3, lift = 2.5) ▣ (V4 ≥ 0.8) ∧ (V3 ≥ 5.05) ↣ virginica : (ninstances = 30, ncovered = 11, coverage = 0.37, confidence = 0.91, natoms = 2, lift = 2.27)

julia> # Directly access rule metrics readmetrics.(listrules(soledt; minlift=1.0, min_ninstances = 0)) 4-element Vector{NamedTuple{(:ninstances, :ncovered, :coverage, :confidence, :natoms, :lift), Tuple{Int64, Int64, Float64, Float64, Int64, Float64}}}: (ninstances = 30, ncovered = 7, coverage = 0.23333333333333334, confidence = 1.0, natoms = 1, lift = 4.285714285714286) (ninstances = 30, ncovered = 10, coverage = 0.3333333333333333, confidence = 1.0, natoms = 3, lift = 2.7272727272727275) (ninstances = 30, ncovered = 2, coverage = 0.06666666666666667, confidence = 1.0, natoms = 4, lift = 2.5) (ninstances = 30, ncovered = 11, coverage = 0.36666666666666664, confidence = 0.9090909090909091, natoms = 3, lift = 2.2727272727272725)

julia> # Show rules with an additional metric (syntax height of the rule's antecedent) printmodel.(sort(interestingrules, by = readmetrics); showmetrics = (; rounddigits = nothing, additionalmetrics = (; height = r->SoleLogics.height(antecedent(r)))));

▣ (V4 ≥ 1.65) ∧ (V3 < 4.95) ∧ (V2 < 3.1) ↣ virginica : (ninstances = 30, ncovered = 2, coverage = 0.06666666666666667, confidence = 1.0, height = 2, lift = 2.5) ▣ V4 < 0.8 ↣ setosa : (ninstances = 30, ncovered = 7, coverage = 0.23333333333333334, confidence = 1.0, height = 0, lift = 4.285714285714286) ▣ (V4 ∈ [0.8,1.65)) ∧ (V3 < 4.95) ↣ versicolor : (ninstances = 30, ncovered = 10, coverage = 0.3333333333333333, confidence = 1.0, height = 1, lift = 2.7272727272727275) ▣ (V4 ≥ 0.8) ∧ (V3 ≥ 5.05) ↣ virginica : (ninstances = 30, ncovered = 11, coverage = 0.36666666666666664, confidence = 0.9090909090909091, height = 1, lift = 2.2727272727272725)

julia> # Pretty table of rules and their metrics metricstable(interestingrules; metricskwargs = (; rounddigits = nothing, additionalmetrics = (; height = r->SoleLogics.height(antecedent(r))))) ┌────────────────────────────────────────┬────────────┬────────────┬──────────┬───────────┬────────────┬────────┬─────────┐ │ Antecedent │ Consequent │ ninstances │ ncovered │ coverage │ confidence │ height │ lift │ ├────────────────────────────────────────┼────────────┼────────────┼──────────┼───────────┼────────────┼────────┼─────────┤ │ V4 < 0.8 │ setosa │ 30 │ 7 │ 0.233333 │ 1.0 │ 0 │ 4.28571 │ │ (V4 ∈ [0.8,1.65)) ∧ (V3 < 4.95) │ versicolor │ 30 │ 10 │ 0.333333 │ 1.0 │ 1 │ 2.72727 │ │ (V4 ≥ 1.65) ∧ (V3 < 4.95) ∧ (V2 < 3.1) │ virginica │ 30 │ 2 │ 0.0666667 │ 1.0 │ 2 │ 2.5 │ │ (V4 ≥ 0.8) ∧ (V3 ≥ 5.05) │ virginica │ 30 │ 11 │ 0.366667 │ 0.909091 │ 1 │ 2.27273 │ └────────────────────────────────────────┴────────────┴────────────┴──────────┴───────────┴────────────┴────────┴─────────┘ ```

Want to know more?

The formal foundations of the framework are given in giopaglia's PhD thesis: Modal Symbolic Learning: from theory to practice, G. Pagliarini (2024)

Additionally, there's a 10-hour PhD course on YouTube, as well as material for it (including Jupyter Notebooks displaying symbolic AI workflows with Sole).

About

The package is developed and maintained by the ACLAI Lab @ University of Ferrara.

Long live transparent modeling!