TransformSpecifications.jl

TransformSpecifications.jl provides tools to define explicitly-specified transformation components. Such components can then be used to construct pipelines and DAGs that are themselves composed of individual explicitly-specified components. Additionally, they can be easily wrapped to catch and nicely handle both expected and unexpected specification and transformation violations.

Basic transform example

A basic TransformSpecification is constructed from the input and output specifications and the transformation function to be applied:

```julia julia> using TransformSpecifications julia> ts = TransformSpecification(String, Integer, length)

Valid input

julia> transform!(ts, "greetings") 9

Invalid input

julia> transform!(ts, 92) ERROR: ArgumentError: Input doesn't conform to specification String ```

A NoThrowTransform facilitates the case where a transformation must be robust to unexpected errors---instead of throwing an error, it returns the violation in a NoThrowResult:

```julia julia> ntt = NoThrowTransform(String, Integer, length)

Valid input:

julia> transform!(ntt, "greetings") NoThrowResult{Int64}: Transform succeeded ✅ result: 9

Invalid input:

julia> transform!(ntt, 92) NoThrowResult{Missing}: Transform failed ❌ Input doesn't conform to specification String. Details: MethodError(convert, (String, 92), 0x0000000000007f48)

Invalid output (from the transform function) also throws:

julia> ts = NoThrowTransform(String, String, length) julia> transform!(ts, "foo") NoThrowResult{Missing}: Transform failed ❌ Output doesn't conform to specification NoThrowResult{String}; is instead a NoThrowResult{Int64} ```

NoThrowDAG example

A NoThrowDAG facilitates composing multiple specified transforms (DAGSteps) into a DAG, such that any errors errors encountered during the application of the DAG will be returned "nicely" (i.e., as a NoThrowResult object with the source of failure noted in the violations field) rather than thrown as an exception:

```julia julia> using Legolas: @schema, @version

julia> @schema "one-var" Huzzah julia> @version HuzzahV1 begin var::String end

julia> @schema "two-var" Hooray julia> @version HoorayV1 begin var1::String var2::String end

Say we have three functions we want to chain together:

julia> fna(x::HuzzahV1) = HuzzahV1(; var=x.var * "a") julia> fnb(x::HuzzahV1) = HuzzahV1(; var=x.var * "b") julia> fnc(x::HoorayV1) = HuzzahV1(; var=x.var1 * x.var2 * "c")

First, specify these functions as transforms: what is the specification of the

function's input and output?

julia> stepatransform = NoThrowTransform(HuzzahV1, HuzzahV1, fna) julia> stepbtransform = NoThrowTransform(HuzzahV1, HuzzahV1, fnb) julia> stepctransform = NoThrowTransform(HoorayV1, HuzzahV1, fn_c)

Next, set up the DAG between the upstream outputs into each step's input:

julia> stepbassembler = inputassembler(upstream -> (; var=upstream["step_a"][:var])) julia> stepcassembler = inputassembler(upstream -> (; var1=upstream["step_a"][:var], var2=upstream["step_b"][:var]))

julia> steps = [DAGStep("stepa", nothing, stepatransform), DAGStep("stepb", stepbassembler, stepbtransform), DAGStep("stepc", stepcassembler, stepc_transform)] julia> dag = NoThrowDAG(steps)

output

NoThrowDAG (HuzzahV1 => HuzzahV1): 🌱 stepa: HuzzahV1 => HuzzahV1: `fna · step_b: HuzzahV1 => HuzzahV1:fnb` 🌷 stepc: HoorayV1 => HuzzahV1: fn_c

Call DAG on valid input:

julia> transform!(dag, HuzzahV1(; var="initial_str"))

output

NoThrowResult{HuzzahV1}: Transform succeeded ✅ result: HuzzahV1: (var = "initialstrainitialstrabc",)

Call DAG on invalid input:

transform!(dag, HoorayV1(; var1="wrong", var2="input schema"))

output

NoThrowResult{Missing}: Transform failed ❌ Input to step step_a doesn't conform to specification HuzzahV1. Details: ArgumentError("Invalid value set for field var, expected String, got a value of type Missing (missing)") ```

Finally, a diagram for the DAG can be generated as a mermaid plot: julia mermaidify(dag) which renders automatically when included in a mermaid markdown block in GitHub: ```mermaid flowchart

%% Define steps (nodes) subgraph OUTERLEVEL[""] direction LR subgraph STEPA[Step a] direction TB subgraph STEPAInputSchema[Input: HuzzahV1] direction RL STEPAInputSchemavar{{"var::String"}} class STEPAInputSchemavar classSpecField end subgraph STEPAOutputSchema[Output: HuzzahV1] direction RL STEPAOutputSchemavar{{"var::String"}} class STEPAOutputSchemavar classSpecField end STEPAInputSchema:::classSpec -- fna --> STEPAOutputSchema:::classSpec end subgraph STEPB[Step b] direction TB subgraph STEPBInputSchema[Input: HuzzahV1] direction RL STEPBInputSchemavar{{"var::String"}} class STEPBInputSchemavar classSpecField end subgraph STEPBOutputSchema[Output: HuzzahV1] direction RL STEPBOutputSchemavar{{"var::String"}} class STEPBOutputSchemavar classSpecField end STEPBInputSchema:::classSpec -- fnb --> STEPBOutputSchema:::classSpec end subgraph STEPC[Step c] direction TB subgraph STEPCInputSchema[Input: HoorayV1] direction RL STEPCInputSchemavar1{{"var1::String"}} class STEPCInputSchemavar1 classSpecField STEPCInputSchemavar2{{"var2::String"}} class STEPCInputSchemavar2 classSpecField end subgraph STEPCOutputSchema[Output: HuzzahV1] direction RL STEPCOutputSchemavar{{"var::String"}} class STEPCOutputSchemavar classSpecField end STEPCInputSchema:::classSpec -- fnc --> STEPCOutputSchema:::classSpec end

%% Link steps (edges) STEPA:::classStep -..-> STEPB:::classStep STEPB:::classStep -..-> STEPC:::classStep

end OUTERLEVEL:::classOuter ~~~ OUTERLEVEL:::classOuter

%% Styling definitions classDef classOuter fill:#cbd7e2,stroke:#000,stroke-width:0px; classDef classStep fill:#eeedff,stroke:#000,stroke-width:2px; classDef classSpec fill:#f8f7ff,stroke:#000,stroke-width:1px; classDef classSpecField fill:#fff,stroke:#000,stroke-width:1px; ```

For more TransformSpecification.jl details and examples, see the documentation!