Installation

Witter is an experimental library, and as such is not yet available in build automation tools (Gradle, etc.)

To use Witter in your project, first build its .jar file using Gradle's build task. The .jar file is generated under the project root in /build/libs. This file should be copied to your own project's libs folder, and then added as a dependency in your build automation tool of choice. For example, in Gradle: kotlin dependencies { implementation(files("libs/witter-0.5.6.jar")) } Note, of course, that the file name can change when updates for Witter are released, and should be changed in your dependency specification accordingly.

You may additionally need to specify dependencies for the Strudel and ANTLR libraries.

Specifying Reference Solutions

Annotating Reference Solutions

One can define the tester cases for a given exercise by writing a reference solution in a Java method, annotated with a header comment that defines the different tester inputs and the metrics that should be measured during tester execution. The content of the comments has to obey Witter’s Test Specification Language (TSL), whose syntax is similar to Java’s annotation syntax: java /* @Test(new int[] { 1, 2, 3, 4, 5 }) @Test(new int[] { 2, 4, 6 }) @CountLoopIterations @CountArrayReads @CheckSideEffects */ public static int sum(int[] a) { ... }

Instantiating Test Specifications Using Witter’s DSL

Test cases can be configured to use any number of white-box metrics either throughout the test or within a bounded scope (using directive). As in the initial version of Witter, evaluation metrics can be optionally instantiated with a margin parameter that specifies an acceptable deviation interval from the reference value.

An object can be created using the new directive by passing the name of the class to instantiate followed by a list of arguments to one of the class constructors. References to the created objects can be stored using ref. Class methods can be invoked by using the call directive on a previously declared reference. A sequence of these directives defines a stateful test case.

The call directive is used by specifying the name of the method to be invoked and a list of arguments. We may use the "dot notation" to perform calls on instance methods given its reference (ref.call(...)), as opposed to calling an instance method by passing its reference as the first argument (call("methodName", ref, ...)). For every call, the return values of the evaluated method are compared to the reference solution, allowing for regular black-box testing. Additionally, if the optional expected argument is passed, Witter will assert that both the reference solution and the solution under evaluation produce the expected result.

The following example illustrates Witter's DSL with a test suite for list data structures, containing two test cases.

```kotlin val test = TestSuite(referencePath = "path/to/reference/solution/ArrayList.java") { Case("testContains") { // Create new object and store a reference to it val list = ref { new("ArrayList", 3) }

    // Executed without white-box metrics (black-box only)
    list.call("size", expected = 0)
    list.call("add", "hello")
    list.call("size", expected = 1)
    list.call("add", "world")
    list.call("size", expected = 2)

    using(CountLoopIterations() + CountArrayReadAccesses()) {
        // These calls compare loop iterations
        call("add", list, "algorithm")
        call("size", list, expected = 3)
    }
}

// All the calls within this case compare loop iterations
Case(CountLoopIterations(), "testIsEmpty") {
    val list = ref { new("ArrayList", 3) }
    list.call("isEmpty", expected = true)
    list.call("add", "hello")
    list.call("add", "icpec")
    list.call("isEmpty", expected = false)
}

} ```

Testing Arbitrary Solutions

As Witter is designed for third-party integration, it provides a form of executing the tests programmatically. Tests are executed providing an annotated reference solution or a test suite created using the DSL as described, and a solution that one wishes to assess.

Using an annotated reference solution, one can execute: kotlin val tester: Test = Test("ReferenceSolution.java") val results: List<TestResult> = tester.execute("Solution.java") For a test suite created using the DSL, the process is similar: kotlin val test = TestSuite(referencePath = "path/to/reference/Solution.java") val results = test.apply(subjectPath = "path/to/submitted/Solution.java")

The results consist of a list of feedback items for each aspect defined in the tester specification, holding the following information: - a flag indicating success or failure; - which kind of metric has failed (recall Table 1); - the location of code elements involved in the failed tests (e.g., procedure, parameters, loop structures); - a human-readable descriptive feedback message.

Witter currently supports the following runtime metrics.

| Metric | TSL Annotation
(DSL is identical, without the @) | Verification | |--------------------|-----------------------------------------------------------|---------------------------------------------------------------------------------------------------| | Return values | @Test([...args]) | Return value is equal to reference solution. Multiple annotations can be used. | | Side effects | @CheckSideEffects | Side effects on arguments (presence and absence) are the same to those of the reference solution. | | Loop iterations | @CountLoopIterations([threshold]) | Total number of loop iterations matches the one of the reference solution. | | Array allocations | @CheckArrayAllocations | The array allocations match those of the reference solution (component types and lengths). | | Array reads | @CountArrayReads([threshold]) | The number of array read accesses is the same as in the reference solution. | | Array writes | @CountArrayWrites([threshold]) | The number of array write accesses is the same as in the reference solution. | | Object allocations | @CheckObjectAllocations | The number of object allocations and their types match those of the reference solution. | | Recursive calls | @CountRecursiveCalls([threshold]) | The number of recursive calls matches the one of the reference solution. |

Examples

Annotated Reference Solutions

Tests Specified Using the DSL

See It in Action 😎

The following is an example of how Witter could be integrated into an existing development system, using a simple GUI custom-made for example purposes.

Citations

If you use or reference Witter in your academic work, you should cite the relevant following publication(s).

Contacts

If you have any questions regarding Witter, its development process, or the related academic publications, feel free to contact the authors: - Afonso B. Caniço - ambco@iscte-iul.pt (Principal Author) - André L. Santos - andre.santos@iscte-iul.pt (Advisor)

Credit

The Witter library is authored, developed, and currently maintained by Afonso B. Caniço. (That's me! :D)

Credit for the Strudel library, which is used within Witter as the runtime environment for code execution, goes to André L. Santos, its author and main contributor. Contributions to Java translation and minor contributions to some aspects of the Strudel library were made by Afonso B. Caniço.