libcdict

libcdict: fast dictionaries in C - Published in JOSS (2023)

https://gitlab.com/rob.izzard/libcdict

Repository

A generic associative array, also known as dictionary or hash, library in C, based on uthash and extended to be more user friendly and include integration features such as JSON output.

https://gitlab.com/rob.izzard/libcdict/blob/master/

Libcdict
========

*Libcdict* is a C library that provides a set functions to 
allow native-C construction of associative arrays, known in Perl
as hashes or in Python as dictionaries. Input and output as JSON 
is included. *Libcdict* is designed to be fast and included most of 
the features you would expect in Perl or Python.

Copyright 2023 Robert Izzard.

Repositories at time of writing:
https://gitlab.com/rob.izzard/libcdict

Please see the file LICENCE for licensing conditions, currently 
*libcdict* is released under the GPL.

Please cite the libcdict academic paper(s) and feel free to
donate cash to the Shepreth Hedgehog Hospital as a contribution 
( http://www.swccharity.org/hedgehog-hospital/ ).

*libcdict* uses code from:

* uthash at:
https://github.com/troydhanson/uthash

* ryu at:
https://github.com/ulfjack/ryu

* jsmn at:
https://github.com/zserge/jsmn

* the fast double parser:
https://github.com/lemire/fast_double_parser

These codes are provided under open-source licences
that are compatible with *libcdict*'s GPL.

Many thanks to the authors of those amazing codes!

------------------------------------------------------------

[[_TOC_]]

------------------------------------------------------------

# Installation

Please see the INSTALL file for full installation details.

You will need a C compiler, e.g. `gcc`, `meson`
(https://mesonbuild.com/) and `ninja` (https://ninja-build.org/).
If meson and ninja are not available as packages on your
system, try something like the following from a terminal:

~~~bash
pip3 install meson
pip3 install ninja
~~~


Then, use git to clone *libcdict* from its repository, e.g.

~~~bash
git clone https://gitlab.com/rob.izzard/libcdict.git
~~~

and then, from the *libcdict* directory, run:

~~~bash
meson setup --prefix=$HOME --buildtype=release builddir
cd builddir
ninja install
~~~

which assumes your installation directories are in $HOME (e.g. $HOME/bin, $HOME/lib, etc.).

*libcdict* has been tested with modern versions of gcc (V7.5+) and clang (V9+).

If you use another compiler, you will doubtless run into
difficulties because *libcdict* uses some gcc extensions.
Fortunately, gcc is available to all from https://gcc.gnu.org/ and clang is at https://clang.llvm.org/

Please note:
If you use gcc version >=10 or clang >= 9, you should include the compiler flag `-Wno-sizeof-pointer-div` to prevent spurious warnings. This is done by default.

## OSX requirements

Please install *gawk* (`brew install gawk`) and *timeout* (`brew install coreutils`).

------------------------------------------------------------

# libcdict Instructions : C

## 0. C libcdict headers

Include the cdict header file in your C source code.

```c
    #include 
```

## 1. New cdict struct

To make a cdict called `c`, use:

```c
CDict_new(c);
```

You can also make a new cdict with another cdict as its "parent". The child cdict will inherit the parent's settings such as its error handler and cache use. This is useful when putting one cdict inside another (e.g. see `CDict_nest` below).

```c
CDict_new(new_cdict,parent_cdict);
```

## 2. Free cdict struct

Free the memory associated with a cdict struct, and any nested cdict structs,

- If you just want to free the cdict `c` and any associated metadata (if set)

```c
CDict_free(c);
```

- If you want to free the cdict struct, its metadata and contents of any pointers it contains as data (except other cdicts), use

```c
CDict_free_and_free_contents(c);
```

- If you want to free the cdict struct and have a custom function called to free each entry, use

```c
CDict_free_with_function(c,my_custom_function);
```
where the function should expect a `struct cdict_entry_t *` to be passed to it corresponding to each entry in `c`.

Please note: if a `cdict` is freed which is pointed to as the ancestor of another `cdict` this will probaly cause an error. You should thus free cdicts in the reverse order they are accessed, or just free the ancestor (which will free its children).

## 3. Setting scalars, arrays and pointers, including other cdicts, in your cdict, with implicit type guessing

You can set entries containing scalars, pointers and arrays into the cdict with the `CDict_set()` function. This implicitly guesses the types of the key and value you try to set as well as array sizes.

```c
    /* make new cdict "teams" */
    CDict_new(teams);

    /* just set a key-value pair "Euro 2022 Winners"->"England" in the teams cdict */
    CDict_set(teams,
              "Euro 2022 Winners","England");
```

You can set cdict entries with metadata, for example if you want to set a static metadata string. 
Note: Such metadata is not output as JSON. 

```c
    /*
     * Set a key-value pair, a with static string as metadata.
     * (Static strings require no freeing.)
     */
    CDict_set(teams,
              "Pride of London","Arsenal",
              "League winners 193031, 193233, 193334, 193435, 193738, 194748, 195253, 197071, 198889, 199091, 199798, 200102 and 200304");

```

You can set non-static (allocated memory) in the metadata too. Again, this metadata is not output as JSON.

```c
    /*
     * Set dynamically allocated strings as metadata: these
     * require a function to free them.
     *
     * Here we CDict_string to allocate space for the string in memory
     * and CDict_free_metadata to free it.
     *
     * CDict_string just allocates some memory and sets the string in
     * it. Any string allocated with CDict_string is automatically freed
     * when the cdict (teams in this case) is freed with CDict_free().
     */
    CDict_set(teams,
              "Tottenham","Hotspur",
              CDict_string(teams,"League winners only 195051 and 196061"));
```

You can use a four-argument version of `CDict_set()` to give a reference a function that is called to free any allocated memory.

```c

    void free_metadata_function(struct cdict_t * const cdict,
                                struct cdict_entry_t * entry)
    {
        /* free data in here */
    }
    
    ...

     CDict_set(teams,
               "Tottenham","Hotspur",
               CDict_string(teams,"League winners only 195051 and 196061"),
               free_metadata_function);
```

## 4. Setting scalars, arrays and pointers, including other cdicts, in your cdict, with explicit types

You can set scalars, pointers and arrays in the cdict with the functions `CDict_set_with_types(cdict,key,keytype,value,valuetype)` and `CDict_set_with_types_and_metadata(cdict,key,keytype,value,valuetype,metadata,metadata_free_function)`. These are different to `CDict_set()` in that they do not guess the data's C type, instead you must set this manually.

The available scalar data types available are:
-  `DOUBLE`
-  `LONG_DOUBLE`
-  `INT`
-  `UNSIGNED_INT`
-  `SHORT_INT`
-  `UNSIGNED_SHORT_INT`
-  `LONG_INT`
-  `UNSIGNED_LONG_INT`
-  `LONG_LONG_INT`
-  `UNSIGNED_LONG_LONG_INT`
-  `CHAR`
-  `FLOAT`
-  `STRING` (corresponding to char*)
-  `BOOLEAN`

the pointer data types available are:

-  `INT_POINTER`
-  `UNSIGNED_INT_POINTER`
-  `SHORT_INT_POINTER`
-  `UNSIGNED_SHORT_INT_POINTER`
-  `LONG_INT_POINTER`
-  `UNSIGNED_LONG_INT_POINTER`
-  `LONG_LONG_INT_POINTER`
-  `UNSIGNED_LONG_LONG_INT_POINTER`
-  `VOID_POINTER`
-  `FLOAT_POINTER`
-  `DOUBLE_POINTER`
-  `LONG_DOUBLE_POINTER`
-  `BOOLEAN_POINTER`
-  `CHAR_POINTER`
-  `CDICT`

note: if you want to explicitly use a `char*` pointer, call it a `VOID_POINTER`, i.e. `void*`, to avoid confusion with `STRING` types.
note: `CDICT` means a pointer to a `struct libcdict *`, e.g. as created by a call to `CDict_new()`.

The array types are:

-  INT_ARRAY
-  UNSIGNED_INT_ARRAY
-  SHORT_INT_ARRAY
-  UNSIGNED_SHORT_INT_ARRAY
-  LONG_INT_ARRAY
-  UNSIGNED_LONG_INT_ARRAY
-  LONG_LONG_INT_ARRAY
-  UNSIGNED_LONG_LONG_INT_ARRAY
-  CHAR_ARRAY
-  FLOAT_ARRAY
-  DOUBLE_ARRAY
-  LONG_DOUBLE_ARRAY
-  BOOLEAN_ARRAY
-  VOID_POINTER_ARRAY
-  STRING_ARRAY


Then you can set

```c
    /*
     * Set key = 350.0 (double), value = 33 (int), and a static metadata
     * string "..." with no metadata free function (NULL).
     */
    CDict_set_with_types_and_metadata(cdict,
                                      350.0,CDICT_DATA_TYPE_DOUBLE,
                                      33,CDICT_DATA_TYPE_INT,
                                      "this is a metadata string",
                                      NULL);

    /*
     * Set array of doubles "xarray" in the cdict.
     */
    double xarray[3] = {100.0,200.0,300.0};
    CDict_set_with_types(cdict,
                         "array X of doubles",CDICT_DATA_TYPE_STRING,
                         xarray,CDICT_DATA_TYPE_DOUBLE_ARRAY);
```

Note that all elements in the array must be of the same type (as in C). This is not a limitation, it is a design choice: when you want to have many elements of different types, you can use a cdict instead (see above).

## 5. Set nested variables 

Setting nested variables in the cdict can be done either using the above function calls, or the `CDict_nest(cdict,key,value,...)` function, where "..." is a list of keys which define the nested location (usually statically, i.e. not with `malloc` or `calloc`, allocated scalar types, e.g. numbers or strings). This is a powerful method for setting, or appending to, complicated nested cdicts quickly. Note that if a cdict entry already exists at the nested location, it will be appended if possible. In the case of numbers this means adding them, strings and arrays are concatenated, Booleans are ANDed, while pointers are incremented by casting the appending value to a ptrdiff_t and then adding it. Note that you cannot append a cdict on a cdict: such cases should be prevented using `CDict_contains(cdict,key)` or `CDict_nest_get_entry(cdict,...)`.

```c
    /*
     * You can use the CDict_nest function to put items
     * into a nested cdict structure.
     */

    CDict_new(nestcdict);
    CDict_nest(
        nestcdict, /* cdict */
        1,0.1234, /* key and value */
        "hello","world" /* nest location "hello" -> "world" -> ... */
        );

    CDict_nest(nestcdict,
               2.0,"nested data",
               "hello" /* nest location "hello" -> ... */ );

    CDict_nest(nestcdict,
               TRUE,"Bergkamp Ajax", /* data to nest */
               "Denis",10 /* nest location "Denis" -> 10 -> ... */);

    CDict_nest(nestcdict,
               1,"Bergkamp Arsenal", /* data to nest */
               "Denis","10" /* nest location "Denis" -> "10" -> ... */);

    CDict_print_JSON(nestcdict,CDICT_JSON_SORT);
    printf("\n");

    CDict_free(nestcdict);
```

which will output

> {
>     "hello" : {
>         "world" : {
>             "1" : 0.1234
>         },
>         2e0 : "nested data"
>     },
>     "Denis" : {
>         "10" : {
>             "1" : "Bergkamp Ajax"
>         },
>         "10" : {
>             1 : "Bergkamp Arsenal"
>         }
>     }
> }

Note that both "10" and 10 are valid keys, and they are different, and that the keys in the above are allocated from the stack. You can also do something like the following, which allocates heap memory to a string:

```c

    char * x = NULL;
    int n = 2;
    CDict_asprintf(nestcdict,x,"key",n); // x is now "key2"
    CDict_nest(nestcdict,
               "some data",
               x);
    n=3;
    char * y = NULL;
    CDict_asprintf(nestcdict,y,"key",n); // y is now "key3"
    CDict_nest(nestcdict,
               "some more data",
               y);
```

Note that if you assign the strings with `CDict_string(cdict,...)`, they are automatically freed when freeing the `nestccdict` struct. Use the (libbsd) function `asprintf` directly if you do not want this useful feature.

Note that `CDict_nest`, and associated functions like `CDict_nest_set`, cannot handle `char *` types that are actual pointers to a char. You have two methods for handling `char *` types using the nest functions. 

* You chould treat them `as void *` types and manually cast them back to `char *`. 
* `CDict_nest` returns the new entry in the `cdict`, so you could set the type manually, e.g.

```c
    /* char * type: special case */
    struct cdict_entry_t * const e = CDict_nest("key",(char*)char_pointer,"nest","location");
    e->value.type = CDICT_DATA_TYPE_CHAR_POINTER;
    e->value.format = "%p";
```

The reason for this is that the compiler cannot distinguish between strings (`char[]`) and `char *` automatically, so we assume a string always. Given that you will almost always *actually* be using strings, and that there is a workaround for `char*` types, this is not a problem.

You can manually add pointers to a cdict's "tofree list" with:

```
    CDict_add_to_free_list(nestcdict,
                           x);
```
Pointers added to the "tofree list" are freed when the cdict, in this case `nestcdict`, is freed. This is a useful form of internal garbage collection.

You may also wish to consider a smarter nesting structure that uses only statically allocated keys, such as:
    
```c

    CDict_nest(nestcdict,
               "some data",
               "key",2);
    CDict_nest(nestcdict,
               "some more data",
               "key",3);

```
Static strings are likely to be more efficiently stored in memory.


## 6. Accessing nested cdict data

Accessing nested cdict data is relatively simple. To access an entry, simply call

```c
    struct cdict_entry_t * entry =
           CDict_nest_get_entry(nestcdict,
                                "nested",
                                "location", 
                                ...);
```

If no data is stored at the requested location available, `entry` will be `NULL`.

You can access a pointer to the the data in an entry by setting up a variable of the appropriate type. If there is an error, e.g. if the data does not exist at the required location, this pointer will be `NULL`.

 ```c

    double * x = CDict_nest_get_data_pointer(nestcdict,
                                             "nested",
                                             "location", 
                                             "of",
                                             "scalar double");
```
If you want access to something that's already a pointer, e.g. a cdict nested in a cdict, you have to be a little more careful because you have to cast the `void*` pointer returned by `CDict_nest_get_data_pointer` to `struct cdict_t **`, as shown below.

```c
    struct cdict_t * nested_cdict = 
               *(struct cdict_t **) CDict_nest_get_data_pointer(parent_cdict,
                                                                "nested",
                                                                "location");
```

More usefully, you can access the data itself (probably only works for scalar types):

```c
    double x;
    x = CDict_nest_get_data_value(nestcdict,
                                  x,
                                  "nested",
                                  "location", 
                                  ...);
```
If the data does not exist, an error is raised.

## 7. Access cdict entries by looping over its keys

You can loop over the keys in a cdict, access the entries' keys and values, with code like

```c
    /* Loop over cdict stored in cdict */
    CDict_sorted_loop(cdict,entry)
    {
        char *keystring, *valuestring;
        int ret = CDict_entry_to_key_and_value_strings(entry,keystring,valuestring);
        if(ret < 0)
        {
            printf("error %d setting key/value strings\n",ret);
            exit(0);
        }
        else
        {
            printf("key (type %s) = %s : value (type %s) = %s\n",
                   CDict_key_descriptor(entry->keytype),
                   keystring,
                   CDict_value_descriptor(entry->valuetype),
                   valuestring);
        }
        CDict_Safe_free(keystring);
        CDict_Safe_free(valuestring);
    }
```

If you do not care to sort the keys, use `CDict_loop(...)` instead of `CDict_sorted_loop(...)`.

You can access the cdict entry's value and key individually as strings from an entry using calls to `CDict_key_to_string()` and `CDict_value_to_string()`, e.g.

```c
        char *keystring, *valuestring;
        CDict_entry_to_value_string(entry,
                                    value_string,
                                    NULL);
        CDict_entry_to_key_string(entry,
                                  key_string,
                                  NULL);

```

Note that these key/value to string functions return the number of bytes put in the string (the value string in the case they are both set), or a negative number in case of an error.

## 8. Accessing cdicts through an iterator

As of V1.5 you can access the data in a (nested) cdict using an iterator set up by `CDict_iter()`. This is useful, for example, when outputting a large cdict to JSON to avoid storing the JSON in a large memory buffer. Nested location information is stored in the entries in `iterator->stack->items` which you can access as in the example below. `CDict_iter` returns the entry on the top of the stack, or `NULL` when there are no more entries.

```c
    struct CDict_iterator_t * iterator = NULL;
    Boolean sort = FALSE; /* perhaps sort output? */
    while(TRUE)
    {
        struct cdict_entry_t * const top_entry = CDict_iter(cdict,
                                                            &iterator,
                                                            sort);
        for(size_t i=0; istack->nstack; i++)
        {
            struct cdict_entry_t * this_entry = iterator->stack->items[i] ? iterator->stack->items[i]->entry : NULL;
            /* 
             * do stuff with this_entry
             * ...
             */
        }
        if(!top_entry) break; /* no more data */
    }
    CDict_iter(NULL,&iterator,FALSE); /* free iterator */
```

## 9. cdict to JSON 

You can easily output a cdict's contents to a JSON buffer using. If you just want to output to stdout (the usual case), use `CDict_print_JSON()`. This takes the cdict itself as the first argument, and the second argument chooses whether to sort the cdict or not, it should be either `CDICT_JSON_SORT` or `CDICT_JSON_NO_SORT`. (It is slower to sort the cdict hence you have the option.)

```c
    /*
     * Output to stdout
     */
    CDict_print_JSON(cdict,CDICT_JSON_NO_SORT);

    /*
     * stream is a file stream, e.g. stdout, stderr
     * or something opened with fopen()
     */
    CDict_print_JSON(cdict,CDICT_JSON_SORT,stream);
```

You can also provide more options and capture to a buffer (i.e. in memory) with a call to `CDict_to_JSON()`:

```c
      char * json_buffer = NULL;
      size_t json_size = 0;
      CDict_to_JSON(cdict,
                    json_buffer,
                    json_size,
                    ",", // item separator, defaults to "," if NULL
                    " : ", // key-value separator, defaults to " : " if NULL
                    CDICT_JSON_INDENT,
                    CDICT_JSON_NEWLINES,
                    CDICT_JSON_SORT);
      printf("JSON buffer at %p, size %zu\n",json_buffer,json_size);
      printf("%s\n",json_buffer);
      CDict_Safe_free(json_buffer);
```

If you wish to compress the output so it uses less memory, use this call to `CDict_to_JSON()` instead which does not include whitespace.

```c
      CDict_to_JSON(cdict,
                    json_buffer,
                    json_size,
                    ",",
                    ":",
                    CDICT_JSON_NO_INDENT,
                    CDICT_JSON_NO_NEWLINES,
                    CDICT_JSON_NO_SORT);
```

*libcdict* has limited capability to load JSON data. Most of the limitation is that C arrays must be of a constant type, i.e. an array of ints, or floats, but not an array of some ints and some floats. If you find this is truly a limitation for your work, you could store arrays of `void*` pointers to your data: these can point to anything.
   
You can set a JSON string into a cdict:

```c
    CDict_JSON_buffer_to_CDict(buffer,cdict);
```

You can also load from a file into a cdict:

```c
    CDict_JSON_file_to_CDict(cdict,filename);
```

## 10. Custom output formatting 

Output formatting of keys and values is done automatically, but you can set a custom format at the same time as setting your data in place. For example, the following sets a double as a key and an int as a value, with the format strings `"%.15e"` and `"%d"` respectively. You can use `CDict_set_with_types_and_formats(cdict,key,keytype,keyformat,value,valuetype,valueformat)` and `CDict_set_with_types_formats_and_metadata(cdict,key,keytype,keyformat,value,valuetype,valueformat,metadata,metadata_free_function)`.

```c
      double r1 = 10.0;
      int i = 1234;
      CDict_set_with_types_and_formats(cdict,
                                       r1,
                                       CDICT_DATA_TYPE_DOUBLE,
                                       "%.15e",
                                       i,
                                       CDICT_DATA_TYPE_INT,
                                       "%d");

```

## 11. Delete cdict entry

Delete a cdict entry and its value's contents, if appropriate (e.g. if they are a pointer) with the following.

```c
    CDict_del_and_contents(cdict,entry,TRUE);
```

Delete a cdict entry but do not delete its contents with this.

```c
    CDict_del_and_contents(cdict,entry,TRUE);
```

Delete a reference to a cdict entry with a call like this, but beware this does *not* delete the entry contents so will, unless you free the entry manually, lead to memory leaks. 

```c
    CDict_del(cdict,entry);
```

## 12. Append cdict entry 

Append to a cdict entry, with given key, by value, with the following code. As with `CDict_set()` the key and value types are guessed automatically.

```c
    CDict_append(cdict,key,value);
```

This is very useful for adding to a cdict value: the data type will automatically do its best to handle your request. For example, numbers will be added and strings will be concatenated. If the entry does not exist, or is not a type supported for addition, it is set.

You can also use `CDict_append_with_types(cdict,key,keytype,value,valuetype)` and `CDict_append_with_types_and_metadata(cdict,key,keytype,value,valuetype,metadata,metadata_free_function)` should you require more control.

## 13. Locate an entry 

Find out if the cdict contains an entry with `Cdict_contains`, which returns the entry if it exists, otherwise returns NULL.

```c
    /*
     * Match a key of any type
     */
     cdict_entry_t * entry = CDict_contains(cdict,key);

    /*
     * Match a key of a certain type
     */
     entry = CDict_contains_with_type(cdict,key,keytype);
     
    /*
     * Then, e.g., key and value to strings (so we can output)
     */
    char *keystring, *valuestring;
    CDict_entry_to_value_string(entry,
                                value_string,
                                NULL);
    CDict_entry_to_key_string(entry,
                              key_string,
                              NULL);
     
```

## 14. Sort a cdict

Sort a cdict, e.g. prior to looping (this is implicitly done if you use `CDict_sorted_loop()`).

```c
    /*
     * Sort automatically, e.g. low to high, alphanumerically
     */
    CDict_sort(cdict);

    /*
     * As CDict_sort in reverse.
     */
    CDict_reverse_sort(cdict);
```

Sorting is done as smartly as possible, e.g. numerically or alphabetically.
    
You can also use your own sort function,

```c
    /*
     * Sort using my_sortfunc
     */
    CDict_sort_by_func(cdict, my_sortfunc);

```

where the `sortfunc` should be defined as for the Gnu C standard-library `qsort` function.

## 15. Internal functionality

A word on some internal functionality. Because *libcdict* automatically handles floating-point numbers, it has to make some choices. These come down to:

-  Choosing how to put floats and doubles into hash buckets, i.e. how to "hash" them. We currently do this by truncating the precision of the float or double to the leading `n` bytes, where you can set `n` as shown below. This algorithm is not perfect [^1], but works enough of the time for reasonable choice of `n`. If you have a very large number of hash items, it makes sense to have a larger `n` so there are more buckets. If you choose `n` to be very small (< 3) there will possibly be few buckets so your hash lookups will be slow.

[^1]: Edge cases will fail, but they are rare, e.g. https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/

-  When comparing two floating-point numbers that are in the same bucket for equality, we need to use an algorithm that takes into account floating-point fuzziness. To does that we use `abseps` and `releps`, where `abseps` is the maximum difference between two numbers, and `releps` is the maximum relative difference, for them to be called equal.
- You can set the above parameters with a call to `CDict_set_numerics()` like this.

```c
    CDict_set_numerics(abseps,
                       releps,
                       nbytes_double,
                       nbytes_float);
```

By default, we set `abseps=releps=2.0*DBL_EPSILON`, `nbytes_double=4` and `nbytes_float=3`. You may well need to use different tolerances in your projects.


## 16. Metadata

You can set metadata into cdict entries. This data can be used for anything you like, but most often it is used as a label for some kind of internal processing. Metadata is not output as JSON.

To set metadata when setting an item in the cdict, to set a metadata label string do something like this:

```c
    CDict_set_with_types_and_metadata(cdict,
                                      "2.0",CDICT_DATA_TYPE_STRING,
                                       2.0,CDICT_DATA_TYPE_DOUBLE);
                                       "metadata label string",
                                       NULL);
```

In the above the final argument is `NULL`. This is the `metadata_free_function`: a function to be called, with the cdict entry as an argument, to free the metadata when the cdict itself is freed (i.e. at the end of whatever you are doing). In the case of static strings, like the above example, no free is required. However, if you have a more complicated data structure as metadata you will be required to free this data yourself to prevent memory leaks. Here is an example.

```c
    /*
     * Define a struct to hold our metadata,
     * in this case a label (char *) and a
     * type number (int).
     */
    struct custom_metadata_struct_t {
        char * label;
        int type;
    };

    /*
     * Allocate memory for the struct
     */
    struct custom_metadata_struct_t * my_metadata =
        malloc(sizeof(struct custom_metadata_struct_t));

    /*
     * Use asprintf to put a string in the label:
     * note that this automatically allocates the memory
     * for the label so we must free it later
     */
    if(asprintf(&my_metadata->label,
                "my metadata label")<0)
    {
        fprintf(stderr,"Could not allocate metadata label\n");
        exit(0);
    }

    /*
     * Set the metadata type : this is useful when
     * giving your data custom typing.
     */
    my_metadata->type = 666;

    /*
     * Here we use the GCC extension "auto" to put the function
     * here... you may need to move this function outside the
     * scope of the current subroutine.
     */
    auto void free_my_metadata(struct cdict_entry_t * entry);
    void free_my_metadata(struct cdict_entry_t * entry)
    {
        struct custom_metadata_struct_t * metadata = entry->metadata;
        printf("free metadata at %p with label \"%s\" and type \"%d\"\n",
               entry,
               metadata->label,
               metadata->type
            );
        if(metadata != NULL)
        {
            CDict_Safe_free(metadata->label);
            CDict_Safe_free(metadata);
        }
    }
    /*
     * Now set the metadata
     */
    CDict_set_with_types_and_metadata(cdict,
                                      400.0,CDICT_DATA_TYPE_DOUBLE,
                                      123.0,CDICT_DATA_TYPE_DOUBLE,
                                      my_metadata,
                                      free_my_metadata);
```

If you are using gcc or clang, you can probably set an anonymous function in the call to `CDict_set_with_metadata()`.

To set metadata info with formats, etc., you can use e.g.,

```c
    CDict_set_with_types_formats_and_metadata(cdict,
                                              key,keytype,keyformat,
                                              value,valuetype,valueformat,
                                              metadata,metadata_free_function,
                                              n);
```

To set and get metadata from a cdict entry, use

```c

    CDict_set_metadata(entry,
                       metadata,
                       metadata_free_function);

    CDict_get_metadata(entry);
```


## 17. Configuration information and testing

If you want to acquire information on the configuration of cdict, after running `ninja install`, and assuming your `$PATH` is set to point to wherever you decided to install *libcdict*, run the following in your terminal:

```bash
    
$ cdict-config

```

which should give you

```
Usage:

cdict-config 

where  are:

--prefix : show PREFIX
--destdir : show DESTDIR
--libs : show linker information
--cflags : show compiler flags
--version : show libcdict version
--tests : run a series of tests and show the output in JSON format.

```

Then, e.g.,

```bash

$ cdict-config --version
libcdict 1.4 : git 214:20231206:822f007 git@gitlab.com:rob.izzard/libcdict.git

```

which gives you the version number, git revision and git URL (which will probably be different to those shown above).

Running `cdict-config --tests` shows you the results of the test code in *cdict_tests.c*. 

The `--libs` and `--cflags` are useful if you want to embed *libcdict* in your project (as it is in *binary_c*, see e.g. https://gitlab.com/binary_c).
    
Note that the "--" are optional.

## 18. Error handling

In case of an error, you can set a function pointer as an error handler and set a custom data pointer for it. This means you can pass whatever you like in this pointer to the handler function, and hence deal with the error in any way you like.

The error handler function will be sent the data pointer, error number and a format statement and subsequent arguments so you can recreate the error string that *libcdict* uses by default.

*libcdict*'s action after the error handler is called depends on its return value. 

- If it returns >=0, or no error handler is set, then the error string is shown. If it returns <0 then the error string is not shown.
- If it returns >0, or no error handler is set, *libcdict* exits. If it turns <0 *libcdict* does not exit (this is for testing purposes).

```c
    int cdict_error_handler_function(void * p,
                                     const int error_number,
                                     char * format,
                                     va_list args)
    {
        struct my_data_t * s = (struct my_data_t *)p;

        /* ... do something with s ... */

        /* maybe show the default error string */
        vprintf(format,args);

        /*
         * then exit(...) or longjmp(...) :
         * if you return, cdict will exit.
         */
        exit(error_number);
    }

    struct my_data_t my_data_structure;
    cdict->error_handler = cdict_error_handler_function;
    cdict->error_data    = &my_data_structure;
```

## 19. Parenting

You can set the parent of a cdict using the two-argument version of `CDict_new()`

```c
    CDict_new(new_cdict,parent_cdict);
```

or you can set a parent manually,

```c
    new_cdict->parent = parent_cdict;
```

You can also set the ancestor cdict, which the parent of the parent of the parent ...  This is done automatically by `CDict_new()`.

```c
    new_cdict->ancestor = parent_cdict->ancestor;
```
Both `parent` and `ancestor` are `NULL` when the single-argument form of `CDict_new()` is used.

## 20. Copying a cdict

You can use `CDict_copy()` to copy a cdict. The new cdict has its array content (e.g. strings and arrays of scalars) copied so you can delete the original cdict, if required.

```c
    CDict_new(cdict);
    CDict_new(cdict_copy);
    CDict_copy(cdict,cdict_copy);
```

## 21. Debugging

You have access to *libcdict*'s debugging API. For example, to assert that x is true, use:

```c
    CDict_assert(x);
```

which will exit through *libcdict*'s `cdict_error()` when `x` is `false`.

You can output cdict statistics with:

```c
    CDict_stats(cdict);
```

You can turn on debugging statements using:

```c
    cdict->vb = TRUE;
```

## 22. Thread safety

*libcdict* is not designed to be explicitly thread safe: you'll need to implement some kind of asyncrhonous thread-locking, e.g. with libpthread mutexes, should you wish to use a single cdict struct in multiple threads. That said, as long as you do this, it should just work: there are no problems with global variables (the only global variables are the floating-point epsilons which you will probably never change). 

## 23. Speed vs memory: the force_maps boolean

When comparing cdict keys to determine whether a path exists in a cdict, often the only way, e.g. when the keys are of different type, is to convert the keys to strings. By default these strings are stored so that further comparisons are as fast as possible. This costs RAM, however. You can change this behaviour with the cdict->force_maps boolean.

* If force_maps is TRUE, the default, then the mapped strings are stored, hence future comparisons are fast but memory use is higher.
* If force_maps is FALSE then mapped strings are not stored, future comparisons are slower but memory use is reduced.

You should just choose which is most important to you: RAM or speed.

## 24. External languages via JSON

You can export *libcdict* JSON data (see [point 8 above](#cdict-to-JSON)) to a file or buffer which can easily be imported into *Perl* or *Python*, both of which have JSON modules to do this. For example, in Python try something like this.
    
```python
    # read from a string 
    data = json.loads(string)
    # or read from a file 
    data = json.load(file)
```

One issue that comes up regularly is that the data keys are read in as strings, not floating point or integer numbers as you probably require. You can fix this in *Python* with the following code (from [binary_c-python](https://pypi.org/project/binarycpython/)). 

```python
    
def keys_to_floats(input_dict: dict) -> dict:
    """
    Function to convert all the keys of the dictionary to floats 
    if they can be converted.

    Args:
        input_dict: dict of which we want to turn all the keys to float types if possible
 
    Returns:
        new_dict: dict of which the keys have been turned to float types where possible
    """

    # this adopts the type correctly
    new_dict = type(input_dict)()

    for k, v in input_dict.items():
        # convert key to a float, if we can
        # otherwise leave as is
        try:
            newkey = float(k)
        except ValueError:
            newkey = k

        # act on value(s)
        if isinstance(v, list):
            # list data
            new_dict[newkey] = [
                keys_to_floats(item)
                if isinstance(item, collections.abc.Mapping)
                else item
                for item in v
            ]
        elif isinstance(v, collections.abc.Mapping):
            # dict, ordereddict, etc. data
            new_dict[newkey] = keys_to_floats(v)
        else:
            # assume all other data are scalars
            new_dict[newkey] = v

    return new_dict
    
    
data = json.load(filename)
print('keys loaded in:',data.keys())

data = keys_to_floats(data)
print('keys converted to floats:',data.keys())
```

# libcdict instructions: Fortran

The Fortran module for libcdict is a set of Fortran interfaces that directly call the C API.
It supports the same general functionality defined above with a few limitations, namely:

* The maximum allowed nesting depth is 2 levels.
* Variable types must be defined using iso_c_binding and are currently limited to:
  * `integer(c_int)`
  * `real(c_float)`
  * `real(c_double)`
  * `character(kind=c_char)`
  * `type(c_ptr)`

The module requires Fortran 90 or higher to use iso_c_binding.

The following functions are currently supported in the Fortran API:

1. `CDict_new`
2. `CDict_copy`
3. `CDict_set`
4. `CDict_nest`
5. `CDict_stats`
6. `CDict_print_JSON`
7. `CDict_free`

Most of the above C documentation is applicable to the Fortran API for these methods, so please read that first for details. The following is a brief summary of how to call the Fortran subroutines and a short example of a complete program that uses libcdict in Fortran.

## 0. Fortran cdict module

Include the cdict module file (cdict_fortran_interface.f90) in your Fortran source code.

First, cdict must be installed in the project directory using the above instructions. This allows `cdict_fortran_interface.f90`,`cdict_fortran_api.h`, and `libcdict.so` to be available. The module file (cdict_fortran_interface.f90) must be compiled with your Fortran source code and the cdict library (libcdict.so) must be linked. See the full example below to see how to do this with a Makefile.

Once your have cdict installed and the necessary files available, you can include it in your Fortran code with:

```fortran
    use libcdict_m
    use, intrinsic :: iso_c_binding
    implicit none
```

## 1. New Fortran cdict

To make a cdict:

```fortran
    type(c_ptr) :: c
    c = CDict_new()
```

And to make a new cdict with another cdict as its "parent":

```fortran
    type(c_ptr) :: c, p
    p = CDict_new()

    c = CDict_new(p)
```

## 2. Free Fortran cdict

To free a cdict and any associated metadata (if set):

```fortran
    type(c_ptr) :: c
    c = CDict_new()

    call CDict_free(c)
```

## 3. Set data in Fortran cdict 

To set entries into a cdict:

```fortran
    type(c_ptr) :: c
    c = CDict_new()

    ! Set a key-value pair 1->100.91 in the c cdict
    call CDict_set(c, 1, 100.91)
```

Like C, this guesses the types of the key and value you try to set as well as array sizes.

To set a pointer, you will need to declare the variable as a `target` and pass the `c_loc()` of the variable to `CDict_set()`:

```fortran
    type(c_ptr) :: c
    integer(c_int), target :: key
    c = CDict_new()
    key = 1

    ! Set a pointer to an integer as the key in the c cdict
    call CDict_set(c, c_loc(key), 100.91)
```

Strings are passed to libcdict by passing a `c_char`. It's worthwhile to trim the string before sending it to libcdict, otherwise an extra space at the end may be included. Also, strings in C are delimited with a NULL character, so it's a good idea to append one to the end of the passed string as well:

```fortran
    type(c_ptr) :: c
    character(kind=c_char, len=10) :: string_key
    c = CDict_new()

    ! Set a key-value pair "First key"->100.91 in the c cdict
    string_key = "First key"
    call CDict_set(c, trim(string_key)//C_NULL_CHAR, 100.91)
```

## 4. Nest data in Fortran cdict

To set nested variables in a cdict:

```fortran
    type(c_ptr) :: c
    integer(c_int), target :: key
    c = CDict_new()

    ! Set a key-value pair 1->100.91 in the cdict at location 999
    call CDict_nest(c, 1, 100.91, 999)
    ! Set a key-value pair 2->200.91 in the cdict at location 999->2
    call CDict_nest(c, 2, 200.91, 999, 2)
```

Note that `CDict_nest` currently has a maximum nesting depth of 2. To achieve a deeper nesting level, set a cdict inside a cdict with repeated calls to `CDict_nest`. If you know how to achieve deeper nesting in Fortran, please let us know!

## 5. Copy Fortran cdict

To copy a cdict `c` into a new cdict `c_copy`, you must first make the new cdict with `CDict_new()` and then call `CDict_copy`:

```fortran
    type(c_ptr) :: c, c_copy
    c = CDict_new()
    call CDict_set(c, 1, 100.91)

    ! Copy the current state of c into c_copy
    c_copy = CDict_new()
    call CDict_copy(c, c_copy)
```

## 6. Fortran cdict to JSON

To print the contents of a cdict `c`, call `CDict_print_JSON()` using one of the sorting constants defined in the module:

```fortran
    type(c_ptr) :: c
    c = CDict_new()
    call CDict_set(c, 2, 200.91)
    call CDict_set(c, 1, 100.91)

    call CDict_print_JSON(c, CDICT_JSON_SORT)
    call CDict_print_JSON(c, CDICT_JSON_NO_SORT)
```

The constants are `CDICT_JSON_SORT` and `CDICT_JSON_NO_SORT` and are defined as `.true.` and `.false.`, respectively.

## 7. Fortran cdict statistics

To print the statistics of a cdict `c`:

```fortran
    type(c_ptr) :: c
    c = CDict_new()

    call CDict_stats(c)
```

# Fortran EXAMPLE

Here is a full example of a Fortran file that uses cdicts along with a Makefile to
compile it.

```fortran
! cdict_example.f90
program cdict_example
  use libcdict_m
  use, intrinsic :: iso_c_binding
  implicit none

  type(c_ptr) :: dict0, dict1, dictc
  integer(c_int) :: int_key, int_val
  ! Using 'target' here to demonstrate setting pointers
  real(c_float), target :: float_key, float_val
  real(c_float) :: nest_key, nest_val
  real(c_double) :: double_key, double_val
  character(kind=c_char, len=10) :: string_key

  ! Creating new cdicts
  dict0 = CDict_new()
  dict1 = CDict_new(dict0)

  ! Setting key->value pairs
  float_key = 1.0
  float_val = 10.0
  call CDict_set(dict0, float_key, float_val)
  double_key = 2.0
  double_val = 20.0
  call CDict_set(dict0, double_key, double_val)
  int_key = 1
  int_val = 10
  call CDict_set(dict0, int_key, int_val)

  ! Setting a pointer
  call CDict_set(dict1, c_loc(float_key), c_loc(float_val))

  ! Setting a cdict
  string_key = "First key"
  call CDict_set(dict1, trim(string_key), dict0)

  ! Nesting key->value pairs
  nest_key = 100.0
  nest_val = 1000.0
  call CDict_nest(dict0, nest_key, nest_val, float_val)
  call CDict_nest(dict0, nest_key, nest_val, float_val, nest_val)

  ! Copying a cdict
  dictc = CDict_new()
  call CDict_copy(dict0, dictc)

  ! JSON output
  call CDict_print_JSON(dict0, CDICT_JSON_SORT)
  call CDict_print_JSON(dict1, CDICT_JSON_NO_SORT)
  call CDict_print_JSON(dictc, CDICT_JSON_NO_SORT)

  ! Stats
  call CDict_stats(dict0)
  call CDict_stats(dict1)
  call CDict_stats(dictc)

  ! Freeing cdicts
  call CDict_free(dict0)
  call CDict_free(dict1)
  call CDict_free(dictc)
end program cdict_example
```

```makefile
# Makefile
FC = gfortran

# Need to add this path to $LD_LIBRARY_PATH before it can run
LIB_DIRS = -Llib/
LIBS = -lcdict
INC_DIRS = -Iinclude/cdict/
FCFLAGS = -g -fcheck=all -Wall $(INC_DIRS)

TARGET = run

# Ordering here determines compilation order
SRCS = include/cdict/cdict_fortran_interface.f90 cdict_example.f90

# Any prerequisites beyond the source file must be predefined
OBJS = $(addsuffix .o,$(basename $(SRCS)))

$(TARGET) : $(OBJS)
	$(FC) $(LIB_DIRS) -o $@ $^ $(LIBS)

%.o: %.f90
	$(FC) -c $(FCFLAGS) -o $@ $<

clean:
	rm -f *.mod *.o $(TARGET)
```

Here is the output of the above example:

```json
{
    "1" : 1e1,
    "1" : 10,
    "2e0" : 2e1,
    "1e1" : {
        "1e2" : 1e3,
        "1e3" : {
            "1e2" : 1e3
        }
    }
}
{
    "0x7ffcb98443a4" : "0x7ffcb98443a0",
    "First key" : "0x55d2cf155910"
}
{
    "1" : 1e1,
    "2e0" : 2e1,
    "1" : 10,
    "1e1" : {
        "1e2" : 1e3,
        "1e3" : {
            "1e2" : 1e3
        }
    }
}
Cdict 0x55d2cf155910 : nkeys 7, nvalues 7, max_depth 2  
Cdict 0x55d2cf155980 is not its own ancestor (which is 0x55d2cf155910): find the ancestor and call cdict_stats() on it.
Cdict 0x55d2cf157030 : nkeys 7, nvalues 7, max_depth 2
```

# Authors

Of libcdict, Robert Izzard. 
Daniel Nemergut very kindly contributed the Fortran interface.
David Hendriks performed many tests of the code, by using it in his binary_c-python project https://gitlab.com/binary_c/binary_c-python, and helped with documentation.

External code from and probably modified in *libcdict*:
* uthash by Troy Hanson and Arthur O'Dwyer
* ryu by Ulf Adams 
* jsmn by Serge Zaitsev
* fast double parser by Daniel Lemire

You have no guarantee that any part of this code works and the author (Robert Izzard) accepts no liability for anything. Please read the LICENCE and the licences for uthash, ryu, jsmn and fast double parser which are all provided in this repository (see below).

Originally part of the binary_c project https://gitlab.com/binary_c/


*libcdict* uses uthash, (c) 2003-2018, Troy D. Hanson, maintained by Arthur O'Dwyer
http://troydhanson.github.com/uthash/
Please see its usage licence in uthash.h.

*libcdict* uses the code from ryu, (c) Ulf Adams
https://github.com/ulfjack/ryu
This is licenced under the Boost licence, see
ryu/LICENSE-Boost
(ryu is also released under the Apache2 licence ryu/LICENSE-Apache2)
Read their paper at https://dl.acm.org/doi/10.1145/3296979.3192369

*libcdict* uses jsmn, by Serge Zaitsev, distributed under an MIT LICENCE
https://github.com/zserge/jsmn https://zserge.com/jsmn/

*libcdict* uses code from the fast double parser (c) Daniel Lemire
https://github.com/lemire/fast_double_parser
which is released under the Apache licence

Many thanks to the authors of these codes, without you
*libcdict* would not be what it is.

# Community guidelines

## Contribution to the software

Should you wish to contribute to the software, please submit bug reports and new code via libcdict's gitlab page https://gitlab.com/rob.izzard/libcdict through the Issues and Merge Requests sections, respectively. You can contact the authors through gitlab 

## Report issues or problems with the software

Should you wish to report issues software, please do this via libcdict's gitlab page "issues" feature. 

## Seek support

You can seek support by raising issues on libcdict's gitlab page.

Owner

Committers

Last synced: about 1 year ago

Issues and Pull Requests

Last synced: 10 months ago