tadar

R interface to Task Dynamic Application

https://github.com/samkirkham/tadar

Repository

R interface to Task Dynamic Application

README.Rmd

---
output: github_document
---



```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
```

# tadaR




The goal of tadaR is to provide R functions for processing and plotting output from TADA (TAsk Dynamics Application). This involves operations such as:

1. Run the TADA application from R without interacting with Matlab
2. Reading TADA Matlab output files
3. Processing these files into a more R-friendly format
4. Calculating derived variables, such as smoothing/filtering articualtor variables and calculating velocities.


## Preliminaries

The code works directly on the exported `.mat` MATLAB objects from TADA. It requires the following packages to be installed.

```{r}
library(tadaR)
library(tidyverse)
```


## Running TADA simulation from R

The function `run_tada` is a wrapper that calls Matlab from R, allows command line input to Matlab, and allows you to specify the directory in which Matlab operates (set via `target_dir`). In all cases, `matlab_path` is the location of the Matlab application on your computer. This is optional, as `run_tada` tries to find it automatically, but if it fails to do so then you can specify it in `matlab_path`. Note that the TADA software must be downloaded and named in your Matlab path for this to work. You can download TADA from: https://haskinslabs.org/research/features-and-demos/tada-task-dynamic-model-inter-articulator-speech-coordination/tada

Here are some examples of using `run_tada`.

The first example runs TADA's `gest` function, which we specify using the `tada_command` argument. `gest` takes orthographic or ARPABET input and generates gestural and coupling specifications for the input and saves two output files. The first argument of `gest()` is the output filename and the second argument is the input string, which can be orthographic or ARPABET (e.g. 'KAE1T'). This will generate the files `TVcat.O` and `PHcat.O` in `target_dir`.

```{r eval=FALSE}
run_tada(
  tada_command = "gest('cat' 'cat')"
  target_dir = "path/to/target/dir/",
  matlab_path = "/Applications/MATLAB_R2021a.app/bin/")
```

The next example runs simulations on a pre-existing TV/PH file pair, such as the ones created by the above example. The below assumes that the files `TVcat.O` and `PHcat.O` already exist in the `target_dir`. It then runs the TADA simulation on them and saves all of the output files to `target_dir`.

```{r eval=FALSE}
run_tada(
  tada_comand = "tada 'cat'",
  target_dir="path/to/target/dir/")
```

The following command does something similar to the previous one, but instead of running the simulation on any files called 'cat', it runs the simulations on all paired TV/PH.O files in `target_dir`. This is useful if you have a large directory of TV/PH file pairs and is preferable to running the previous command multiple times, as the below only calls Matlab once, where the previous example starts/stops Matlab each time the function is run.

```{r eval=FALSE}
run_tada(
  tada_comand = "tada 'all'",
  target_dir="path/to/target/dir/")
```

It is also possible to run `gest` and the `tada` simulations using a single line of code. To do this you just the TADA syntax `tada 'cat' 'cat'`, where the first argument is the output filename and the second argument is the input string (which can be orthographic or ARPABET). This is essentially equivalent to running the first two examples given earlier, but in only a single function call.

```{r eval=FALSE}
run_tada(
  tada_comand = "tada 'cat' 'cat'",
  target_dir="path/to/target/dir/")
```

And just for completeness, here's the same as above, but using ARPABET input.

```{r eval=FALSE}
run_tada(
  tada_comand = "tada 'cat' '(KAE1T)'",
  target_dir="path/to/output/dir/")
```

Note that the ARPABET can also be a non-word. For example, if we wanted to synthesise /ba.ba/ then we can do the following, using parentheses to mark syllables.

```{r eval=FALSE}
run_tada(
  tada_comand = "tada 'baba' '(BAE1)(BAE1)'",
  target_dir="path/to/output/dir/")
```

Note that instead of using the above syntax in the `run_tada` commands, you can use a convenience function `tada_stimulus` that will generate this sequence for you given the input. E.g.

```{r}
# returns "tada 'cat' 'cat'"
tada_stimulus("cat")
```

```{r}
# returns "gest 'cat' 'cat'"
tada_stimulus("cat", gest=TRUE)
```

```{r}
# returns "tada 'cat' '(KAE1T)'"
tada_stimulus("cat", "(KAE1T)")
```

Note that this always returns two arguments to tada or gest, so it's not suitable for when you want to evaluate a pair of pre-existing TV/PH files (but a shortcut for that would be unnecessary anyway).


## Loading data and accessing variables

Load a TADA file using `read_tada` function. As an example, we use a TADA synthesis of the word 'pipe', which is available as an example in the `tadaR` package.

```{r}
matfile <- system.file("extdata", "pipe_traj_mv.mat", package = "tadaR")

d <- read_tada(matfile)
```

Now we can plot columns as follows.

```{r}
plot(d$audio$SIGNAL, type = "l") # plot audio signal
```


## Plotting signals and wide data

Generally, it's much easier to use the data in an unnested form, with one column for each variable.

```{r}
d.wide <- unnest_tada(d)
```

The data object now looks like this:

```{r}
head(d.wide)
```

We can now refer directly to variables in order to plot the whole signal file. The new object also has a `Time` column.

```{r}
plot(d.wide$Time, d.wide$TBCL, type = "l", xlab = "Time (secs)", ylab = "TBCL")
```

## Smoothing articulatory variables and obtaining velocities

The `process_tada` function smooths the articulatory variables (such as TT, TD, UL, LL, etc) using a Butterworth filter and then obtains velocities for those variables. It also calculates tangential velocity in the x/y plane for each variable. Note that the function only does this for EMA-like articulatory variables, such as TT, TD, etc - it does not process the tract variables (e.g. TBCD, TBCL) in this way, but I can add that functionality if needed.

```{r}
d.wide <- process_tada(d.wide)
```

## Long formant data

We can also convert the data to long format. This allows us to easily show multiple variables on a single plot, which is very useful for generating something that looks comparable to a gestural score.

First we can create a long data object.

```{r}
d.long <- tidyr::gather(d.wide, "Variable", "Value", -Sample, -Time)
```

In order to get a long audio object, we use `getAudioLong` and add it to the above object. This is necessary because the TADA audio object has a different sampling rate from the articulatory channels.

```{r}
d.long <- dplyr::bind_rows(d.long, getAudioLong(d))
```

The following code shows us which articulators/variables are available to us in `d.long`.

```{r}
unique(d.long$Variable)
```

We can then plot selected variables over time using ggplot. The below code plots the following: Audio, Lip Aperture (LA), TBCL, TBCD, Glottis (GLO).

```{r}
d.long %>% 
  dplyr::filter(Variable %in% c("Audio", "LA", "TBCL", "TBCD", "GLO")) %>% 
  ggplot() +
  aes(x = Time, y = Value) +
  geom_path() +
  facet_wrap(~Variable, ncol = 1, scales = "free_y") + # let *only* y-scaling be free
  theme_minimal()
```


## Further notes and future plans

Coming soon...

1. Vignette showing a more comprehensive example for a typical workflow.

2. Facility to convert TADA files to SSFF format for use with the EMU system: https://github.com/IPS-LMU/emuR This will allow for interactive viewing of gestural synthesis and use of EMU-webApp's gestural annotation algorithms.

3. More specific plotting functions for generating gestural scores.

4. Force-alignment of TADA acoustic data to impose segmental boundaries on the gestural score (TADA itself is not segmental, as it synthesises output based on the overlapping gestures)

If you are interested in further articulatory-to-acoustic synthesis using TADA data then HLSyn may also be helpful: https://github.com/samkirkham/hlsyn

Owner

Citation (CITATION.cff)

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Kirkham"
  given-names: "Sam"
title: "tadaR: R interface to Task Dynamic Application"
version: 0.0.1
doi: https://doi.org/10.5281/zenodo.13329512
date-released: 2024-08-16
url: "https://github.com/samkirkham/tadaR/"

GitHub Events