(Your)-Transient Auxiliary Library

bueno el readme ya tal

y-tal (or just tal) is a Python library with many utilities for people who work in the development of non-line-of-sight imaging techniques. This library provides different tools to generate and analyze data and implementations of different non-line-of-sight reconstruction algorithms. Some parts are also accessible through command line for for ease of use.

Authors: Diego Royo, Pablo Luesia

Installation

Working with HDF5 files requires the following packages:

sudo apt install libhdf5-dev

You will also need the required packages with the included requirements.txt file in this repo.

pip3 install -r requirements.txt

To install tal you have the following options:

1) Latest published version (recommended):

pip3 install y-tal

2) Latest version in GitHub (more features, more unstable):

pip3 install git+https://github.com/diegoroyo/tal

Usage:

Python interface

```python import tal

data = tal.io.readcapture('capture.hdf5') tal.plot.xyinteractive(data, cmap='nipy_spectral') ```

Command line

Some Python functions also have an interface through the command line, including all their parameters:

tal plot xy_interactive capture.hdf5 --cmap nipy_spectral

``` ❯ tal -h usage: tal [-h] [-v] {config,render,plot} ...

Y-TAL - (Your) Transient Auxiliary Library - v0.10.2

positional arguments: {config,render,plot} Command config Edit the TAL configuration file render Create, edit or execute renders of simulated NLOS scene data captures plot Plot capture data using one of the configured methods

optional arguments: -h, --help show this help message and exit -v, --version show program's version number and exit ```

tal render: Command line tool to render mitransient scenes

You will need to have mitransient installed in your PC. You can install just via pip install mitransient or follow their README for custom installation instructions.

[!IMPORTANT] On the first run of tal render, it will ask you if you have installed Mitsuba 2 (mitsuba2-transient-nlos) or 3 (mitransient aka. mitsuba3-transient-nlos), and (only if you have compiled Mitsuba youself) the location of your installation folder: for Mitsuba 2, please use the root folder of the repo (i.e. /path/to/mitsuba2-transient-nlos); for Mitsuba 3 you'll have to use the root folder of your custom mitsuba instalation (i.e. /path/to/mitsuba3). If at any time you need to switch from Mitsuba 2 to 3 or vice-versa, or want to switch your installation folder, please use the tal config command.

mitransient must be installed in your device. On your first tal render <scene> command, it will detect (or ask you) where your installation folder is located, and will execute the necessary mitsuba commands and generate the specified scene XML files.

You can find examples for how to render a scene in the examples folder of this repository. You can always use the tal render -h command too:

``` ❯ tal render -h usage: tal render [-h] [-t THREADS] [-s SEED] [-n NICE] [-q] [-r] [--no-steady] [--no-logging] [--no-partial-results] [config_file ...]

positional arguments: configfile Can be: 1) Path to a TAL scene config YAML file 2) Path to a TAL scene directory (must have a YAML file inside with the same name as the directory) 3) 'new <foldername>' to create a new folder (i.e. tal render new )

optional arguments: -h, --help show this help message and exit -t THREADS, --threads THREADS Number of threads -s SEED, --seed SEED Random seed for the sampler. Without setting this value to different values, the same results will be produced everytime. -n NICE, --nice NICE Change +/- in nice factor. Positive values = lower priority. Negative values = higher priority (needs sudo) -q, --quiet Disable progress bars and other verbose outputs -r, --dry-run Do not execute mitsuba, just print out the commands that would be executed --no-steady Disable generation of steady state images --no-logging Disable logging of mitsuba output --no-partial-results Remove the "partial" folder which stores temporal data after creating the final hdf5 file (e.g. multiple experiments for confocal/exhaustive) ```

tal plot: Visualize time-resolved capture data

Used to visualize the capture data (maybe generated using tal render). It accepts HDF5 files in a format compatible with TAL.

``` ❯ tal plot -h usage: tal plot [-h] [--normalize NORMALIZE] [--opacity OPACITY] [--slider-step SLIDERSTEP] [--slider-title SLIDERTITLE] [--y Y] [--slice-axis SLICE_AXIS] [--t-end TEND] [--color COLOR] [--cmap CMAP] [--labels LABELS] [--title TITLE] [--x X] [--a-min A_MIN] [--backgroundcolor BACKGROUNDCOLOR] [--size-x SIZE_X] [--t-start TSTART] [--t-step TSTEP] [--size-y SIZEY] [--a-max AMAX] preset [capturefiles ...]

positional arguments: preset Plot method. Can be one of: amplitudephase tcomparison txinteractive txyinteractive tyinteractive volume xygrid xyinteractive capturefiles One or more paths to capture files

optional arguments: -h, --help show this help message and exit --normalize NORMALIZE --opacity OPACITY --slider-step SLIDERSTEP --slider-title SLIDERTITLE --y Y --slice-axis SLICEAXIS --t-end TEND --color COLOR --cmap CMAP --labels LABELS --title TITLE --x X --a-min AMIN --backgroundcolor BACKGROUNDCOLOR --size-x SIZEX --t-start TSTART --t-step TSTEP --size-y SIZEY --a-max AMAX ```

tal reconstruct: Implementation of multiple non-line-of-sight reconstruction algorithms

NOTE: No command-line version for now

You can check the implemented algorithms here. As of Nov. 2023, implemented: backprojection, filtered backprojection, and different phasor-field cameras.

You can find examples for how use the reconstruction algorithms in the examples folder of this repository. Note that to test the reconstruction algorithms you will need to have a HDF5 capture file. If you don't, please check the tal render section or convert your data to a format usable by tal.

Filtering the impulse response H

```python import tal

data = tal.io.read_capture('capture.hdf5')

for more info on the parameters: https://github.com/diegoroyo/tal/blob/master/tal/reconstruct/init.py#L25

NOTE: if you use fbp, pf or pf_dev, you do not need to perform this filtering step

data.H = tal.reconstruct.filterH(data, filtername='pf', wlmean=0.05, wlsigma=0.05) ```

Selecting the bounding volume for the reconstruction

```python import tal import numpy as np

data = tal.io.read_capture('capture.hdf5')

Option 1: You can create it manually:

volume_xyz = np.array(...) # (x, y, z, 3) or (x, y, 3) or (n, 3) shape

Option 2: Create a volume from two points and a scalar resolution:

volumexyz = tal.reconstruct.getvolumeminmaxresolution(minimalpos=np.array([-3, -2, -1]), maximalpos=np.array([3, 2, 1]), resolution=0.01) print(volumexyz.shape) # (600, 400, 200, 3)

Option 3: Create a volume coplanar to the relay wall, displaced by a distance d

volumexyz = tal.reconstruct.getvolumeprojectrw(data, depths=[1.0, 1.5, 2.0, 2.5, 3.0]) print(volume_xyz.shape) # (sx, sy, 5, 3) where sx, sy are SPAD scan dimensions on X and Y axes ```

You can now use volume_xyz to specify the reconstruction volume for the bp, fbp or pf_dev reconstruction methods.

bp

Implementation of the backprojection algorithm [Velten2012].

```python

follow steps above to read data and obtain volume_xyz

H1 = tal.reconstruct.bp.solve(data, volumexyz=volumexyz, camerasystem=tal.enums.CameraSystem.DIRECT_LIGHT)

the camera_system parameter specifies the concrete camera implementation in the phasor-field framework

by default most papers use the DIRECT_LIGHT equivalent so you probably want to leave it as-is

visualize your result

tal.plot.amplitudephase(H1)
```

fbp, pf_dev

Filtered backprojection [Velten2012] and the pf_dev implementation [Liu2019] of phasor fields accept the same arguments.

```python

follow steps above to read data and obtain volume_xyz

you can switch pf_dev and fbp interchangeably

H1 = tal.reconstruct.pfdev.solve(data, wlmean=0.06, wlsigma=0.06, volumexyz=volumexyz, camerasystem=tal.enums.CameraSystem.DIRECTLIGHT) H1 = tal.reconstruct.fbp.solve(data, wlmean=0.06, wlsigma=0.06, volumexyz=volumexyz, camerasystem=tal.enums.CameraSystem.DIRECT_LIGHT)

the wlmean and wlsigma parameters set the band pass filter that is the phasor-field-based filter

the camera_system parameter specifies the concrete camera implementation in the phasor-field framework

by default most papers use the DIRECT_LIGHT equivalent so you probably want to leave it as-is

visualize your result

tal.plot.amplitudephase(H1)
```

pf

An implementation of phasor-field cameras [Liu2019]. See also pf_dev.

```python import tal

data = tal.io.read_capture('capture.hdf5') V = np.moveaxis(np.mgrid[-1:1.1:0.1, -1:1.1:0.1, 0.5:2.6:0.1], 0, -1).reshape(-1,3)

Reconstruct the data to the volume V with virtual illumination pulse

with central wavefactor 6 and 4 cycles

reconstruction = tal.reconstruct.pf.solve(data, 6, 4, V, verbose=3, n_threads=1) ```

Logging

The verbosity of the output can be controlled through tal.set_log_level(level). For the available values of level, see tal.LogLevel. For example, if you only wish to see progress bars, warnings and more you can use:

```python import tal

tal.setloglevel(tal.LogLevel.PROGRESS)

...rest of your code...

```

Your choice of logging level is also stored in the configuration file, so it is kept between executions. It also can be changed using tal config in the command line.

tal.resources

The filter_H, bp, fbp and pf_dev functions/modules support multi-threading. The number of threads can be set using:

```python import tal

Option 1: Scoped

with tal.resources('max'): # all CPUs # ...work...

Option 2: Set

tal.set_resources(4) # use 4 CPUs

...work...

```

References

[Velten2012] Velten, A., Willwacher, T., Gupta, O., Veeraraghavan, A., Bawendi, M. G., & Raskar, R. (2012). Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature communications, 3(1), 745.

[Liu2019] Liu, X., Guillén, I., La Manna, M., Nam, J. H., Reza, S. A., Huu Le, T., ... & Velten, A. (2019). Non-line-of-sight imaging using phasor-field virtual wave optics. Nature, 572(7771), 620-623.

License and citation

tal is licensed under the GPL-3.0 license. If you use tal in an academic work, we would appreciate if you cited our work:

@software{Royo_y-tal, author = {Royo, Diego and Luesia-Lahoz, Pablo}, license = {GPL-3.0}, title = {{y-tal}}, url = {https://github.com/diegoroyo/tal}, publisher = {GitHub}, doi = {https://doi.org/10.5281/zenodo.11197745}, }