bubblemask

Python package for applying the Gaussian 'Bubbles' mask to image stimuli, as described by Gosselin and Schyns (2001). This approach applies a mask to an image, with a number of Gaussian 'bubbles' providing windows to the actual pixel values. The method is useful for probing the functional impact of information at different locations in an image (e.g., informativeness of different face regions for emotion recognition). The method can also be applied to examine the size of such functional regions (varying sigma of the Gaussian bubbles), or features like colour (applying the technique to RGB separately) or the spatial frequency of relevant information (applying to specific frequency bandwidths).

This method applies a mask with any number of bubbles, optionally with per-bubble sigma parameters, to a given image. The 2-D bubbles are calculated using the outer product of 1-D Gaussian densities.

Installation

Install via pip:

sh pip install bubblemask

Basic Usage

python from bubblemask import mask import os.path as op from PIL import Image

mask.bubbles_mask() is the main function, which generates and applies a mask with len(sigma) bubbles to a PIL image. By default, these will be positioned randomly. Here, we add 5 bubbles, of various sigmas, to an image of a face on grey background.

```python face = Image.open(op.join('img', 'pre', 'face.png')) face1, mask1, mux, muy, sigma = mask.bubbles_mask(im=face, sigma=[17, 19, 20.84, 25, 30], bg=127)

face.show(); face1.show() ```

The function also outputs the mask as a numpy array.

python import matplotlib.pyplot as plt plt.imshow(mask1) plt.colorbar()

The function also outputs the x and y locations of the centres of the Gaussian bubbles (mu_x and mu_y) and the corresponding sigma values (equal to provided sigma argument).

python print(mu_x)

[151.47868249 30.62953573 67.66242641 248.33505263 189.49367428]

python print(mu_y)

[ 27.5013962 231.37643177 292.48458643 215.76040095 87.04159864]

python print(sigma)

[17, 19, 20.84, 25, 30]

Specifying Bubble Locations

By default, bubbles_mask() will position bubbles randomly in the image. The exact desired locations of bubbles can be specified via the mu_x and mu_y arguments. Here I specify two bubbles to be centred on eyes, with different sigma values, of 20 and 10. Note that mu_x and mu_y can be floats.

```python face2 = mask.bubblesmask( im=face, mux=[85, 186.7], mu_y=[182.5, 182.5], sigma=[20, 10], bg=127 )[0]

face2.show() ```

Using a Convolution-Based Method

Previous implementations I've seen have used a convolution-based approach, where bubble locations are convolved with a Gaussian kernel. This is also available, with the build.build_conv_mask() and mask.bubbles_conv_mask() functions. Key differences are that: * Sigma values must be identical for all bubbles if one kernel is applied globally (could alternatively average over multiple per-sigma convolutions) * Locations of x and y must be integers (rounded if floats) so that bubble precision is limited by resolution of the image

Here is a comparison of the methods:

```python mux = [85, 21, 47, 254, 193] muy = [186, 102, 219, 63, 80] sigma = [20, 20, 20, 20, 20]

method using outer products of Gaussian densities

face3a, mask3a, , _, _ = mask.bubblesmask(im=face, mux=mux, muy=muy, sigma=sigma, bg=127)

method using convolution with Gaussian kernel

face3b, mask3b, , _, _ = mask.bubblesconvmask(im=face, mux=mux, muy=mu_y, sigma=sigma, bg=127)

compare faces

face3a.show(); face3b.show() ```

```python

compare masks

plt.imshow(mask3a); plt.colorbar() plt.imshow(mask3b); plt.colorbar() ```

There are only small differences in the approaches, owing to (I think?) imprecision at the extremeties of bubbles in the convolution-based method:

python plt.imshow(mask3a-mask3b) plt.colorbar()

This means that with reasonable rounding of the masks, the approaches would be functionally equivalent, except that the method using the outer product of densities allows you to give mu as floats (better precision).

The density approach of bubblemask is also generally faster and scales better with increases in image size and sigma values:

Time taken to create a bubbles mask for the convolution and density methods, averaged over 50 iterations per combination of size, sigma, and N bubbles.

Naturalistic Images

Examples above use artificial stimuli on grey backgrounds, but this method can also be applied to more naturalistic, colour stimuli, with the background defined by the bg argument.

```python cat = Image.open(op.join('img', 'pre', 'cat.jpg'))

cat1 = mask.bubblesmask(im=cat, sigma=np.repeat(10, 20), bg=127)[0] # grey background cat2 = mask.bubblesmask(im=cat, sigma=np.repeat(10, 20), bg=[127, 0, 127])[0] # magenta background cat3 = mask.bubbles_mask(im=cat.convert('RGBA'), sigma=np.repeat(10, 20), bg=[0, 0, 0, 0])[0] # transparent background

cat.show(); cat1.show(); cat2.show(); cat3.show() ```

Avoiding Uninformative Locations

It is often more efficient to avoid adding bubbles to regions that you know have no informative information or are irrelevant to your hypothesis, such as the background. bubbles_mask_nonzero() will exclude regions of the background which are sufficiently distant from an informative region.

The centres of each bubble (mu_x, mu_y) will be within max_sigma_from_nonzero multiples of that bubble's sigma value from a non-background (by default, non-zero) pixel in a reference image, ref_im. Background pixels are identified as ref_im <= ref_bg.

The usage is similar to bubbles_mask(), but with additional arguments ref_im (reference image), ref_bg (the cutoff for deciding whether a ref_im pixel is informative), and max_sigma_from_nonzero (how far away from the informative regions can a bubble be).

Imagine we are only interested in the letter a in this image:

python a_cat = Image.open(op.join('img', 'pre', 'a_cat.png')) a_cat.show()

First, we need a reference image, where the target region has values of >0 (e.g., of 1), and the uninformative regions have values of 0. If there is an alpha channel, this should also have a value of 0 for the uninformative regions.

python a_cat_ref = Image.open(op.join('img', 'pre', 'a_cat_ref.png')) a_cat_ref.show()

Now we can apply bubbles to the original image, targeting the letter a. Here, we apply 5 bubbles and specify that the centre of each bubble should be no more than 1 standard deviation away from the non-background pixels of the letter a. Also note that we give ref_bg as [0,0,0,255], because we do not have a transparent alpha in the reference image.

```python acat1 = mask.bubblesmasknonzero( im=acat, refim=acatref, sigma=[10,10,10,10,10], refbg=[0,0,0,255], bg=[0,0,0,255], maxsigmafrom_nonzero=1 )[0]

a_cat1.show() ```

Here is a snippet showing that bubbles_mask_nonzero() only selects bubble locations whose centres are $\le$max_sigma_from_nonzero standard deviations of the non-background pixels. Here we apply 1000 bubbles to the letter a, with bubbles' centres at a maximum distance of 1 standard deviations from the character.

```python acat2, maskacat2, mux, muy, sigma = mask.bubblesmasknonzero( im=acat, refim=acatref, sigma=np.repeat(3, repeats=1000), refbg=[0,0,0,255], bg=[0,0,0,255], maxsigmafrom_nonzero=1 )

a_cat2.show() plt.imshow(maskacat2); plt.colorbar() ```

Finally, you can also define per-bubble constraints for max_sigma_from_nonzero, and values of np.inf and 0 are supported:

```python acat3, maskacat3, mux, muy, sigma = mask.bubblesmasknonzero( im=acat, refim=acatref, sigma = [25, 10, 5], maxsigmafromnonzero = [np.inf, 2.75, 0], ref_bg=[0,0,0,255], bg=[0,0,0,255])

a_cat3.show() ```

Bubble Merging Method

An advantage of this approach is that bubbles of different sizes can be merged. By default, this implementation averages the bubbles and scales the result to within [0, 1]. An alternative may be to take the sum and apply a threshold of the pre-sum maximum across the bubbles. Similarly, the method scales bubbles by default, so that bubbles of different sigma have equal maxima in their densities, where an alternative would be to leave the bubbles unscaled.

Here is a visualisation of the possible options in mask construction, using sum_merge and scale arguments, which can be passed to bubbles_mask():

```python from bubblemask import build

same bubble parameters for all masks

muy = [20, 30, 70] mux = [20, 30, 90] sigma = [5, 10, 7.5] sh = (100, 100)

plot all mask options (the first is the default)

masks = [build.buildmask(muy, mux, sigma, sh, scale=True, summerge=False), build.buildmask(muy, mux, sigma, sh, scale=True, summerge=True), build.buildmask(muy, mux, sigma, sh, scale=False, summerge=False), build.buildmask(muy, mux, sigma, sh, scale=False, summerge=True)]

for i in range(4): plt.imshow(masks[i]) plt.colorbar() ```

Command Line Interface

The bubblemask.mask.bubbles_mask() function can be accessed from the command line. This requires an input argument for a file path to the original image, and an --output argument, to write the result to file.

sh python -m bubblemask --help

``` usage: bubblemask [-h] -i INPUT -o OUTPUT -s SIGMA [SIGMA ...] [-x MUX [MUX ...]] [-y MUY [MUY ...]] [-b BACKGROUND [BACKGROUND ...]] [--unscaled] [--summerge] [--seed SEED]

optional arguments: -h, --help show this help message and exit -i INPUT, --input INPUT the file path for the input image -o OUTPUT, --output OUTPUT the path of the desired output file -s SIGMA [SIGMA ...], --sigma SIGMA [SIGMA ...] a list of sigmas for the bubbles, in space-separated format (e.g., "10 10 15") -x MUX [MUX ...], --mux MUX [MUX ...] x indices (axis 1 in numpy) for bubble locations, in space- separated format - leave blank (default) for random location -y MUY [MUY ...], --muy MUY [MUY ...] y indices (axis 0 in numpy) for bubble locations, in space- separated format - leave blank (default) for random location -b BACKGROUND [BACKGROUND ...], --background BACKGROUND [BACKGROUND ...] the desired background for the image, as a single integer from 0 to 255 (default=0), or space-separated values for each channel in the image --unscaled do not scale the densities of the bubbles to have the same maxima --summerge sum_merge -- should merges, where bubbles overlap, be completed using a simple sum of the bubbles, thresholded to the maxima of the pre-merged bubbles? If not (the default), densities are instead averaged (mean). --seed SEED random seed to use ```

Example usage:

sh python -m bubblemask -i img/pre/face.png -o img/post/cli_masked_face.png -s 30 30 20 -b 127 --seed 42