Thermal Face

Thermal Face is a machine learning model for fast face detection in thermal images. It was built for Fever, the contactless fever thermometer with auto-aim.

Inference

The face detection model is using TensorFlow Lite for optimal performance on mobile/edge devices. The recommended inference setup is a Raspberry Pi 4 Model B with a Coral USB Accelerator.

The following is an example for inference from Python on an image file using the compiled model thermal_face_automl_edge_fast_edgetpu.tflite, downloaded from the latest release, and the Edge TPU API:

bash pip3 install Pillow sudo apt-get install python3-edgetpu

```python from edgetpu.detection.engine import DetectionEngine from PIL import Image

One-time initialization:

facedetector = DetectionEngine('thermalfaceautomledgefastedgetpu.tflite')

Per-image detection:

image = Image.open('image.png').convert('RGB') faces = facedetector.detectwithimage(image, threshold=0.5 topk=10, keepaspectratio=True, relativecoord=False, resample=Image.BILINEAR) for face in faces: # np.array([[left, top], [right, bottom]], dtype=float64) print(face.boundingbox) ```

Alternatively, you can use the TF Lite API directly on the compiled model or, in the absence of an Edge TPU, on the uncompiled model thermal_face_automl_edge_fast.tflite, which is also in the latest release.

Training

The model is trained with Cloud AutoML using a face dataset that combines a large set of images in visible light from the WIDER FACE database and a smaller set of thermal images from the Tufts Face Database and the FLIR ADAS Dataset.

1. Create the dataset

There are a total of 18,418 images and 164,915 face bounding box annotations in the combined dataset. The WIDER FACE set is large and diverse, but only contains visible-light images. The thermal images from the Tufts Face Database and FLIR ADAS Dataset are fewer and less diverse, so we mix the three sets before splitting them into training, validation, and test sets. The relative size of the test and validation sets are unusually small to achieve a better balance among the source datasets while still using a significant fraction of all available training data. The exact breakdown of images (and annotations) is as follows:

| | Training set | Validation set | Test set | | -: | -: | -: | -: | | Tufts Face Database (IR) | 1,247 (1,247) | 155 (155) | 155 (155) | | Fraction of source images | 80% | 10% | 10% | | Fraction of combined | 7% (1%) | 39% (8%) | 39% (8%) | | FLIR ADAS (Faces) | 617 (854) | 77 (108) | 77 (99) | | Fraction of source images | 80% | 10% | 10% | | Fraction of combined | 3% (1%) | 20% (6%) | 20% (5%) | | WIDER FACE (Validation) | 2,898 (28,859) | 161 (1,611) | 161 (1,741) | | Fraction of source images | 90% | 5% | 5% | | Fraction of combined | 16% (18%) | 41% (86%) | 41% (87%) | | WIDER FACE (Training) | 12,870 (130,086) | 0 (0) | 0 (0) | | Fraction of source images | 100% | 0% | 0% | | Fraction of combined | 73% (81%) | 0% (0%) | 0% (0%) | | Combined | 17,632 (161,046) | 393 (1,874) | 393 (1,995) | | Fraction of combined sources | 96% (98%) | 2% (1%) | 2% (1%) |

1.1 Get the Tufts Face Database

Download the thermal images from the Tufts Face Database and upload them to Cloud Storage:

```bash LOCATION="us-central1" TDFACEDIR="tufts-face-database" TDFACEBUCKET="gs://$TDFACE_DIR"

cd training

for i in $(seq 1 4) do curl -O http://tdface.ece.tufts.edu/downloads/TDIRA/TDIRASet$i.zip curl -O http://tdface.ece.tufts.edu/downloads/TDIRE/TDIRESet$i.zip done

mkdir $TDFACEDIR for f in TDIR*.zip do unzip $f -d $TDFACEDIR/$(basename $f .zip) done

gsutil mb -l $LOCATION $TDFACEBUCKET gsutil -m rsync -r $TDFACEDIR $TDFACE_BUCKET ```

Create a dataset spec of the images in the AutoML format using the separately created bounding box annotations and upload it:

```bash TDFACEANNOTATIONS="$TDFACEDIR/bounding-boxes.csv" TDFACEAUTOML="tdface-automl.csv" TDFACETRAININGFRACTION=0.8 TDFACEVALIDATIONFRACTION=0.1 TDFACETEST_FRACTION=0.1

curl https://github.com/maxbbraun/tdface-annotations/releases/latest/download/bounding-boxes.csv -o $TDFACE_ANNOTATIONS

python3 -m venv venv . venv/bin/activate pip3 install -r requirements.txt

python automlconvert.py \ --mode=TDFACE \ --tdfacedir=$TDFACEDIR \ --tdfacebucket=$TDFACEBUCKET \ --tdfaceannotations=$TDFACEANNOTATIONS \ --trainingfraction=$TDFACETRAININGFRACTION \ --validationfraction=$TDFACEVALIDATIONFRACTION \ --testfraction=$TDFACETESTFRACTION \ --automlout=$TDFACEAUTOML

gsutil cp $TDFACEAUTOML $TDFACEBUCKET ```

1.2 Get FLIR ADAS

Download the FLIR_ADAS_1_3.tar.* files and separately created bounding box annotations and process them:

```bash FLIRADASREPO="flir-adas-faces" FLIRADASDIR="flir-adas-database" TMPFLIRADASDIR="/tmp/$FLIRADASDIR" FLIRADASBUCKET="gs://$FLIRADASDIR" FLIRADASANNOTATIONS="$FLIRADASREPO/bounding-boxes.csv" FLIRADASAUTOML="flir-adas-automl.csv" FLIRADASTRAININGFRACTION=0.8 FLIRADASVALIDATIONFRACTION=0.1 FLIRADASTESTFRACTION=0.1

git clone https://github.com/maxbbraun/flir-adas-faces.git $FLIRADASREPO cd $FLIRADASREPO

curl https://github.com/maxbbraun/flir-adas-faces/releases/latest/download/bounding-boxes.csv -o $FLIRADASANNOTATIONS

mkdir $FLIRADASDIR tar -C $FLIRADASDIR -xvf FLIRADAS1_3.tar.001 --strip-components=1

python3 -m venv venv . venv/bin/activate pip install -r requirements.txt

python flirconvert.py \ --inputdir=$FLIRADASDIR/train \ --outputdir=$TMPFLIRADASDIR/train \ --outputcsv=/dev/null python flirconvert.py \ --inputdir=$FLIRADASDIR/val \ --outputdir=$TMPFLIRADASDIR/val \ --outputcsv=/dev/null python flirconvert.py \ --inputdir=$FLIRADASDIR/video \ --outputdir=$TMPFLIRADASDIR/video \ --output_csv=/dev/null

cd .. . venv/bin/activate

python automlconvert.py \ --mode=FLIR \ --tdfacedir=$TMPFLIRADASDIR \ --tdfacebucket=$FLIRADASBUCKET \ --tdfaceannotations=$FLIRADASANNOTATIONS \ --trainingfraction=$FLIRADASTRAININGFRACTION \ --validationfraction=$FLIRADASVALIDATIONFRACTION \ --testfraction=$FLIRADASTESTFRACTION \ --automlout=$FLIRADASAUTOML

gsutil cp $FLIRADASAUTOML $FLIRADASBUCKET ```

1.3 Get WIDER FACE

Download and upload the WIDER FACE dataset: - WIDER_train.zip - WIDER_val.zip - WIDER_test.zip - widerfacesplit.zip

```bash WIDERFACEDIR="wider-face-database" WIDERFACEBUCKET="gs://$WIDERFACE_DIR"

mkdir $WIDERFACEDIR for f in WIDER.zip wider_.zip do unzip $f -d $WIDERFACE_DIR/ done

gsutil mb -l $LOCATION $WIDERFACEBUCKET gsutil -m rsync -r $WIDERFACEDIR $WIDERFACE_BUCKET ```

Create and upload the AutoML spec using the included bounding boxes:

```bash WIDERFACETRAININGAUTOML="widerface-training-automl.csv" WIDERFACEVALIDATIONAUTOML="widerface-validation-automl.csv" WIDERFACETRAININGTRAININGFRACTION=1.0 WIDERFACETRAININGVALIDATIONFRACTION=0.0 WIDERFACETRAININGTESTFRACTION=0.0 WIDERFACEVALIDATIONTRAININGFRACTION=0.9 WIDERFACEVALIDATIONVALIDATIONFRACTION=0.05 WIDERFACEVALIDATIONTESTFRACTION=0.05

python automlconvert.py \ --mode=WIDERFACE \ --widerfacedir=$WIDERFACEDIR/WIDERtrain \ --widerfacebucket=$WIDERFACEBUCKET/WIDERtrain \ --widerfaceannotations=$WIDERFACEDIR/widerfacesplit/widerfacetrainbbxgt.txt \ --trainingfraction=$WIDERFACETRAININGTRAININGFRACTION \ --validationfraction=$WIDERFACETRAININGVALIDATIONFRACTION \ --testfraction=$WIDERFACETRAININGTESTFRACTION \ --automlout=$WIDERFACETRAININGAUTOML python automlconvert.py \ --mode=WIDERFACE \ --widerfacedir=$WIDERFACEDIR/WIDERval \ --widerfacebucket=$WIDERFACEBUCKET/WIDERval \ --widerfaceannotations=$WIDERFACEDIR/widerfacesplit/widerfacevalbbxgt.txt \ --trainingfraction=$WIDERFACEVALIDATIONTRAININGFRACTION \ --validationfraction=$WIDERFACEVALIDATIONVALIDATIONFRACTION \ --testfraction=$WIDERFACEVALIDATIONTESTFRACTION \ --automlout=$WIDERFACEVALIDATIONAUTOML

gsutil cp $WIDERFACETRAININGAUTOML $WIDERFACEBUCKET gsutil cp $WIDERFACEVALIDATIONAUTOML $WIDERFACEBUCKET ```

1.4 Combine the datasets

Combine all AutoML dataset specs into one and upload it:

```bash THERMALFACEAUTOML="automl.csv" MODELBUCKET="gs://thermal-face" MODELNAME="thermalfaceautomledgefast"

gsutil mb -l $LOCATION $MODEL_BUCKET

rm -f $THERMALFACEAUTOML cat $TDFACEAUTOML >> $THERMALFACEAUTOML cat $WIDERFACETRAININGAUTOML >> $THERMALFACEAUTOML cat $WIDERFACEVALIDATIONAUTOML >> $THERMALFACEAUTOML cat $FLIRADASAUTOML >> $THERMALFACE_AUTOML

gsutil cp $THERMALFACEAUTOML $MODEL_BUCKET ```

2. Train the model

Use Cloud AutoML Vision with the following options:

• New dataset name: tdface_x_widerface_x_fliradas
• Model objective: Object detection
• CSV file on Cloud Storage: $MODEL_BUCKET/$THERMAL_FACE_AUTOML
• Model name: $MODEL_NAME
• Model type: Edge
• Optimize for: Faster predictions
• Node budget: 24 node hours
• Use model: TF Lite
• Export to Cloud Storage: $MODEL_BUCKET/

3. Compile the model

Use Docker to compile the model for Edge TPU:

```bash MODELFILE="$MODELNAME.tflite" TPUMODELFILE="${MODELFILE%.*}edgetpu.${MODELFILE##*.}" MODELDIR="$(pwd)/../models" COMPILERNAME="compiler" OUTDIR="/out"

gsutil cp $MODELBUCKET/*/$MODELNAME*/model.tflite $MODEL_FILE

docker build . \ --no-cache \ --file=$COMPILERNAME.Dockerfile \ --tag=$COMPILERNAME \ --build-arg=MODELFILE=$MODELFILE \ --build-arg=OUTDIR=$OUTDIR docker run \ --mount=type=bind,source=$MODELDIR,target=$OUTDIR \ --rm \ $COMPILERNAME mv $MODELFILE $MODEL_DIR/ ```