In today’s tutorial, you will learn how to configure your NVIDIA Jetson Nano for Computer Vision and Deep Learning with TensorFlow, Keras, TensorRT, and OpenCV.

Two weeks ago, we discussed how to use my pre-configured Nano .img file — today, you will learn how to configure your own Nano from scratch.

This guide requires you to have at least 48 hours of time to kill as you configure your NVIDIA Jetson Nano on your own (yes, it really is that challenging)

If you decide you want to skip the hassle and use my pre-configured Nano .img, you can find it as part of my brand-new book, Raspberry Pi for Computer Vision.

But for those brave enough to go through the gauntlet, this post is for you!

To learn how to configure your NVIDIA Jetson Nano for computer vision and deep learning, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

How to configure your NVIDIA Jetson Nano for Computer Vision and Deep Learning

The NVIDIA Jetson Nano packs 472GFLOPS of computational horsepower. While it is a very capable machine, configuring it is not (complex machines are typically not easy to configure).

In this tutorial, we’ll work through 16 steps to configure your Jetson Nano for computer vision and deep learning.

Prepare yourself for a long, grueling process — you may need 2-5 days of your time to configure your Nano following this guide.

Once we are done, we will test our system to ensure it is configured properly and that TensorFlow/Keras and OpenCV are operating as intended. We will also test our Nano’s camera with OpenCV to ensure that we can access our video stream.

If you encounter a problem with the final testing step, then you may need to go back and resolve it; or worse, start back at the very first step and endure another 2-5 days of pain and suffering through the configuration tutorial to get up and running (but don’t worry, I present an alternative at the end of the 16 steps).

Step #1: Flash NVIDIA’s Jetson Nano Developer Kit .img to a microSD for Jetson Nano

In this step, we will download NVIDIA’s Jetpack 4.2 Ubuntu-based OS image and flash it to a microSD. You will need the microSD flashed and ready to go to follow along with the next steps.

Go ahead and start your download here, ensuring that you download the “Jetson Nano Developer Kit SD Card image” as shown in the following screenshot:

We recommend the Jetpack 4.2 for compatibility with the Complete Bundle of Raspberry Pi for Computer Vision (our recommendation will inevitably change in the future).

While your Nano SD image is downloading, go ahead and download and install balenaEtcher, a disk image flashing tool:

Once both (1) your Nano Jetpack image is downloaded, and (2) balenaEtcher is installed, you are ready to flash the image to a microSD.

You will need a suitable microSD card and microSD reader hardware. We recommend either a 32GB or 64GB microSD card (SanDisk’s 98MB/s cards are high quality, and Amazon carries them if they are a distributor in your locale). Any microSD card reader should work.

Insert the microSD into the card reader, and then plug the card reader into a USB port on your computer. From there, fire up balenaEtcher and proceed to flash.

When flashing has successfully completed, you are ready to move on to Step #2.

Step #2: Boot your Jetson Nano with the microSD and connect to a network

In this step, we will power up our Jetson Nano and establish network connectivity.

This step requires the following:

1. The flashed microSD from Step #1
2. An NVIDIA Jetson Nano dev board
3. HDMI screen
4. USB keyboard + mouse
5. A power supply — either (1) a 5V 2.5A (12.5W) microSD power supply or (2) a 5V 4A (20W) barrel plug power supply with a jumper at the J48 connector
6. Network connection — either (1) an Ethernet cable connecting your Nano to your network or (2) a wireless module. The wireless module can come in the form of a USB WiFi adapter or a WiFi module installed under the Jetson Nano heatsink

If you want WiFi (most people do), you must add a WiFi module on your own. Two great options for adding WiFi to your Jetson Nano include:

• USB to WiFi adapter (Figure 4, top-right). No tools are required and it is portable to other devices. Pictured is the Geekworm Dual Band USB 1200m
• WiFi module such as the Intel Dual Band Wireless-Ac 8265 W/Bt (Intel 8265NGW) and 2x Molex Flex 2042811100 Flex Antennas (Figure 5, bottom-center). You must install the WiFi module and antennas under the main heatsink on your Jetson Nano. This upgrade requires a Phillips #2 screwdriver, the wireless module, and antennas (not to mention about 10-20 minutes of your time)

We recommend going with a USB WiFi adapter if you need to use WiFi with your Jetson Nano. There are many options available online, so try to purchase one that has Ubuntu 18.04 drivers preinstalled on the OS so that you don’t need to scramble to download and install drivers as we did following these instructions for the Geekworm product (it could be tough if you don’t have a wired connection available in the first place to download and install the drivers).

Once you have gathered all the gear, insert your microSD into your Jetson Nano as shown in Figure 5:

From there, connect your screen, keyboard, mouse, and network interface.

Finally, apply power. Insert the power plug of your power adapter into your Jetson Nano (use the J48 jumper if you are using a 20W barrel plug supply).

Once you see your NVIDIA + Ubuntu 18.04 desktop, you should configure your wired or wireless network settings as needed using the icon in the menubar as shown in Figure 6.

When you have confirmed that you have internet access on your NVIDIA Jetson Nano, you can move on to the next step.

Step #3: Open a terminal or start an SSH session

In this step we will do one of the following:

1. Option 1: Open a terminal on the Nano desktop, and assume that you’ll perform all steps from here forward using the keyboard and mouse connected to your Nano
2. Option 2: Initiate an SSH connection from a different computer so that we can remotely configure our NVIDIA Jetson Nano for computer vision and deep learning

Both options are equally good.

Option 1: Use the terminal on your Nano desktop

For Option 1, open up the application launcher, and select the terminal app. You may wish to right click it in the left menu and lock it to the launcher, since you will likely use it often.

You may now continue to Step #4 while keeping the terminal open to enter commands.

Option 2: Initiate an SSH remote session

For Option 2, you must first determine the username and IP address of your Jetson Nano. On your Nano, fire up a terminal from the application launcher, and enter the following commands at the prompt:

$ whoami
nvidia
$ ifconfig
en0: flags=8863 mtu 1500
	options=400
	ether 8c:85:90:4f:b4:41
	inet6 fe80::14d6:a9f6:15f8:401%en0 prefixlen 64 secured scopeid 0x8
	inet6 2600:100f:b0de:1c32:4f6:6dc0:6b95:12 prefixlen 64 autoconf secured
	inet6 2600:100f:b0de:1c32:a7:4e69:5322:7173 prefixlen 64 autoconf temporary
	inet 192.168.1.4 netmask 0xffffff00 broadcast 192.168.1.255
	nd6 options=201
	media: autoselect
	status: active

Grab your IP address (it is on the highlighted line). My IP address is 192.168.1.4; however, your IP address will be different, so make sure you check and verify your IP address!

Then, on a separate computer, such as your laptop/desktop, initiate an SSH connection as follows:

$ ssh nvidia@192.168.1.4

Notice how I’ve entered the username and IP address of the Jetson Nano in my command to remotely connect. You should now have a successful connection to your Jetson Nano, and you can continue on with Step #4.

Step #4: Update your system and remove programs to save space

In this step, we will remove programs we don’t need and update our system.

First, let’s set our Nano to use maximum power capacity:

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

The nvpmodel command handles two power options for your Jetson Nano: (1) 5W is mode 1 and (2) 10W is mode 0. The default is the higher wattage mode, but it is always best to force the mode before running the jetson_clocks command.

According to the NVIDIA devtalk forums:

The jetson_clocks script disables the DVFS governor and locks the clocks to their maximums as defined by the active nvpmodel power mode. So if your active mode is 10W, jetson_clocks will lock the clocks to their maximums for 10W mode. And if your active mode is 5W, jetson_clocks will lock the clocks to their maximums for 5W mode (NVIDIA DevTalk Forums).

Note: There are two typical ways to power your Jetson Nano. A 5V 2.5A (10W) microUSB power adapter is a good option. If you have a lot of gear being powered by the Nano (keyboards, mice, WiFi, cameras), then you should consider a 5V 4A (20W) power supply to ensure that your processors can run at their full speeds while powering your peripherals. Technically there’s a third power option too if you want to apply power directly on the header pins.

After you have set your Nano for maximum power, go ahead and remove LibreOffice — it consumes lots of space, and we won’t need it for computer vision and deep learning:

$ sudo apt-get purge libreoffice*
$ sudo apt-get clean

From there, let’s go ahead and update system level packages:

$ sudo apt-get update && sudo apt-get upgrade

In the next step, we’ll begin installing software.

Step #5: Install system-level dependencies

The first set of software we need to install includes a selection of development tools:

$ sudo apt-get install git cmake
$ sudo apt-get install libatlas-base-dev gfortran
$ sudo apt-get install libhdf5-serial-dev hdf5-tools
$ sudo apt-get install python3-dev
$ sudo apt-get install nano locate

Next, we’ll install SciPy prerequisites (gathered from NVIDIA’s devtalk forums) and a system-level Cython library:

$ sudo apt-get install libfreetype6-dev python3-setuptools
$ sudo apt-get install protobuf-compiler libprotobuf-dev openssl
$ sudo apt-get install libssl-dev libcurl4-openssl-dev
$ sudo apt-get install cython3

We also need a few XML tools for working with TensorFlow Object Detection (TFOD) API projects:

$ sudo apt-get install libxml2-dev libxslt1-dev

Step #6: Update CMake

Now we’ll update the CMake precompiler tool as we need a newer version in order to successfully compile OpenCV.

First, download and extract the CMake update:

$ wget http://www.cmake.org/files/v3.13/cmake-3.13.0.tar.gz
$ tar xpvf cmake-3.13.0.tar.gz cmake-3.13.0/

Next, compile CMake:

$ cd cmake-3.13.0/
$ ./bootstrap --system-curl
$ make -j8

And finally, update your bash profile:

$ echo 'export PATH=/home/nvidia/cmake-3.13.0/bin/:$PATH' >> ~/.bashrc
$ source ~/.bashrc

CMake is now ready to go on your system. Ensure that you do not delete the cmake-3.13.0/ directory in your home folder.

Step #7: Install OpenCV system-level dependencies and other development dependencies

Let’s now install OpenCV dependecies on our system beginning with tools needed to build and compile OpenCV with parallelism:

$ sudo apt-get install build-essential pkg-config
$ sudo apt-get install libtbb2 libtbb-dev

Next, we’ll install a handful of codecs and image libraries:

$ sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev
$ sudo apt-get install libxvidcore-dev libavresample-dev
$ sudo apt-get install libtiff-dev libjpeg-dev libpng-dev

And then we’ll install a selection of GUI libraries:

$ sudo apt-get install python-tk libgtk-3-dev
$ sudo apt-get install libcanberra-gtk-module libcanberra-gtk3-module

Lastly, we’ll install Video4Linux (V4L) so that we can work with USB webcams and install a library for FireWire cameras:

$ sudo apt-get install libv4l-dev libdc1394-22-dev

Step #8: Set up Python virtual environments on your Jetson Nano

I can’t stress this enough: Python virtual environments are a best practice when both developing and deploying Python software projects.

Virtual environments allow for isolated installs of different Python packages. When you use them, you could have one version of a Python library in one environment and another version in a separate, sequestered environment.

In the remainder of this tutorial, we’ll create one such virtual environment; however, you can create multiple environments for your needs after you complete this Step #8. Be sure to read the RealPython guide on virtual environments if you aren’t familiar with them.

First, we’ll install the de facto Python package management tool, pip:

$ wget https://bootstrap.pypa.io/get-pip.py
$ sudo python3 get-pip.py
$ rm get-pip.py

And then we’ll install my favorite tools for managing virtual environments, virtualenv and virtualenvwrapper:

$ sudo pip install virtualenv virtualenvwrapper

The virtualenvwrapper tool is not fully installed until you add information to your bash profile. Go ahead and open up your ~/.bashrc with the nano ediitor:

$ nano ~/.bashrc

And then insert the following at the bottom of the file:

# virtualenv and virtualenvwrapper
export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3
source /usr/local/bin/virtualenvwrapper.sh

Save and exit the file using the keyboard shortcuts shown at the bottom of the nano editor, and then load the bash profile to finish the virtualenvwrapper installation:

$ source ~/.bashrc

So long as you don’t encounter any error messages, both virtualenv and virtualenvwrapper are now ready for you to create and destroy virtual environments as needed in Step #9.

Step #9: Create your ‘py3cv4’ virtual environment

This step is dead simple once you’ve installed virtualenv and virtualenvwrapper in the previous step. The virtualenvwrapper tool provides the following commands to work with virtual environments:

• mkvirtualenv: Create a Python virtual environment
• lsvirtualenv: List virtual environments installed on your system
• rmvirtualenv: Remove a virtual environment
• workon: Activate a Python virtual environment
• deactivate: Exits the virtual environment taking you back to your system environment

Assuming Step #8 went smoothly, let’s create a Python virtual environment on our Nano:

$ mkvirtualenv py3cv4 -p python3

I’ve named the virtual environment py3cv4 indicating that we will use Python 3 and OpenCV 4. You can name yours whatever you’d like depending on your project and software needs or even your own creativity.

When your environment is ready, your bash prompt will be preceded by (py3cv4). If your prompt is not preceded by the name of your virtual environment name, at any time you can use the workon command as follows:

$ workon py3cv4

For the remaining steps in this tutorial, you must be “in” the py3cv4 virtual environment.

Step #10: Install the Protobuf Compiler

This section walks you through the step-by-step process for configuring protobuf so that TensorFlow will be fast.

TensorFlow’s performance can be significantly impacted (in a negative way) if an efficient implementation of protobuf and libprotobuf are not present.

When we pip-install TensorFlow, it automatically installs a version of protobuf that might not be the ideal one. The issue with slow TensorFlow performance has been detailed in this NVIDIA Developer forum.

First, download and install an efficient implementation of the protobuf compiler (source):

$ wget https://raw.githubusercontent.com/jkjung-avt/jetson_nano/master/install_protobuf-3.6.1.sh
$ sudo chmod +x install_protobuf-3.6.1.sh
$ ./install_protobuf-3.6.1.sh

This will take approximately one hour to install, so go for a nice walk, or read a good book such as Raspberry Pi for Computer Vision or Deep Learning for Computer Vision with Python.

Once protobuf is installed on your system, you need to install it inside your virtual environment:

$ workon py3cv4 # if you aren't inside the environment
$ cd ~
$ cp -r ~/src/protobuf-3.6.1/python/ .
$ cd python
$ python setup.py install --cpp_implementation

Notice that rather than using pip to install the protobuf package, we used a setup.py installation script. The benefit of using setup.py is that we compile software specifically for the Nano processor rather than using generic precompiled binaries.

In the remaining steps we will use a mix of setup.py (when we need to optimize a compile) and pip (when the generic compile is sufficient).

Let’s move on to Step #11 where we’ll install deep learning software.

Step #11: Install TensorFlow, Keras, NumPy, and SciPy on Jetson Nano

In this section, we’ll install TensorFlow/Keras and their dependencies.

First, ensure you’re in the virtual environment:

$ workon py3cv4

And then install NumPy and Cython:

$ pip install numpy cython

You may encounter the following error message:

ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly.

If you come across that message, then follow these additional steps. First, install NumPy with super user privileges:

$ sudo pip install numpy

Then, create a symbolic link from your system’s NumPy into your virtual environment site-packages. To be able to do that you would need the installation path of numpy, which can be found out by issuing a NumPy uninstall command, and then canceling it as follows:

$ sudo pip uninstall numpy
Uninstalling numpy-1.18.1:
  Would remove:
    /usr/bin/f2py
    /usr/local/bin/f2py
    /usr/local/bin/f2py3
    /usr/local/bin/f2py3.6
    /usr/local/lib/python3.6/dist-packages/numpy-1.18.1.dist-info/*
    /usr/local/lib/python3.6/dist-packages/numpy/*
Proceed (y/n)? n

Note that you should type n at the prompt because we do not want to proceed with uninstalling NumPy. Then, note down the installation path (highlighted), and execute the following commands (replacing the paths as needed):

$ cd ~/.virtualenvs/py3cv4/lib/python3.6/site-packages/
$ ln -s ~/usr/local/lib/python3.6/dist-packages/numpy numpy
$ cd ~

At this point, NumPy is sym-linked into your virtual environment. We should quickly test it as NumPy is needed for the remainder of this tutorial. Issue the following commands in a terminal:

$ workon py3cv4
$ python
>>> import numpy

Now that NumPy is installed, let’s install SciPy. We need SciPy v1.3.3, so we cannot use pip. Instead, we’re going to grab a release directly from GitHub and install it:

$ wget https://github.com/scipy/scipy/releases/download/v1.3.3/scipy-1.3.3.tar.gz
$ tar -xzvf scipy-1.3.3.tar.gz scipy-1.3.3
$ cd scipy-1.3.3/
$ python setup.py install

Installing SciPy will take approximately 35 minutes. Watching and waiting for it to install is like watching paint dry, so you might as well pop open one of my books or courses and brush up on your computer vision and deep learning skills.

Now we will install NVIDIA’s TensorFlow 1.13 optimized for the Jetson Nano. Of course you’re wondering:

Why shouldn’t I use TensorFlow 2.0 on the NVIDIA Jetson Nano?

That’s a great question, and I’m going to bring in my NVIDIA Jetson Nano expert, Sayak Paul, to answer that very question:

Although TensorFlow 2.0 is available for installation on the Nano it is not recommended because there can be incompatibilities with the version of TensorRT that comes with the Jetson Nano base OS. Furthermore, the TensorFlow 2.0 wheel for the Nano has a number of memory leak issues which can make the Nano freeze and hang. For these reasons, we recommend TensorFlow 1.13 at this point in time.

Given Sayak’s expert explanation, let’s go ahead and install TF 1.13 now:

$ pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v42 tensorflow-gpu==1.13.1+nv19.3

Let’s now move on to Keras, which we can simply install via pip:

$ pip install keras

Next, we’ll install the TFOD API on the Jetson Nano.

Step #12: Install the TensorFlow Object Detection API on Jetson Nano

In this step, we’ll install the TFOD API on our Jetson Nano.

TensorFlow’s Object Detection API (TFOD API) is a library that we typically know for developing object detection models. We also need it to optimize models for the Nano’s GPU.

TensorFlow’s tf_trt_models is a wrapper around the TFOD API, which allows for building frozen graphs, a necessary for model deployment. More information on tf_trt_models can be found in this NVIDIA repository.

Again, ensure that all actions take place “in” your py3cv4 virtual environment:

$ cd ~
$ workon py3cv4

First, clone the models repository from TensorFlow:

$ git clone https://github.com/tensorflow/models

In order to be reproducible, you should checkout the following commit that supports TensorFlow 1.13.1:

$ cd models && git checkout -q b00783d

From there, install the COCO API for working with the COCO dataset and, in particular, object detection:

$ cd ~
$ git clone https://github.com/cocodataset/cocoapi.git
$ cd cocoapi/PythonAPI
$ python setup.py install

The next step is to compile the Protobuf libraries used by the TFOD API. The Protobuf libraries enable us (and therefore the TFOD API) to serialize structured data in a language-agnostic way:

$ cd ~/models/research/
$ protoc object_detection/protos/*.proto --python_out=.

From there, let’s configure a useful script I call setup.sh. This script will be needed each time you use the TFOD API for deployment on your Nano. Create such a file with the Nano editor:

$ nano ~/setup.sh

Insert the following lines in the new file:

#!/bin/sh

export PYTHONPATH=$PYTHONPATH:/home/`whoami`/models/research:\
/home/`whoami`/models/research/slim

The shebang at the top indicates that this file is executable and then the script configures your PYTHONPATH according to the TFOD API installation directory. Save and exit the file using the keyboard shortcuts shown at the bottom of the nano editor.

Step #13: Install NVIDIA’s ‘tf_trt_models’ for Jetson Nano

In this step, we’ll install the tf_trt_models library from GitHub. This package contains TensorRT-optimized models for the Jetson Nano.

First, ensure you’re working in the py3cv4 virtual environment:

$ workon py3cv4

Go ahead and clone the GitHub repo, and execute the installation script:

$ cd ~
$ git clone --recursive https://github.com/NVIDIA-Jetson/tf_trt_models.git
$ cd tf_trt_models
$ ./install.sh

That’s all there is to it. In the next step, we’ll install OpenCV!

Step #14: Install OpenCV 4.1.2 on Jetson Nano

In this section, we will install the OpenCV library with CUDA support on our Jetson Nano.

OpenCV is the common library we use for image processing, deep learning via the DNN module, and basic display tasks. I’ve created an OpenCV Tutorial for you if you’re interested in learning some of the basics.

CUDA is NVIDIA’s set of libraries for working with their GPUs. Some non-deep learning tasks can actually run on a CUDA-capable GPU faster than on a CPU. Therefore, we’ll install OpenCV with CUDA support, since the NVIDIA Jetson Nano has a small CUDA-capable GPU.

This section of the tutorial is based on the hard work of the owners of the PythOps website.

We will be compiling from source, so first let’s download the OpenCV source code from GitHub:

$ cd ~
$ wget -O opencv.zip https://github.com/opencv/opencv/archive/4.1.2.zip
$ wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.1.2.zip

Notice that the versions of OpenCV and OpenCV-contrib match. The versions must match for compatibility.

From there, extract the files and rename the directories for convenience:

$ unzip opencv.zip
$ unzip opencv_contrib.zip
$ mv opencv-4.1.2 opencv
$ mv opencv_contrib-4.1.2 opencv_contrib

Go ahead and activate your Python virtual environment if it isn’t already active:

$ workon py3cv4

And change into the OpenCV directory, followed by creating and entering a build directory:

$ cd opencv
$ mkdir build
$ cd build

It is very important that you enter the next CMake command while you are inside (1) the ~/opencv/build directory and (2) the py3cv4 virtual environment. Take a second now to verify:

(py3cv4) $ pwd
/home/nvidia/opencv/build

I typically don’t show the name of the virtual environment in the bash prompt because it takes up space, but notice how I have shown it at the beginning of the prompt above to indicate that we are “in” the virtual environment.

Additionally, the result of the pwd command indicates we are “in” the build/ directory.

Provided you’ve met both requirements, you’re now ready to use the CMake compile prep tool:

$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
	-D WITH_CUDA=ON \
	-D CUDA_ARCH_PTX="" \
	-D CUDA_ARCH_BIN="5.3,6.2,7.2" \
	-D WITH_CUBLAS=ON \
	-D WITH_LIBV4L=ON \
	-D BUILD_opencv_python3=ON \
	-D BUILD_opencv_python2=OFF \
	-D BUILD_opencv_java=OFF \
	-D WITH_GSTREAMER=ON \
	-D WITH_GTK=ON \
	-D BUILD_TESTS=OFF \
	-D BUILD_PERF_TESTS=OFF \
	-D BUILD_EXAMPLES=OFF \
	-D OPENCV_ENABLE_NONFREE=ON \
	-D OPENCV_EXTRA_MODULES_PATH=/home/`whoami`/opencv_contrib/modules ..

There are a lot of compiler flags here, so let’s review them. Notice that WITH_CUDA=ON is set, indicating that we will be compiling with CUDA optimizations.

Secondly, notice that we have provided the path to our opencv_contrib folder in the OPENCV_EXTRA_MODULES_PATH, and we have set OPENCV_ENABLE_NONFREE=ON, indicating that we are installing the OpenCV library with full support for external and patented algorithms.

Be sure to copy the entire command above, including the .. at the very bottom. When CMake finishes, you’ll encounter the following output in your terminal:

I highly recommend you scroll up and read the terminal output with a keen eye to see if there are any errors. Errors need to be resolved before moving on. If you do encounter an error, it is likely that one or more prerequisites from Steps #5-#11 are not installed properly. Try to determine the issue, and fix it.

If you do fix an issue, then you’ll need to delete and re-creating your build directory before running CMake again:

$ cd ..
$ rm -rf build
$ mkdir build
$ cd build
# run CMake command again

When you’re satisfied with your CMake output, it is time to kick of the compilation process with Make:

$ make -j4

Compiling OpenCV will take approximately 2.5 hours. When it is done, you’ll see 100%, and your bash prompt will return:

From there, we need to finish the installation. First, run the install command:

$ sudo make install

Then, we need to create a symbolic link from OpenCV’s installation directory to the virtual environment. A symbolic link is like a pointer in that a special operating system file points from one place to another on your computer (in this case our Nano). Let’s create the sym-link now:

$ cd ~/.virtualenvs/py3cv4/lib/python3.6/site-packages/
$ ln -s /home/`whoami`/opencv-4.1.2/build/lib/python3/cv2.cpython-36m-aarch64-linux-gnu.so cv2.so

OpenCV is officially installed. In the next section, we’ll install a handful of useful libraries to accompany everything we’ve installed so far.

Step #15: Install other useful libraries via pip

In this section, we’ll use pip to install additional packages into our virtual environment.

Go ahead and activate your virtual environment:

$ workon py3cv4

And then install the following packages for machine learning, image processing, and plotting:

$ pip install matplotlib scikit-learn
$ pip install pillow imutils scikit-image

Followed by Davis King’s dlib library:

$ pip install dlib

Note: While you may be tempted to compile dlib with CUDA capability for your NVIDIA Jetson Nano, currently dlib does not support the Nano’s GPU. Sources: (1) dlib GitHub issues and (2) NVIDIA devtalk forums.

Now go ahead and install Flask, a Python micro web server; and Jupyter, a web-based Python environment:

$ pip install flask jupyter

And finally, install our XML tool for the TFOD API, and progressbar for keeping track of terminal programs that take a long time:

$ pip install lxml progressbar2

Great job, but the party isn’t over yet. In the next step, we’ll test our installation.

Step #16: Testing and Validation

I always like to test my installation at this point to ensure that everything is working as I expect. This quick verification can save time down the road when you’re ready to deploy computer vision and deep learning projects on your NVIDIA Jetson Nano.

Testing TensorFlow and Keras

To test TensorFlow and Keras, simply import them in a Python shell:

$ workon py3cv4
$ python
>>> import tensorflow
>>> import keras
>>> print(tensorflow.__version__)
1.13.1
>>> print(keras.__version__)
2.3.0

Again, we are purposely not using TensorFlow 2.0. As of March 2020, when this post was written, TensorFlow 2.0 is/was not supported by TensorRT and it has memory leak issues.

Testing TFOD API and TRT Models

To test the TFOD API, we first need to run the setup script:

$ cd ~
$ ./setup.sh

And then execute the test routine as shown in Figure 12:

Assuming you see “OK” next to each test that was run, you are good to go.

Testing OpenCV

To test OpenCV, we’ll simply import it in a Python shell and load + display an image:

$ workon py3cv4
$ wget -O penguins.jpg http://pyimg.co/avp96
$ python
>>> import cv2
>>> import imutils
>>> image = cv2.imread("penguins.jpg")
>>> image = imutils.resize(image, width=400)
>>> message = "OpenCV Jetson Nano Success!"
>>> font = cv2.FONT_HERSHEY_SIMPLEX
>>> _ = cv2.putText(image, message, (30, 130), font, 0.7, (0, 255, 0), 2)
>>> cv2.imshow("Penguins", image); cv2.waitKey(0); cv2.destroyAllWindows()

Testing your webcam

In this section, we’ll develop a quick and dirty script to test your NVIDIA Jetson Nano camera using either (1) a PiCamera or (2) a USB camera.

Did you know that the NVIDIA Jetson Nano is compatible with your Raspberry Pi picamera?

In fact it is, but it requires a long source string to interact with the driver. In this section, we’ll fire up a script to see how it works.

First, connect your PiCamera to your Jetson Nano with the ribbon cable as shown:

Next, be sure to grab the “Downloads” associated with this blog post for the test script. Let’s review the test_camera_nano.py script now:

# import the necessary packages
from imutils.video import VideoStream
import imutils
import time
import cv2

# grab a reference to the webcam
print("[INFO] starting video stream...")
#vs = VideoStream(src=0).start()
vs = VideoStream(src="nvarguscamerasrc ! video/x-raw(memory:NVMM), " \
	"width=(int)1920, height=(int)1080,format=(string)NV12, " \
	"framerate=(fraction)30/1 ! nvvidconv ! video/x-raw, " \
	"format=(string)BGRx ! videoconvert ! video/x-raw, " \
	"format=(string)BGR ! appsink").start()
time.sleep(2.0)

This script uses both OpenCV and imutils as shown in the imports on Lines 2-4.

Using the video module of imutils, let’s create a VideoStream on Lines 9-14:

• USB Camera: Currently commented out on Line 9, to use your USB webcam, you simply need to provide src=0 or another device ordinal if you have more than one USB camera connected to your Nano
• PiCamera: Currently active on Lines 10-14, a lengthy src string is used to work with the driver on the Nano to access a PiCamera plugged into the MIPI port. As you can see, the width and height in the format string indicate 1080p resolution. You can also use other resolutions that your PiCamera is compatible with

We’re more interested in the PiCamera right now, so let’s focus on Lines 10-14. These lines activate a stream for the Nano to use the PiCamera interface. Take note of the commas, exclamation points, and spaces. You definitely want to get the src string correct, so enter all parameters carefully!

Next, we’ll capture and display frames:

# loop over frames
while True:
	# grab the next frame
	frame = vs.read()

	# resize the frame to have a maximum width of 500 pixels
	frame = imutils.resize(frame, width=500)

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the `q` key was pressed, break from the loop
	if key == ord("q"):
		break

# release the video stream and close open windows
vs.stop()
cv2.destroyAllWindows()

Here we begin looping over frames. We resize the frame, and display it to our screen in an OpenCV window. If the q key is pressed, we exit the loop and cleanup.

To execute the script, simply enter the following command:

$ workon py3cv4
$ python test_camera_nano.py

As you can see, now our PiCamera is working properly with the NVIDIA Jetson Nano.

Is there a faster way to get up and running?

As an alternative to the painful, tedious, and time consuming process of configuring your Nano over the course of 2+ days, I suggest grabbing a copy off the Complete Bundle of Raspberry Pi for Computer Vision.

My book includes a pre-configured Nano .img developed with my team that is ready to go out of the box. It includes TensorFlow/Keras, TensorRT, OpenCV, scikit-image, scikit-learn, and more.

All you need to do is simply:

1. Download the Jetson Nano .img file
2. Flash it to your microSD card
3. Boot your Nano
4. And begin your projects

The .img file is worth the price of the Complete Bundle bundle alone.

As Peter Lans, a Senior Software Consultant, said:

Setting up a development environment for the Jetson Nano is horrible to do. After a few attempts, I gave up and left it for another day.

Until now my Jetson does what it does best: collecting dust in a drawer. But now I have an excuse to clean it and get it running again.

Besides the fact that Adrian’s material is awesome and comprehensive, the pre-configured Nano .img bonus is the cherry on the pie, making the price of Raspberry Pi for Computer Vision even more attractive.

To anyone interested in Adrian’s RPi4CV book, be fair to yourself and calculate the hours you waste getting nowhere. It will make you realize that you’ll have spent more in wasted time than on the book bundle.

One of my Twitter followers echoed the statement:

My .img files are updated on a regular basis and distributed to customers. I also provide priority support to customers of my books and courses, something that I’m unable to offer for free to everyone on the internet who visits this website.

Simply put, if you need support with your Jetson Nano from me, I recommend picking up a copy of Raspberry Pi for Computer Vision, which offers the best embedded computer vision and deep learning education available on the internet.

In addition to the .img files, RPi4CV covers how to successfully apply Computer Vision, Deep Learning, and OpenCV to embedded devices such as the:

• Raspberry Pi
• Intel Movidus NCS
• Google Coral
• NVIDIA Jetson Nano

Inside, you’ll find over 40 projects (including 60+ chapters) on embedded Computer Vision and Deep Learning.

A handful of the highlighted projects include:

• Traffic counting and vehicle speed detection
• Real-time face recognition
• Building a classroom attendance system
• Automatic hand gesture recognition
• Daytime and nighttime wildlife monitoring
• Security applications
• Deep Learning classification, object detection, and human pose estimation on resource-constrained devices
• … and much more!

If you’re just as excited as I am, grab the free table of contents by clicking here:

Summary

In this tutorial, we configured our NVIDIA Jetson Nano for Python-based deep learning and computer vision.

We began by flashing the NVIDIA Jetpack .img. From there we installed prerequisites. We then configured a Python virtual environment for deploying computer vision and deep learning projects.

Inside our virtual environment, we installed TensorFlow, TensorFlow Object Detection (TFOD) API, TensorRT, and OpenCV.

We wrapped up by testing our software installations. We also developed a quick Python script to test both PiCamera and USB cameras.

If you’re interested in a computer vision and deep learning on the Raspberry Pi and NVIDIA Jetson Nano, be sure to pick up a copy of Raspberry Pi for Computer Vision.

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), just enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post How to configure your NVIDIA Jetson Nano for Computer Vision and Deep Learning appeared first on PyImageSearch.

In this tutorial, you will learn how to use convolutional autoencoders to create a Content-based Image Retrieval system (i.e., image search engine) using Keras and TensorFlow.

A few weeks ago, I authored a series of tutorials on autoencoders:

1. Part 1: Intro to autoencoders
2. Part 2: Denoising autoencoders
3. Part 3: Anomaly detection with autoencoders

The tutorials were a big hit; however, one topic I did not touch on was Content-based Image Retrieval (CBIR), which is really just a fancy academic word for image search engines.

Image search engines are similar to text search engines, only instead of presenting the search engine with a text query, you instead provide an image query — the image search engine then finds all visually similar/relevant images in its database and returns them to you (just as a text search engine would return links to articles, blog posts, etc.).

Deep learning-based CBIR and image retrieval can be framed as a form of unsupervised learning:

• When training the autoencoder, we do not use any class labels
• The autoencoder is then used to compute the latent-space vector representation for each image in our dataset (i.e., our “feature vector” for a given image)
• Then, at search time, we compute the distance between the latent-space vectors — the smaller the distance, the more relevant/visually similar two images are

We can thus break up the CBIR project into three distinct phases:

1. Phase #1: Train the autoencoder
2. Phase #2: Extract features from all images in our dataset by computing their latent-space representations using the autoencoder
3. Phase #3: Compare latent-space vectors to find all relevant images in the dataset

I’ll show you how to implement each of these phases in this tutorial, leaving you with a fully functioning autoencoder and image retrieval system.

To learn how to use autoencoders for image retrieval with Keras and TensorFlow, just keep reading!

Looking for the source code to this post?

Jump Right To The Downloads Section

Autoencoders for Content-based Image Retrieval with Keras and TensorFlow

In the first part of this tutorial, we’ll discuss how autoencoders can be used for image retrieval and building image search engines.

From there, we’ll implement a convolutional autoencoder that we’ll then train on our image dataset.

Once the autoencoder is trained, we’ll compute feature vectors for each image in our dataset. Computing the feature vector for a given image requires only a forward-pass of the image through the network — the output of the encoder (i.e., the latent-space representation) will serve as our feature vector.

After all images are encoded, we can then compare vectors by computing the distance between them. Images with a smaller distance will be more similar than images with a larger distance.

Finally, we will review the results of applying our autoencoder for image retrieval.

How can autoencoders be used for image retrieval and image search engines?

As discussed in my intro to autoencoders tutorial, autoencoders:

1. Accept an input set of data (i.e., the input)
2. Internally compress the input data into a latent-space representation (i.e., a single vector that compresses and quantifies the input)
3. Reconstruct the input data from this latent representation (i.e., the output)

To build an image retrieval system with an autoencoder, what we really care about is that latent-space representation vector.

Once an autoencoder has been trained to encode images, we can:

1. Use the encoder portion of the network to compute the latent-space representation of each image in our dataset — this representation serves as our feature vector that quantifies the contents of an image
2. Compare the feature vector from our query image to all feature vectors in our dataset (typically you would use either the Euclidean or cosine distance)

Feature vectors that have a smaller distance will be considered more similar, while images with a larger distance will be deemed less similar.

We can then sort our results based on the distance (from smallest to largest) and finally display the image retrieval results to the end user.

Project structure

Go ahead and grab this tutorial’s files from the “Downloads” section. From there, extract the .zip, and open the folder for inspection:

$ tree --dirsfirst
.
├── output
│   ├── autoencoder.h5
│   ├── index.pickle
│   ├── plot.png
│   └── recon_vis.png
├── pyimagesearch
│   ├── __init__.py
│   └── convautoencoder.py
├── index_images.py
├── search.py
└── train_autoencoder.py

2 directories, 9 files

This tutorial consists of three Python driver scripts:

• train_autoencoder.py: Trains an autoencoder on the MNIST handwritten digits dataset using the ConvAutoencoder CNN/class
• index_images.py: Using the encoder portion of our trained autoencoder, we’ll compute feature vectors for each image in the dataset and add the features to a searchable index
• search.py: Queries our index for similar images using a similarity metric

Our output/ directory contains our trained autoencoder and index. Training also results in a training history plot and visualization image that can be exported to the output/ folder.

Implementing our convolutional autoencoder architecture for image retrieval

Before we can train our autoencoder, we must first implement the architecture itself. To do so, we’ll be using Keras and TensorFlow.

We’ve already implemented convolutional autoencoders a handful of times before on the PyImageSearch blog, so while I’ll be covering the complete implementation here today, you’ll want to refer to my intro to autoencoders tutorial for more details.

Open up the convautoencoder.py file in the pyimagesearch module, and let’s get to work:

# import the necessary packages
from tensorflow.keras.layers import BatchNormalization
from tensorflow.keras.layers import Conv2D
from tensorflow.keras.layers import Conv2DTranspose
from tensorflow.keras.layers import LeakyReLU
from tensorflow.keras.layers import Activation
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Reshape
from tensorflow.keras.layers import Input
from tensorflow.keras.models import Model
from tensorflow.keras import backend as K
import numpy as np

Imports include a selection from tf.keras as well as NumPy. We’ll go ahead and define our autoencoder class next:

class ConvAutoencoder:
	@staticmethod
	def build(width, height, depth, filters=(32, 64), latentDim=16):
		# initialize the input shape to be "channels last" along with
		# the channels dimension itself
		# channels dimension itself
		inputShape = (height, width, depth)
		chanDim = -1

		# define the input to the encoder
		inputs = Input(shape=inputShape)
		x = inputs

		# loop over the number of filters
		for f in filters:
			# apply a CONV => RELU => BN operation
			x = Conv2D(f, (3, 3), strides=2, padding="same")(x)
			x = LeakyReLU(alpha=0.2)(x)
			x = BatchNormalization(axis=chanDim)(x)

		# flatten the network and then construct our latent vector
		volumeSize = K.int_shape(x)
		x = Flatten()(x)
		latent = Dense(latentDim, name="encoded")(x)

Our ConvAutoencoder class contains one static method, build, which accepts five parameters: (1) width, (2) height, (3) depth, (4) filters, and (5) latentDim.

The Input is then defined for the encoder, at which point we use Keras’ functional API to loop over our filters and add our sets of CONV => LeakyReLU => BN layers (Lines 21-33).

We then flatten the network and construct our latent vector (Lines 36-38).

The latent-space representation is the compressed form of our data — once trained, the output of this layer will be our feature vector used to quantify and represent the contents of the input image.

From here, we will construct the input to the decoder portion of the network:

		# start building the decoder model which will accept the
		# output of the encoder as its inputs
		x = Dense(np.prod(volumeSize[1:]))(latent)
		x = Reshape((volumeSize[1], volumeSize[2], volumeSize[3]))(x)

		# loop over our number of filters again, but this time in
		# reverse order
		for f in filters[::-1]:
			# apply a CONV_TRANSPOSE => RELU => BN operation
			x = Conv2DTranspose(f, (3, 3), strides=2,
				padding="same")(x)
			x = LeakyReLU(alpha=0.2)(x)
			x = BatchNormalization(axis=chanDim)(x)

		# apply a single CONV_TRANSPOSE layer used to recover the
		# original depth of the image
		x = Conv2DTranspose(depth, (3, 3), padding="same")(x)
		outputs = Activation("sigmoid", name="decoded")(x)

		# construct our autoencoder model
		autoencoder = Model(inputs, outputs, name="autoencoder")

		# return the autoencoder model
		return autoencoder

The decoder model accepts the output of the encoder as its inputs (Lines 42 and 43).

Looping over filters in reverse order, we construct CONV_TRANSPOSE => LeakyReLU => BN layer blocks (Lines 47-52).

Lines 56-63 recover the original depth of the image.

We wrap up by constructing and returning our autoencoder model (Lines 60-63).

For more details on our implementation, be sure to refer to our intro to autoencoders with Keras and TensorFlow tutorial.

Creating the autoencoder training script using Keras and TensorFlow

With our autoencoder implemented, let’s move on to the training script (Phase #1).

Open the train_autoencoder.py script, and insert the following code:

# set the matplotlib backend so figures can be saved in the background
import matplotlib
matplotlib.use("Agg")

# import the necessary packages
from pyimagesearch.convautoencoder import ConvAutoencoder
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.datasets import mnist
import matplotlib.pyplot as plt
import numpy as np
import argparse
import cv2

On Lines 2-12, we handle our imports. We’ll use the "Agg" backend of matplotlib so that we can export our training plot to disk. We need our custom ConvAutoencoder architecture class from the previous section. We will take advantage of the Adam optimizer as we train on the MNIST benchmarking dataset.

For visualization, we’ll employ OpenCV in the visualize_predictions helper function:

def visualize_predictions(decoded, gt, samples=10):
	# initialize our list of output images
	outputs = None

	# loop over our number of output samples
	for i in range(0, samples):
		# grab the original image and reconstructed image
		original = (gt[i] * 255).astype("uint8")
		recon = (decoded[i] * 255).astype("uint8")

		# stack the original and reconstructed image side-by-side
		output = np.hstack([original, recon])

		# if the outputs array is empty, initialize it as the current
		# side-by-side image display
		if outputs is None:
			outputs = output

		# otherwise, vertically stack the outputs
		else:
			outputs = np.vstack([outputs, output])

	# return the output images
	return outputs

Inside the visualize_predictions helper, we compare our original ground-truth input images (gt) to the output reconstructed images from the autoencoder (decoded) and generate a side-by-side comparison montage.

Line 16 initializes our list of output images.

We then loop over the samples:

• Grabbing both the original and reconstructed images (Lines 21 and 22)
• Stacking the pair of images side-by-side (Line 25)
• Stacking the pairs vertically (Lines 29-34)

Finally, we return the visualization image to the caller (Line 37).

We’ll need a few command line arguments for our script to run from our terminal/command line:

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-m", "--model", type=str, required=True,
	help="path to output trained autoencoder")
ap.add_argument("-v", "--vis", type=str, default="recon_vis.png",
	help="path to output reconstruction visualization file")
ap.add_argument("-p", "--plot", type=str, default="plot.png",
	help="path to output plot file")
args = vars(ap.parse_args())

Here we parse three command line arguments:

• --model: Points to the path of our trained output autoencoder — the result of executing this script
• --vis: The path to the output visualization image. We’ll name our visualization recon_vis.png by default
• --plot: The path to our matplotlib output plot. A default of plot.png is assigned if this argument is not provided in the terminal

Now that our imports, helper function, and command line arguments are ready, we’ll prepare to train our autoencoder:

# initialize the number of epochs to train for, initial learning rate,
# and batch size
EPOCHS = 20
INIT_LR = 1e-3
BS = 32

# load the MNIST dataset
print("[INFO] loading MNIST dataset...")
((trainX, _), (testX, _)) = mnist.load_data()

# add a channel dimension to every image in the dataset, then scale
# the pixel intensities to the range [0, 1]
trainX = np.expand_dims(trainX, axis=-1)
testX = np.expand_dims(testX, axis=-1)
trainX = trainX.astype("float32") / 255.0
testX = testX.astype("float32") / 255.0

# construct our convolutional autoencoder
print("[INFO] building autoencoder...")
autoencoder = ConvAutoencoder.build(28, 28, 1)
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
autoencoder.compile(loss="mse", optimizer=opt)

# train the convolutional autoencoder
H = autoencoder.fit(
	trainX, trainX,
	validation_data=(testX, testX),
	epochs=EPOCHS,
	batch_size=BS)

Hyperparameter constants including the number of training epochs, learning rate, and batch size are defined on Lines 51-53.

Our autoencoder (and therefore our CBIR system) will be trained on the MNIST handwritten digits dataset which we load from disk on Line 57.

To preprocess MNIST images, we add a channel dimension to the training/testing sets (Lines 61 and 62) and scale pixel intensities to the range [0, 1] (Lines 63 and 64).

With our data ready to go, Lines 68-70 compile our autoencoder with the Adam optimizer and mean-squared error loss.

Lines 73-77 then fit our model to the data (i.e., train our autoencoder).

Once the model is trained, we’ll make predictions with it:

# use the convolutional autoencoder to make predictions on the
# testing images, construct the visualization, and then save it
# to disk
print("[INFO] making predictions...")
decoded = autoencoder.predict(testX)
vis = visualize_predictions(decoded, testX)
cv2.imwrite(args["vis"], vis)

# construct a plot that plots and saves the training history
N = np.arange(0, EPOCHS)
plt.style.use("ggplot")
plt.figure()
plt.plot(N, H.history["loss"], label="train_loss")
plt.plot(N, H.history["val_loss"], label="val_loss")
plt.title("Training Loss and Accuracy")
plt.xlabel("Epoch #")
plt.ylabel("Loss/Accuracy")
plt.legend(loc="lower left")
plt.savefig(args["plot"])

# serialize the autoencoder model to disk
print("[INFO] saving autoencoder...")
autoencoder.save(args["model"], save_format="h5")

Lines 83 and 84 make predictions on the testing set and generate our autoencoder visualization using our helper function. Line 85 writes the visualization to disk using OpenCV.

Finally, we plot training history (Lines 88-97) and serialize our autoencoder to disk (Line 101).

In the next section, we’ll put the training script to work.

Training the autoencoder

We are now ready to train our convolutional autoencoder for image retrieval.

Make sure you use the “Downloads” section of this tutorial to download the source code, and from there, execute the following command to start the training process:

$ python train_autoencoder.py --model output/autoencoder.h5 \
    --vis output/recon_vis.png --plot output/plot.png
[INFO] loading MNIST dataset...
[INFO] building autoencoder...
Train on 60000 samples, validate on 10000 samples
Epoch 1/20
60000/60000 [==============================] - 73s 1ms/sample - loss: 0.0182 - val_loss: 0.0124
Epoch 2/20
60000/60000 [==============================] - 73s 1ms/sample - loss: 0.0101 - val_loss: 0.0092
Epoch 3/20
60000/60000 [==============================] - 73s 1ms/sample - loss: 0.0090 - val_loss: 0.0084
...
Epoch 18/20
60000/60000 [==============================] - 72s 1ms/sample - loss: 0.0065 - val_loss: 0.0067
Epoch 19/20
60000/60000 [==============================] - 73s 1ms/sample - loss: 0.0065 - val_loss: 0.0067
Epoch 20/20
60000/60000 [==============================] - 73s 1ms/sample - loss: 0.0064 - val_loss: 0.0067
[INFO] making predictions...
[INFO] saving autoencoder...

On my 3Ghz Intel Xeon W processor, the entire training process took ~24 minutes.

Looking at the plot in Figure 2, we can see that the training process was stable with no signs of overfitting:

Furthermore, the following reconstruction plot shows that our autoencoder is doing a fantastic job of reconstructing our input digits.

The fact that our autoencoder is doing such a good job also implies that our latent-space representation vectors are doing a good job compressing, quantifying, and representing the input image — having such a representation is a requirement when building an image retrieval system.

If the feature vectors cannot capture and quantify the contents of the image, then there is no way that the CBIR system will be able to return relevant images.

If you find that your autoencoder is failing to properly reconstruct your images, then it’s unlikely your autoencoder will perform well for image retrieval.

Take the proper care to train an accurate autoencoder — doing so will help ensure your image retrieval system returns similar images.

Implementing image indexer using the trained autoencoder

With our autoencoder successfully trained (Phase #1), we can move on to the feature extraction/indexing phase of the image retrieval pipeline (Phase #2).

This phase, at a bare minimum, requires us to use our trained autoencoder (specifically the “encoder” portion) to accept an input image, perform a forward pass, and then take the output of the encoder portion of the network to generate our index of feature vectors. These feature vectors are meant to quantify the contents of each image.

Optionally, we may also use specialized data structures such as VP-Trees and Random Projection Trees to improve the query speed of our image retrieval system.

Open up the index_images.py file in your directory structure and we’ll get started:

# import the necessary packages
from tensorflow.keras.models import Model
from tensorflow.keras.models import load_model
from tensorflow.keras.datasets import mnist
import numpy as np
import argparse
import pickle

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-m", "--model", type=str, required=True,
	help="path to trained autoencoder")
ap.add_argument("-i", "--index", type=str, required=True,
	help="path to output features index file")
args = vars(ap.parse_args())

We begin with imports. Our tf.keras imports include (1) Model so we can construct our encoder, (2) load_model so we can load our autoencoder model we trained in the previous step, and (3) our mnist dataset. Our feature vector index will be serialized as a Python pickle file.

We have two required command line arguments:

• --model: The trained autoencoder input path from the previous step
• --index: The path to the output features index file in .pickle format

From here, we’ll load and preprocess our MNIST digit data:

# load the MNIST dataset
print("[INFO] loading MNIST training split...")
((trainX, _), (testX, _)) = mnist.load_data()

# add a channel dimension to every image in the training split, then
# scale the pixel intensities to the range [0, 1]
trainX = np.expand_dims(trainX, axis=-1)
trainX = trainX.astype("float32") / 255.0

Notice that the preprocessing steps are identical to that of our training procedure.

We’ll then load our autoencoder:

# load our autoencoder from disk
print("[INFO] loading autoencoder model...")
autoencoder = load_model(args["model"])

# create the encoder model which consists of *just* the encoder
# portion of the autoencoder
encoder = Model(inputs=autoencoder.input,
	outputs=autoencoder.get_layer("encoded").output)

# quantify the contents of our input images using the encoder
print("[INFO] encoding images...")
features = encoder.predict(trainX)

Line 28 loads our autoencoder (trained in the previous step) from disk.

Then, using the autoencoder’s input, we create a Model while only accessing the encoder portion of the network (i.e., the latent-space feature vector) as the output (Lines 32 and 33).

We then pass the MNIST digit image data through the encoder to compute our feature vectors (features) on Line 37.

Finally, we construct a dictionary map of our feature data:

# construct a dictionary that maps the index of the MNIST training
# image to its corresponding latent-space representation
indexes = list(range(0, trainX.shape[0]))
data = {"indexes": indexes, "features": features}

# write the data dictionary to disk
print("[INFO] saving index...")
f = open(args["index"], "wb")
f.write(pickle.dumps(data))
f.close()

Line 42 builds a data dictionary consisting of two components:

• indexes: Integer indices of each MNIST digit image in the dataset
• features: The corresponding feature vector for each image in the dataset

To close out, Lines 46-48 serialize the data to disk in Python’s pickle format.

Indexing our image dataset for image retrieval

We are now ready to quantify our image dataset using the autoencoder, specifically using the latent-space output of the encoder portion of the network.

To quantify our image dataset using the trained autoencoder, make sure you use the “Downloads” section of this tutorial to download the source code and pre-trained model.

From there, open up a terminal and execute the following command:

$ python index_images.py --model output/autoencoder.h5 \
	--index output/index.pickle
[INFO] loading MNIST training split...
[INFO] loading autoencoder model...
[INFO] encoding images...
[INFO] saving index...

If you check the contents of your output directory, you should now see your index.pickle file:

$ ls output/*.pickle
output/index.pickle

Implementing the image search and retrieval script using Keras and TensorFlow

Our final script, our image searcher, puts all the pieces together and allows us to complete our autoencoder image retrieval project (Phase #3). Again, we’ll be using Keras and TensorFlow for this implementation.

Open up the search.py script, and insert the following contents:

# import the necessary packages
from tensorflow.keras.models import Model
from tensorflow.keras.models import load_model
from tensorflow.keras.datasets import mnist
from imutils import build_montages
import numpy as np
import argparse
import pickle
import cv2

As you can see, this script needs the same tf.keras imports as our indexer. Additionally, we’ll use my build_montages convenience script in my imutils package to display our autoencoder CBIR results.

Let’s define a function to compute the similarity between two feature vectors:

def euclidean(a, b):
	# compute and return the euclidean distance between two vectors
	return np.linalg.norm(a - b)

Here we’re the Euclidean distance to calculate the similarity between two feature vectors, a and b.

There are multiple ways to compute distances — the cosine distance can be a good alternative for many CBIR applications. I also cover other distance algorithms inside the PyImageSearch Gurus course.

Next, we’ll define our searching function:

def perform_search(queryFeatures, index, maxResults=64):
	# initialize our list of results
	results = []

	# loop over our index
	for i in range(0, len(index["features"])):
		# compute the euclidean distance between our query features
		# and the features for the current image in our index, then
		# update our results list with a 2-tuple consisting of the
		# computed distance and the index of the image
		d = euclidean(queryFeatures, index["features"][i])
		results.append((d, i))

	# sort the results and grab the top ones
	results = sorted(results)[:maxResults]

	# return the list of results
	return results

Our perform_search function is responsible for comparing all feature vectors for similarity and returning the results.

This function accepts both the queryFeatures, a feature vector for the query image, and the index of all features to search through.

Our results will contain the top maxResults (in our case 64 is the default but we will soon override it to 225).

Line 17 initializes our list of results, which Lines 20-20 then populate. Here, we loop over all entries in our index, computing the Euclidean distance between our queryFeatures and the current feature vector in the index.

When it comes to the distance:

• The smaller the distance, the more similar the two images are
• The larger the distance, the less similar they are

We sort and grab the top results such that images that are more similar to the query are at the front of the list via Line 29.

Finally, we return the the search results to the calling function (Line 32).

With both our distance metric and searching utility defined, we’re now ready to parse command line arguments:

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-m", "--model", type=str, required=True,
	help="path to trained autoencoder")
ap.add_argument("-i", "--index", type=str, required=True,
	help="path to features index file")
ap.add_argument("-s", "--sample", type=int, default=10,
	help="# of testing queries to perform")
args = vars(ap.parse_args())

Our script accepts three command line arguments:

• --model: The path to the trained autoencoder from the “Training the autoencoder” section
• --index: Our index of features to search through (i.e., the serialized index from the “Indexing our image dataset for image retrieval” section)
• --sample: The number of testing queries to perform with a default of 10

Now, let’s load and preprocess our digit data:

# load the MNIST dataset
print("[INFO] loading MNIST dataset...")
((trainX, _), (testX, _)) = mnist.load_data()

# add a channel dimension to every image in the dataset, then scale
# the pixel intensities to the range [0, 1]
trainX = np.expand_dims(trainX, axis=-1)
testX = np.expand_dims(testX, axis=-1)
trainX = trainX.astype("float32") / 255.0
testX = testX.astype("float32") / 255.0

And then we’ll load our autoencoder and index:

# load the autoencoder model and index from disk
print("[INFO] loading autoencoder and index...")
autoencoder = load_model(args["model"])
index = pickle.loads(open(args["index"], "rb").read())

# create the encoder model which consists of *just* the encoder
# portion of the autoencoder
encoder = Model(inputs=autoencoder.input,
	outputs=autoencoder.get_layer("encoded").output)

# quantify the contents of our input testing images using the encoder
print("[INFO] encoding testing images...")
features = encoder.predict(testX)

Here, Line 57 loads our trained autoencoder from disk, while Line 58 loads our pickled index from disk.

We then build a Model that will accept our images as an input and the output of our encoder layer (i.e., feature vector) as our model’s output (Lines 62 and 63).

Given our encoder, Line 67 performs a forward-pass of our set of testing images through the network, generating a list of features to quantify them.

We’ll now take a random sample of images, marking them as queries:

# randomly sample a set of testing query image indexes
queryIdxs = list(range(0, testX.shape[0]))
queryIdxs = np.random.choice(queryIdxs, size=args["sample"],
	replace=False)

# loop over the testing indexes
for i in queryIdxs:
	# take the features for the current image, find all similar
	# images in our dataset, and then initialize our list of result
	# images
	queryFeatures = features[i]
	results = perform_search(queryFeatures, index, maxResults=225)
	images = []

	# loop over the results
	for (d, j) in results:
		# grab the result image, convert it back to the range
		# [0, 255], and then update the images list
		image = (trainX[j] * 255).astype("uint8")
		image = np.dstack([image] * 3)
		images.append(image)

	# display the query image
	query = (testX[i] * 255).astype("uint8")
	cv2.imshow("Query", query)

	# build a montage from the results and display it
	montage = build_montages(images, (28, 28), (15, 15))[0]
	cv2.imshow("Results", montage)
	cv2.waitKey(0)

Lines 70-72 sample a set of testing image indices, marking them as our search engine queries.

We then loop over the queries beginning on Line 75. Inside, we:

• Grab the queryFeatures, and perform the search (Lines 79 and 80)
• Initialize a list to hold our result images (Line 81)
• Loop over the results, scaling the image back to the range [0, 255], creating an RGB representation from the grayscale image for display, and then adding it to our images results (Lines 84-89)
• Display the query image in its own OpenCV window (Lines 92 and 93)
• Display a montage of search engine results (Lines 96 and 97)
• When the user presses a key, we repeat the process (Line 98) with a different query image; you should continue to press a key as you inspect results until all of our query samples have been searched

To recap our search searching script, first we loaded our autoencoder and index.

We then grabbed the encoder portion of the autoencoder and used it to quantify our images (i.e., create feature vectors).

From there, we created a sample of random query images to test our searching method which is based on the Euclidean distance computation. Smaller distances indicate similar images — the similar images will be shown first because our results are sorted (Line 29).

We searched our index for each query showing only a maximum of maxResults in each montage.

In the next section, we’ll get the chance to visually validate how our autoencoder-based search engine works.

Image retrieval results using autoencoders, Keras, and TensorFlow

We are now ready to see our autoencoder image retrieval system in action!

Start by making sure you have:

1. Used the “Downloads” section of this tutorial to download the source code
2. Executed the train_autoencoder.py file to train the convolutional autoencoder
3. Run the index_images.py to quantify each image in our dataset

From there, you can execute the search.py script to perform a search:

$ python search.py --model output/autoencoder.h5 \
	--index output/index.pickle
[INFO] loading MNIST dataset...
[INFO] loading autoencoder and index...
[INFO] encoding testing images...

Below is an example providing a query image containing the digit 9 (top) along with the search results from our autoencoder image retrieval system (bottom):

Here, you can see that our system has returned search results also containing nines.

Let’s now use a 2 as our query image:

Sure enough, our CBIR system returns digits containing twos, implying that latent-space representation has correctly quantified what a 2 looks like.

Here’s an example of using a 4 as a query image:

Again, our autoencoder image retrieval system returns all fours as the search results.

Let’s look at one final example, this time using a 0 as a query image:

This result is more interesting — note the two highlighted results in the screenshot.

The first highlighted result is likely a 5, but the tail of the five seems to be connecting to the middle part, creating a digit that looks like a cross between a 0 and an 8.

We then have what I think is an 8 near the bottom of the search results (also highlighted in red). Again, we can appreciate how our image retrieval system may see that 8 as visually similar to a 0.

Tips to improve autoencoder image retrieval accuracy and speed

In this tutorial, we performed image retrieval on the MNIST dataset to demonstrate how autoencoders can be used to build image search engines.

However, you will more than likely want to use your own image dataset rather than the MNIST dataset.

Swapping in your own dataset is as simple as replacing the MNIST dataset loader helper function with your own dataset loader — you can then train an autoencoder on your dataset.

However, make sure your autoencoder accuracy is sufficient.

If your autoencoder cannot reasonably reconstruct your input data, then:

1. The autoencoder is failing to capture the patterns in your dataset
2. The latent-space vector will not properly quantify your images
3. And without proper quantification, your image retrieval system will return irrelevant results

Therefore, nearly the entire accuracy of your CBIR system hinges on your autoencoder — take the time to ensure it is properly trained.

Once your autoencoder is performing well, you can then move on to optimizing the speed of your search procedure.

Secondly, you should also consider the scalability of your CBIR system.

Our implementation here is an example of a linear search with O(N) complexity, meaning that it will not scale well.

To improve the speed of the retrieval system, you should use Approximate Nearest Neighbor algorithms and specialized data structures such as VP-Trees, Random Projection trees, etc., which can reduce the computational complexity to O(log N).

To learn more about these techniques, refer to my article on Building an Image Hashing Search Engine with VP-Trees and OpenCV.

What’s next?

If you want to increase your computer vision knowledge, then look no further than the PyImageSearch Gurus course and community.

Inside the course you’ll find:

• An actionable, real-world course on OpenCV and computer vision. Each lesson in PyImageSearch Gurus is taught in the same trademark, hands-on, easy-to-understand PyImageSearch style that you know and love
• The most comprehensive computer vision education online today. The PyImageSearch Gurus course covers 13 modules broken out into 168 lessons, with over 2,161 pages of content. You won’t find a more detailed computer vision course anywhere else online; I guarantee it
• A community of like-minded developers, researchers, and students just like you, who are eager to learn computer vision and level-up their skills

The course covers breadth and depth in the following subject areas, giving you the skills to rise in the ranks at your institution or even to land that next job:

• Automatic License Plate Recognition (ANPR) — recognize license plates of vehicles, or apply the concepts to your own OCR project
• Face Detection and Recognition — recognize who’s entering/leaving your house, build a smart classroom attendance system, or identify who’s who in your collection of family portraits
• Image Search Engines also known as Content Based Image Retrieval (CBIR)
• Object Detection — and my 6-step framework to accomplish it
• Big Data methodologies — use Hadoop for executing image processing algorithms in parallel on large computing clusters
• Machine Learning and Deep Learning — learn just what you need to know to be dangerous in today’s AI age, and prime your pump for even more advanced deep learning inside my book, Deep Learning for Computer Vision with Python

If the course sounds interesting to you, I’d love to send you 10 free sample lessons and the entire course syllabus so you can get a feel for what the course has to offer. Just click the link below!

Summary

In this tutorial, you learned how to use convolutional autoencoders for image retrieval using TensorFlow and Keras.

To create our image retrieval system, we:

1. Trained a convolutional autoencoder on our image dataset
2. Used the trained autoencoder to compute the latent-space representation of each image in our dataset — this representation serves as our feature vector that quantifies the contents of the image
3. Compared the feature vector from our query image to all feature vectors in our dataset using a distance function (in this case, the Euclidean distance, but cosine distance would also work well here). The smaller the distance between the vectors the more similar our images were.

We then sorted our results based on the computed distance and displayed our results to the user.

Autoencoders can be extremely useful for CBIR applications — the downside is that they require a lot of training data, which you may or may not have.

More advanced deep learning image retrieval systems rely on siamese networks and triplet loss to embed vectors for images such that more similar images lie closer together in a Euclidean space, while less similar images are farther away — I’ll be covering these types of network architectures and techniques at a future date.

To download the source code to this post (including the pre-trained autoencoder), just enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Autoencoders for Content-based Image Retrieval with Keras and TensorFlow appeared first on PyImageSearch.

In this tutorial, you will learn how to blur and anonymize faces using OpenCV and Python.

Today’s blog post is inspired by an email I received last week from PyImageSearch reader, Li Wei:

Hi Adrian, I’m working on a research project for my university.

I’m in charge of creating the dataset but my professor has asked me to “anonymize” each image by detecting faces and then blurring them to ensure privacy is protected and that no face can be recognized (apparently this is a requirement at my institution before we publicly distribute the dataset).

Do you have any tutorials on face anonymization? How can I blur faces using OpenCV?

Thanks,

Li Wei

Li asks a great question — we often utilize face detection in our projects, typically as the first step in a face recognition pipeline.

But what if we wanted to do the “opposite” of face recognition? What if we instead wanted to anonymize the face by blurring it, thereby making it impossible to identify the face?

Practical applications of face blurring and anonymization include:

• Privacy and identity protection in public/private areas
• Protecting children online (i.e., blur faces of minors in uploaded photos)
• Photo journalism and news reporting (e.g., blur faces of people who did not sign a waiver form)
• Dataset curation and distribution (e.g., anonymize individuals in dataset)
• … and more!

To learn how to blur and anonymize faces with OpenCV and Python, just keep reading!

Looking for the source code to this post?

Jump Right To The Downloads Section

Blur and anonymize faces with OpenCV and Python

In the first part of this tutorial, we’ll briefly discuss what face blurring is and how we can use OpenCV to anonymize faces in images and video streams.

From there, we’ll discuss the four-step method to blur faces with OpenCV and Python.

We’ll then review our project structure and implement two methods for face blurring with OpenCV:

1. Using a Gaussian blur to anonymize faces in images and video streams
2. Applying a “pixelated blur” effect to anonymize faces in images and video

Given our two implementations, we’ll create Python driver scripts to apply these face blurring methods to both images and video.

We’ll then review the results of our face blurring and anonymization methods.

What is face blurring, and how can it be used for face anonymization?

Face blurring is a computer vision method used to anonymize faces in images and video.

An example of face blurring and anonymization can be seen in Figure 1 above — notice how the face is blurred, and the identity of the person is indiscernible.

We use face blurring to help protect the identity of a person in an image.

4 Steps to perform face blurring and anonymization

Applying face blurring with OpenCV and computer vision is a four-step process.

Step #1 is to perform face detection.

Any face detector can be used here, provided that it can produce the bounding box coordinates of a face in an image or video stream.

Typical face detectors that you may use include

• Haar cascades
• HOG + Linear SVM
• Deep learning-based face detectors.

You can refer to this face detection guide for more information on how to detect faces in an image.

Once you have detected a face, Step #2 is to extract the Region of Interest (ROI):

Your face detector will give you the bounding box (x, y)-coordinates of a face in an image.

These coordinates typically represent:

• The starting x-coordinate of the face bounding box
• The ending x-coordinate of the face
• The starting y-coordinate of the face location
• The ending y-coordinate of the face

You can then use this information to extract the face ROI itself, as shown in Figure 4 above.

Given the face ROI, Step #3 is to actually blur/anonymize the face:

Typically, you’ll apply a Gaussian blur to anonymize the face. You may also apply methods to pixelate the face if you find the end result more aesthetically pleasing.

Exactly how you “blur” the image is up to you — the important part is that the face is anonymized.

With the face blurred and anonymized, Step #4 is to store the blurred face back in the original image:

Using the original (x, y)-coordinates from the face detection (i.e., Step #2), we can take the blurred/anonymized face and then store it back in the original image (if you’re utilizing OpenCV and Python, this step is performed using NumPy array slicing).

The face in the original image has been blurred and anonymized — at this point the face anonymization pipeline is complete.

Let’s see how we can implement face blurring and anonymization with OpenCV in the remainder of this tutorial.

How to install OpenCV for face blurring

To follow my face blurring tutorial, you will need OpenCV installed on your system. I recommend installing OpenCV 4 using one of my tutorials:

• pip install opencv — the easiest and fastest method
• How to install OpenCV 4 on Ubuntu
• Install OpenCV 4 on macOS

I recommend the pip installation method for 99% of readers — it’s also how I typically install OpenCV for quick projects like face blurring.

If you think you might need the full install of OpenCV with patented algorithms, you should consider either the second or third bullet depending on your operating system. Both of these guides require compiling from source, which takes considerably longer as well, but can (1) give you the full OpenCV install and (2) allow you to optimize OpenCV for your operating system and system architecture.

Once you have OpenCV installed, you can move on with the rest of the tutorial.

Note: I don’t support the Windows OS here at PyImageSearch. See my FAQ page.

Project structure

Go ahead and use the “Downloads” section of this tutorial to download the source code, example images, and pre-trained face detector model. From there, let’s inspect the contents:

$ tree --dirsfirst
.
├── examples
│   ├── adrian.jpg
│   ├── chris_evans.png
│   ├── robert_downey_jr.png
│   ├── scarlett_johansson.png
│   └── tom_king.jpg
├── face_detector
│   ├── deploy.prototxt
│   └── res10_300x300_ssd_iter_140000.caffemodel
├── pyimagesearch
│   ├── __init__.py
│   └── face_blurring.py
├── blur_face.py
└── blur_face_video.py

3 directories, 11 files

The first step of face blurring is perform face detection to localize faces in a image/frame. We’ll use a deep learning-based Caffe model as shown in the face_detector/ directory.

Our two Python driver scripts, blur_face.py and blur_face_video.py, first detect faces and then perform face blurring in images and video streams. We will step through both scripts so that you can adapt them for your own projects.

First, we’ll review face blurring helper functions inside the face_blurring.py file.

Blurring faces with a Gaussian blur and OpenCV

We’ll be implementing two helper functions to aid us in face blurring and anonymity:

• anonymize_face_simple: Performs a simple Gaussian blur on the face ROI (such as in Figure 7 above)
• anonymize_face_pixelate: Creates a pixelated blur-like effect (which we’ll cover in the next section)

Let’s take a look at the implementation of anonymize_face_simple — open up the face_blurring.py file in the pyimagesearch module, and insert the following code:

# import the necessary packages
import numpy as np
import cv2

def anonymize_face_simple(image, factor=3.0):
	# automatically determine the size of the blurring kernel based
	# on the spatial dimensions of the input image
	(h, w) = image.shape[:2]
	kW = int(w / factor)
	kH = int(h / factor)

	# ensure the width of the kernel is odd
	if kW % 2 == 0:
		kW -= 1

	# ensure the height of the kernel is odd
	if kH % 2 == 0:
		kH -= 1

	# apply a Gaussian blur to the input image using our computed
	# kernel size
	return cv2.GaussianBlur(image, (kW, kH), 0)

Our face blurring utilities require NumPy and OpenCV imports as shown on Lines 2 and 3.

Beginning on Line 5, we define our anonymize_face_simple function, which accepts an input face image and blurring kernel scale factor.

Lines 8-18 derive the blurring kernel’s width and height as a function of the input image dimensions:

• The larger the kernel size, the more blurred the output face will be
• The smaller the kernel size, the less blurred the output face will be

Increasing the factor will therefore increase the amount of blur applied to the face.

When applying a blur, our kernel dimensions must be odd integers such that the kernel can be placed at a central (x, y)-coordinate of the input image (see my tutorial on convolutions with OpenCV for more information on why kernels must be odd integers).

Once we have our kernel dimensions, kW and kH, Line 22 applies a Gaussian blur kernel to the face image and returns the blurred face to the calling function.

In the next section, we’ll cover an alternative anonymity method: pixelated blurring.

Creating a pixelated face blur with OpenCV

The second method we’ll be implementing for face blurring and anonymization creates a pixelated blur-like effect — an example of such a method can be seen in Figure 8.

Notice how we have pixelated the image and made the identity of the person indiscernible.

This pixelated type of face blurring is typically what most people think of when they hear “face blurring” — it’s the same type of face blurring you’ll see on the evening news, mainly because it’s a bit more “aesthetically pleasing” to the eye than a Gaussian blur (which is indeed a bit “jarring”).

Let’s learn how to implement this pixelated face blurring method with OpenCV — open up the face_blurring.py file (the same file we used in the previous section), and append the following code:

def anonymize_face_pixelate(image, blocks=3):
	# divide the input image into NxN blocks
	(h, w) = image.shape[:2]
	xSteps = np.linspace(0, w, blocks + 1, dtype="int")
	ySteps = np.linspace(0, h, blocks + 1, dtype="int")

	# loop over the blocks in both the x and y direction
	for i in range(1, len(ySteps)):
		for j in range(1, len(xSteps)):
			# compute the starting and ending (x, y)-coordinates
			# for the current block
			startX = xSteps[j - 1]
			startY = ySteps[i - 1]
			endX = xSteps[j]
			endY = ySteps[i]

			# extract the ROI using NumPy array slicing, compute the
			# mean of the ROI, and then draw a rectangle with the
			# mean RGB values over the ROI in the original image
			roi = image[startY:endY, startX:endX]
			(B, G, R) = [int(x) for x in cv2.mean(roi)[:3]]
			cv2.rectangle(image, (startX, startY), (endX, endY),
				(B, G, R), -1)

	# return the pixelated blurred image
	return image

Beginning on Line 24, we define our anonymize_face_pixilate function and parameters. This function accepts a face image and the number of pixel blocks.

Lines 26-28 grab our face image dimensions and divide it into MxN blocks.

From there, we proceed to loop over the blocks in both the x and y directions (Lines 31 and 32).

In order to compute the starting and ending bounding coordinates for the current block, we use our step indices, i and j (Lines 35-38).

Subsequently, we extract the current block ROI and compute the mean RGB pixel intensities for the ROI (Lines 43 and 44).

We then annotate a rectangle on the block using the computed mean RGB values, thereby creating the “pixelated”-like effect (Lines 45 and 46).

Note: To learn more about OpenCV drawing functions, be sure to spend some time on my OpenCV Tutorial.

Finally, Line 49 returns our pixelated face image to the caller.

Implementing face blurring in images with OpenCV

Now that we have our two face blurring methods implemented, let’s learn how we can apply them to blur a face in an image using OpenCV and Python.

Open up the blur_face.py file in your project structure, and insert the following code:

# import the necessary packages
from pyimagesearch.face_blurring import anonymize_face_pixelate
from pyimagesearch.face_blurring import anonymize_face_simple
import numpy as np
import argparse
import cv2
import os

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
	help="path to input image")
ap.add_argument("-f", "--face", required=True,
	help="path to face detector model directory")
ap.add_argument("-m", "--method", type=str, default="simple",
	choices=["simple", "pixelated"],
	help="face blurring/anonymizing method")
ap.add_argument("-b", "--blocks", type=int, default=20,
	help="# of blocks for the pixelated blurring method")
ap.add_argument("-c", "--confidence", type=float, default=0.5,
	help="minimum probability to filter weak detections")
args = vars(ap.parse_args())

Our most notable imports are both our face pixelation and face blurring functions from the previous two sections (Lines 2 and 3).

Our script accepts five command line arguments, the first two of which are required:

• --image: The path to your input image containing faces
• --face: The path to your face detector model directory
• --method: Either the simple blurring or pixelated methods can be chosen with this flag. The simple method is the default
• --blocks: For pixelated face anonymity, you must provide the number of blocks you want to use, or you can keep the default of 20
• --confidence: The minimum probability to filter weak face detections is set to 50% by default

Given our command line arguments, we’re now ready to perform face detection:

# load our serialized face detector model from disk
print("[INFO] loading face detector model...")
prototxtPath = os.path.sep.join([args["face"], "deploy.prototxt"])
weightsPath = os.path.sep.join([args["face"],
	"res10_300x300_ssd_iter_140000.caffemodel"])
net = cv2.dnn.readNet(prototxtPath, weightsPath)

# load the input image from disk, clone it, and grab the image spatial
# dimensions
image = cv2.imread(args["image"])
orig = image.copy()
(h, w) = image.shape[:2]

# construct a blob from the image
blob = cv2.dnn.blobFromImage(image, 1.0, (300, 300),
	(104.0, 177.0, 123.0))

# pass the blob through the network and obtain the face detections
print("[INFO] computing face detections...")
net.setInput(blob)
detections = net.forward()

First, we load the Caffe-based face detector model (Lines 26-29).

We then load and preprocess our input --image, generating a blob for inference (Lines 33-39). Read my How OpenCV’s blobFromImage works tutorial to learn the “why” and “how” behind the function call on Lines 38 and 39.

Deep learning face detection inference (Step #1) takes place on Lines 43 and 44.

Next, we’ll begin looping over the detections:

# loop over the detections
for i in range(0, detections.shape[2]):
	# extract the confidence (i.e., probability) associated with the
	# detection
	confidence = detections[0, 0, i, 2]

	# filter out weak detections by ensuring the confidence is greater
	# than the minimum confidence
	if confidence > args["confidence"]:
		# compute the (x, y)-coordinates of the bounding box for the
		# object
		box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
		(startX, startY, endX, endY) = box.astype("int")

		# extract the face ROI
		face = image[startY:endY, startX:endX]

Here, we loop over detections and check the confidence, ensuring it meets the minimum threshold (Lines 47-54).

Assuming so, we then extract the face ROI (Step #2) via Lines 57-61.

We’ll then anonymize the face (Step #3):

		# check to see if we are applying the "simple" face blurring
		# method
		if args["method"] == "simple":
			face = anonymize_face_simple(face, factor=3.0)

		# otherwise, we must be applying the "pixelated" face
		# anonymization method
		else:
			face = anonymize_face_pixelate(face,
				blocks=args["blocks"])

		# store the blurred face in the output image
		image[startY:endY, startX:endX] = face

Depending on the --method, we’ll perform simple blurring or pixelation to anonymize the face (Lines 65-72).

Step #4 entails overwriting the original face ROI in the image with our anonymized face ROI (Line 75).

Steps #2-#4 are then repeated for all faces in the input --image until we’re ready to display the result:

# display the original image and the output image with the blurred
# face(s) side by side
output = np.hstack([orig, image])
cv2.imshow("Output", output)
cv2.waitKey(0)

To wrap up, the original and altered images are displayed side by side until a key is pressed (Lines 79-81).

Face blurring and anonymizing in images results

Let’s now put our face blurring and anonymization methods to work.

Go ahead and use the “Downloads” section of this tutorial to download the source code, example images, and pre-trained OpenCV face detector.

From there, open up a terminal, and execute the following command:

$ python blur_face.py --image examples/adrian.jpg --face face_detector
[INFO] loading face detector model...
[INFO] computing face detections...

On the left, you can see the original input image (i.e., me), while the right shows that my face has been blurred using the Gaussian blurring method — without seeing the original image, you would have no idea it was me (other than the tattoos, I suppose).

Let’s try another image, this time applying the pixelated blurring technique:

$ python blur_face.py --image examples/tom_king.jpg --face face_detector --method pixelated
[INFO] loading face detector model...
[INFO] computing face detections...

On the left, we have the original input image of Tom King, one of my favorite comic writers.

Then, on the right, we have the output of the pixelated blurring method — without seeing the original image, you would have no idea whose face was in the image.

Implementing face blurring in real-time video with OpenCV

Our previous example only handled blurring and anonymizing faces in images — but what if we wanted to apply face blurring and anonymization to real-time video streams?

Is that possible?

You bet it is!

Open up the blur_face_video.py file in your project structure, and let’s learn how to blur faces in real-time video with OpenCV:

# import the necessary packages
from pyimagesearch.face_blurring import anonymize_face_pixelate
from pyimagesearch.face_blurring import anonymize_face_simple
from imutils.video import VideoStream
import numpy as np
import argparse
import imutils
import time
import cv2
import os

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-f", "--face", required=True,
	help="path to face detector model directory")
ap.add_argument("-m", "--method", type=str, default="simple",
	choices=["simple", "pixelated"],
	help="face blurring/anonymizing method")
ap.add_argument("-b", "--blocks", type=int, default=20,
	help="# of blocks for the pixelated blurring method")
ap.add_argument("-c", "--confidence", type=float, default=0.5,
	help="minimum probability to filter weak detections")
args = vars(ap.parse_args())

We begin with our imports on Lines 2-10. For face recognition in video, we’ll use the VideoStream API in my imutils package (Line 4).

Our command line arguments are the same as previously (Lines 13-23).

We’ll then load our face detector and initialize our video stream:

# load our serialized face detector model from disk
print("[INFO] loading face detector model...")
prototxtPath = os.path.sep.join([args["face"], "deploy.prototxt"])
weightsPath = os.path.sep.join([args["face"],
	"res10_300x300_ssd_iter_140000.caffemodel"])
net = cv2.dnn.readNet(prototxtPath, weightsPath)

# initialize the video stream and allow the camera sensor to warm up
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
time.sleep(2.0)

Our video stream accesses our computer’s webcam (Line 34).

We’ll then proceed to loop over frames in the stream and perform Step #1 — face detection:

# loop over the frames from the video stream
while True:
	# grab the frame from the threaded video stream and resize it
	# to have a maximum width of 400 pixels
	frame = vs.read()
	frame = imutils.resize(frame, width=400)

	# grab the dimensions of the frame and then construct a blob
	# from it
	(h, w) = frame.shape[:2]
	blob = cv2.dnn.blobFromImage(frame, 1.0, (300, 300),
		(104.0, 177.0, 123.0))

	# pass the blob through the network and obtain the face detections
	net.setInput(blob)
	detections = net.forward()

Once faces are detected, we’ll ensure they meet the minimum confidence threshold:

	# loop over the detections
	for i in range(0, detections.shape[2]):
		# extract the confidence (i.e., probability) associated with
		# the detection
		confidence = detections[0, 0, i, 2]

		# filter out weak detections by ensuring the confidence is
		# greater than the minimum confidence
		if confidence > args["confidence"]:
			# compute the (x, y)-coordinates of the bounding box for
			# the object
			box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
			(startX, startY, endX, endY) = box.astype("int")

			# extract the face ROI
			face = frame[startY:endY, startX:endX]

			# check to see if we are applying the "simple" face
			# blurring method
			if args["method"] == "simple":
				face = anonymize_face_simple(face, factor=3.0)

			# otherwise, we must be applying the "pixelated" face
			# anonymization method
			else:
				face = anonymize_face_pixelate(face,
					blocks=args["blocks"])

			# store the blurred face in the output image
			frame[startY:endY, startX:endX] = face

Looping over high confidence detections, we extract the face ROI (Step #2) on Lines 55-69.

To accomplish Step #3, we apply our chosen anonymity --method via Lines 73-80.

And finally, for Step #4, we replace the anonymous face in our camera’s frame (Line 83).

To close out our face blurring loop, we display the frame (with blurred out faces) on the screen:

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the `q` key was pressed, break from the loop
	if key == ord("q"):
		break

# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()

If the q key is pressed, we break out of the face blurring loop and perform cleanup.

Great job — in the next section, we’ll analyze results!

Real-time face blurring OpenCV results

We are now ready to apply face blurring with OpenCV to real-time video streams.

Start by using the “Downloads” section of this tutorial to download the source code and pre-trained OpenCV face detector.

You can then launch the blur_face_video.py using the following command:

$ python blur_face_video.py --face face_detector --method simple
[INFO] loading face detector model...
[INFO] starting video stream...

Notice how my face is blurred in the video stream using the Gaussian blurring method.

We can apply the pixelated face blurring method by supplying the --method pixelated flag:

$ python blur_face_video.py --face face_detector --method pixelated
[INFO] loading face detector model...
[INFO] starting video stream...

Again, my face is anonymized/blurred using OpenCV, but using the more “aesthetically pleasing” pixelated method.

Handling missed face detections and “detection flickering”

The face blurring method we’re applying here assumes that a face can be detected in each and every frame of our input video stream.

But what happens if our face detector misses a detection, such as in video at the top of this section?

If our face detector misses a face detection, then the face cannot be blurred, thereby defeating the purpose of face blurring and anonymization.

So what do we do in those situations?

Typically, the easiest method is to take the last known location of the face (i.e., the previous detection location) and then blur that region.

Faces don’t tend to move very quickly, so blurring the last known location will help ensure the face is anonymized even when your face detector misses the face.

A more advanced option is to use dedicated object trackers similar to what we do in our people/footfall counter guide.

Using this method you would:

1. Detect faces in the video stream
2. Create an object tracker for each face
3. Use the object tracker and face detector to correlate the position of the face
4. If the face detector misses a detection, then fall back on the tracker to provide the location of the face

This method is more computationally complex than the simple “last known location,” but it’s also far more robust.

I’ll leave implementing those methods up to you (although I am tempted to cover them in a future tutorial, as they are pretty fun methods to implement).

Are you interested in learning more about Computer Vision, OpenCV, and Face Applications?

If so, you’ll want to take a look at the PyImageSearch Gurus course.

Inside the course you’ll find:

• An actionable, real-world course on Computer Vision, Deep Learning, and OpenCV. Each lesson in the course is taught in the same hands-on, easy-to-understand PyImageSearch style that you know and love
• The most comprehensive computer vision education online today. The PyImageSearch Gurus course covers 13 modules broken out into 168 lessons, with over 2,161 pages of content. You won’t find a more detailed computer vision course anywhere else online; I guarantee it
• A community of like-minded developers, researchers, and students just like you, who are eager to learn computer vision and level-up their skills
• Access to private course forums that I personally participate in nearly every day. These forums are a great way to get expert advice, both from me as well as the more advanced students

If you’re interested in learning more, I’d love to send you (1) a PDF containing the course syllabus and (2) a set of sample lessons in the course:

Summary

In this tutorial, you learned how to blur and anonymize faces in both images and real-time video streams using OpenCV and Python.

Face blurring and anonymization is a four-step process:

1. Step #1: Apply a face detector (i.e., Haar cascades, HOG + Linear SVM, deep learning-based face detectors) to detect the presence of a face in an image
2. Step #2: Use the bounding box (x, y)-coordinates to extract the face ROI from the input image
3. Step #3: Blur the face in the image, typically with a Gaussian blur or pixelated blur, thereby anonymizing the face and protecting the identity of the person in the image
4. Step #4: Store the blurred/anonymized face back in the original image

We then implemented this entire pipeline using only OpenCV and Python.

I hope you’ve found this tutorial helpful!

To download the source code to this post (including the example images and pre-trained face detector), just enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Blur and anonymize faces with OpenCV and Python appeared first on PyImageSearch.

In this tutorial, you will learn how to perform automatic age detection/prediction using OpenCV, Deep Learning, and Python.

By the end of this tutorial, you will be able to automatically predict age in static image files and real-time video streams with reasonably high accuracy.

To learn how to perform age detection with OpenCV and Deep Learning, just keep reading!

Looking for the source code to this post?

Jump Right To The Downloads Section

OpenCV Age Detection with Deep Learning

In the first part of this tutorial, you’ll learn about age detection, including the steps required to automatically predict the age of a person from an image or a video stream (and why age detection is best treated as a classification problem rather than a regression problem).

From there, we’ll discuss our deep learning-based age detection model and then learn how to use the model for both:

1. Age detection in static images
2. Age detection in real-time video streams

We’ll then review the results of our age prediction work.

What is age detection?

Age detection is the process of automatically discerning the age of a person solely from a photo of their face.

Typically, you’ll see age detection implemented as a two-stage process:

1. Stage #1: Detect faces in the input image/video stream
2. Stage #2: Extract the face Region of Interest (ROI), and apply the age detector algorithm to predict the age of the person

For Stage #1, any face detector capable of producing bounding boxes for faces in an image can be used, including but not limited to Haar cascades, HOG + Linear SVM, Single Shot Detectors (SSDs), etc.

Exactly which face detector you use depends on your project:

• Haar cascades will be very fast and capable of running in real-time on embedded devices — the problem is that they are less accurate and highly prone to false-positive detections
• HOG + Linear SVM models are more accurate than Haar cascades but are slower. They also aren’t as tolerant with occlusion (i.e., not all of the face visible) or viewpoint changes (i.e., different views of the face)
• Deep learning-based face detectors are the most robust and will give you the best accuracy, but require even more computational resources than both Haar cascades and HOG + Linear SVMs

When choosing a face detector for your application, take the time to consider your project requirements — is speed or accuracy more important for your use case? I also recommend running a few experiments with each of the face detectors so you can let the empirical results guide your decisions.

Once your face detector has produced the bounding box coordinates of the face in the image/video stream, you can move on to Stage #2 — identifying the age of the person.

Given the bounding box (x, y)-coordinates of the face, you first extract the face ROI, ignoring the rest of the image/frame. Doing so allows the age detector to focus solely on the person’s face and not any other irrelevant “noise” in the image.

The face ROI is then passed through the model, yielding the actual age prediction.

There are a number of age detector algorithms, but the most popular ones are deep learning-based age detectors — we’ll be using such a deep learning-based age detector in this tutorial.

Our age detector deep learning model

The deep learning age detector model we are using here today was implemented and trained by Levi and Hassner in their 2015 publication, Age and Gender Classification Using Convolutional Neural Networks.

In the paper, the authors propose a simplistic AlexNet-like architecture that learns a total of eight age brackets:

1. 0-2
2. 4-6
3. 8-12
4. 15-20
5. 25-32
6. 38-43
7. 48-53
8. 60-100

You’ll note that these age brackets are noncontiguous — this done on purpose, as the Adience dataset, used to train the model, defines the age ranges as such (we’ll learn why this is done in the next section).

We’ll be using a pre-trained age detector model in this post, but if you are interested in learning how to train it from scratch, be sure to read Deep Learning for Computer Vision with Python, where I show you how to do exactly that.

Why aren’t we treating age prediction as a regression problem?

You’ll notice from the previous section that we have discretized ages into “buckets,” thereby treating age prediction as a classification problem — why not frame it as a regression problem instead (the way we did in our house price prediction tutorial)?

Technically, there’s no reason why you can’t treat age prediction as a regression task. There are even some models that do just that.

The problem is that age prediction is inherently subjective and based solely on appearance.

A person in their mid-50s who has never smoked in their life, always wore sunscreen when going outside, and took care of their skin daily will likely look younger than someone in their late-30s who smokes a carton a day, works manual labor without sun protection, and doesn’t have a proper skin care regime.

And let’s not forget the most important driving factor in aging, genetics — some people simply age better than others.

For example, take a look at the following image of Matthew Perry (who played Chandler Bing on the TV sitcom, Friends) and compare it to an image of Jennifer Aniston (who played Rachel Green, alongside Perry):

Could you guess that Matthew Perry (50) is actually a year younger than Jennifer Aniston (51)?

Unless you have prior knowledge about these actors, I doubt it.

But, on the other hand, could you guess that these actors were 48-53?

I’m willing to bet you probably could.

While humans are inherently bad at predicting a single age value, we are actually quite good at predicting age brackets.

This is a loaded example, of course.

Jennifer Aniston’s genetics are near perfect, and combined with an extremely talented plastic surgeon, she seems to never age.

But that goes to show my point — people purposely try to hide their age.

And if a human struggles to accurately predict the age of a person, then surely a machine will struggle as well.

Once you start treating age prediction as a regression problem, it becomes significantly harder for a model to accurately predict a single value representing that person’s image.

However, if you treat it as a classification problem, defining buckets/age brackets for the model, our age predictor model becomes easier to train, often yielding substantially higher accuracy than regression-based prediction alone.

Simply put: Treating age prediction as classification “relaxes” the problem a bit, making it easier to solve — typically, we don’t need the exact age of a person; a rough estimate is sufficient.

Project structure

Be sure to grab the code, models, and images from the “Downloads” section of this blog post. Once you extract the files, your project will look like this:

$ tree --dirsfirst
.
├── age_detector
│   ├── age_deploy.prototxt
│   └── age_net.caffemodel
├── face_detector
│   ├── deploy.prototxt
│   └── res10_300x300_ssd_iter_140000.caffemodel
├── images
│   ├── adrian.png
│   ├── neil_patrick_harris.png
│   └── samuel_l_jackson.png
├── detect_age.py
└── detect_age_video.py

3 directories, 9 files

The first two directories consist of our age predictor and face detector. Each of these deep learning models is Caffe-based.

I’ve provided three testing images for age prediction; you can add your own images as well.

In the remainder of this tutorial, we will review two Python scripts:

• detect_age.py: Single image age prediction
• detect_age_video.py: Age prediction in video streams

Each of these scripts detects faces in an image/frame and then performs age prediction on them using OpenCV.

Implementing our OpenCV age detector for images

Let’s get started by implementing age detection with OpenCV in static images.

Open up the detect_age.py file in your project directory, and let’s get to work:

# import the necessary packages
import numpy as np
import argparse
import cv2
import os

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
	help="path to input image")
ap.add_argument("-f", "--face", required=True,
	help="path to face detector model directory")
ap.add_argument("-a", "--age", required=True,
	help="path to age detector model directory")
ap.add_argument("-c", "--confidence", type=float, default=0.5,
	help="minimum probability to filter weak detections")
args = vars(ap.parse_args())

To kick off our age detector script, we import NumPy and OpenCV. I recommend using my pip install opencv tutorial to configure your system.

Additionally, we need to import Python’s built-in os module for joining our model paths.

And finally, we import argparse to parse command line arguments.

Our script requires four command line arguments:

• --image: Provides the path to the input image for age detection
• --face: The path to our pre-trained face detector model directory
• --age: Our pre-trained age detector model directory
• --confidence: The minimum probability threshold in order to filter weak detections

As we learned above, our age detector is a classifier that predicts a person’s age using their face ROI according to predefined buckets — we aren’t treating this as a regression problem. Let’s define those age range buckets now:

# define the list of age buckets our age detector will predict
AGE_BUCKETS = ["(0-2)", "(4-6)", "(8-12)", "(15-20)", "(25-32)",
	"(38-43)", "(48-53)", "(60-100)"]

Our ages are defined in buckets (i.e., class labels) for our pre-trained age detector. We’ll use this list and an associated index to grab the age bucket to annotate on the output image.

Given our imports, command line arguments, and age buckets, we’re now ready to load our two pre-trained models:

# load our serialized face detector model from disk
print("[INFO] loading face detector model...")
prototxtPath = os.path.sep.join([args["face"], "deploy.prototxt"])
weightsPath = os.path.sep.join([args["face"],
	"res10_300x300_ssd_iter_140000.caffemodel"])
faceNet = cv2.dnn.readNet(prototxtPath, weightsPath)

# load our serialized age detector model from disk
print("[INFO] loading age detector model...")
prototxtPath = os.path.sep.join([args["age"], "age_deploy.prototxt"])
weightsPath = os.path.sep.join([args["age"], "age_net.caffemodel"])
ageNet = cv2.dnn.readNet(prototxtPath, weightsPath)

Here, we load two models:

• Our face detector finds and localizes faces in the image (Lines 25-28)
• The age classifier determines which age range a particular face belongs to (Lines 32-34)

Each of these models was trained with the Caffe framework. I cover how to train Caffe classifiers inside the PyImageSearch Gurus course.

Now that all of our initializations are taken care of, let’s load an image from disk and detect face ROIs:

# load the input image and construct an input blob for the image
image = cv2.imread(args["image"])
(h, w) = image.shape[:2]
blob = cv2.dnn.blobFromImage(image, 1.0, (300, 300),
	(104.0, 177.0, 123.0))

# pass the blob through the network and obtain the face detections
print("[INFO] computing face detections...")
faceNet.setInput(blob)
detections = faceNet.forward()

Lines 37-40 load and preprocess our input --image. We use OpenCV’s blobFromImage method — be sure to read more about blobFromImage in my tutorial.

To detect faces in our image, we send the blob through our CNN, resulting in a list of detections. Let’s loop over the face ROI detections now:

# loop over the detections
for i in range(0, detections.shape[2]):
	# extract the confidence (i.e., probability) associated with the
	# prediction
	confidence = detections[0, 0, i, 2]

	# filter out weak detections by ensuring the confidence is
	# greater than the minimum confidence
	if confidence > args["confidence"]:
		# compute the (x, y)-coordinates of the bounding box for the
		# object
		box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
		(startX, startY, endX, endY) = box.astype("int")

		# extract the ROI of the face and then construct a blob from
		# *only* the face ROI
		face = image[startY:endY, startX:endX]
		faceBlob = cv2.dnn.blobFromImage(face, 1.0, (227, 227),
			(78.4263377603, 87.7689143744, 114.895847746),
			swapRB=False)

As we loop over the detections, we filter out weak confidence faces (Lines 51-55).

For faces that meet the minimum confidence criteria, we extract the ROI coordinates (Lines 58-63). At this point, we have a small crop from the image containing only a face. We go ahead and create a blob from this ROI (i.e., faceBlob) via Lines 64-66.

And now we’ll perform age detection:

		# make predictions on the age and find the age bucket with
		# the largest corresponding probability
		ageNet.setInput(faceBlob)
		preds = ageNet.forward()
		i = preds[0].argmax()
		age = AGE_BUCKETS[i]
		ageConfidence = preds[0][i]

		# display the predicted age to our terminal
		text = "{}: {:.2f}%".format(age, ageConfidence * 100)
		print("[INFO] {}".format(text))

		# draw the bounding box of the face along with the associated
		# predicted age
		y = startY - 10 if startY - 10 > 10 else startY + 10
		cv2.rectangle(image, (startX, startY), (endX, endY),
			(0, 0, 255), 2)
		cv2.putText(image, text, (startX, y),
			cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 0, 255), 2)

# display the output image
cv2.imshow("Image", image)
cv2.waitKey(0)

Using our face blob, we make age predictions (Lines 70-74) resulting in an age bucket and ageConfidence. We use these data points along with the coordinates of the face ROI to annotate the original input --image (Lines 77-86) and display results (Lines 89 and 90).

In the next section, we’ll analyze our results.

OpenCV age detection results

Let’s put our OpenCV age detector to work.

Start by using the “Downloads” section of this tutorial to download the source code, pre-trained age detector model, and example images.

From there, open up a terminal, and execute the following command:

$ python detect_age.py --image images/adrian.png --face face_detector --age age_detector
[INFO] loading face detector model...
[INFO] loading age detector model...
[INFO] computing face detections...
[INFO] (25-32): 57.51%

Here, you can see that our OpenCV age detector has predicted my age to be 25-32 with 57.51% confidence — indeed, the age detector is correct (I was 30 when that picture was taken).

Let’s try another example, this one of the famous actor, Neil Patrick Harris when he was a kid:

$ python detect_age.py --image images/neil_patrick_harris.png --face face_detector --age age_detector
[INFO] loading face detector model...
[INFO] loading age detector model...
[INFO] computing face detections...
[INFO] (8-12): 85.72%

Our age predictor is one again correct — Neil Patrick Harris certainly looked to be somewhere in the 8-12 age group when this photo was taken.

Let’s try another image; this image is of one of my favorite actors, the infamous Samuel L. Jackson:

$ python detect_age.py --image images/samuel_l_jackson.png --face face_detector --age age_detector
[INFO] loading face detector model...
[INFO] loading age detector model...
[INFO] computing face detections...
[INFO] (48-53): 69.38%

Here our OpenCV age detector is incorrect — Samuel L. Jackson is ~71 years old, making our age prediction off by approximately 18 years.

That said, look at the photo — does Mr. Jackson actually look to be 71?

My guess would have been late 50s to early 60s. At least to me, he certainly doesn’t look like a man in his early 70s.

But that just goes to show my point earlier in this post:

The process of visual age prediction is difficult, and I’d consider it subjective when either a computer or a person tries to guess someone’s age.

In order to evaluate an age detector, you cannot rely on the person’s actual age. Instead, you need to measure the accuracy between the predicted age and the perceived age.

Implementing our OpenCV age detector for real-time video streams

At this point, we can perform age detection in static images, but what about real-time video streams?

Can we do that as well?

You bet we can. Our video script very closely aligns with our image script. The difference is that we need to set up a video stream and perform age detection on each and every frame in a loop. This review will focus on the video features, so be sure to refer to the walkthrough above as needed.

To see how to perform age recognition in video, let’s take a look at detect_age_video.py.

# import the necessary packages
from imutils.video import VideoStream
import numpy as np
import argparse
import imutils
import time
import cv2
import os

We have three new imports: (1) VideoStream, (2) imutils, and (3) time. Each of these imports allow us to set up and use our webcam for our video stream.

I’ve decided to define a convenience function for accepting a frame, localizing faces, and predicting ages. By putting the detect and predict logic here, our frame processing loop will be less bloated (you could also offload this function to a separate file). Let’s dive into this utility now:

def detect_and_predict_age(frame, faceNet, ageNet, minConf=0.5):
	# define the list of age buckets our age detector will predict
	AGE_BUCKETS = ["(0-2)", "(4-6)", "(8-12)", "(15-20)", "(25-32)",
		"(38-43)", "(48-53)", "(60-100)"]

	# initialize our results list
	results = []

	# grab the dimensions of the frame and then construct a blob
	# from it
	(h, w) = frame.shape[:2]
	blob = cv2.dnn.blobFromImage(frame, 1.0, (300, 300),
		(104.0, 177.0, 123.0))

	# pass the blob through the network and obtain the face detections
	faceNet.setInput(blob)
	detections = faceNet.forward()

Our detect_and_predict_age helper function accepts the following parameters:

• frame: A single frame from your webcam video stream
• faceNet: The initialized deep learning face detector
• ageNet: Our initialized deep learning age classifier
• minConf: The confidence threshold to filter weak face detections

These parameters draw parallels from the command line arguments of our single image age detector script.

Again, our AGE_BUCKETS are defined (Lines 12 and 13).

We then initialize an empty list to hold the results of face localization and age detection.

Lines 20-26 handle performing face detection.

Next, we’ll process each of the detections:

	# loop over the detections
	for i in range(0, detections.shape[2]):
		# extract the confidence (i.e., probability) associated with
		# the prediction
		confidence = detections[0, 0, i, 2]

		# filter out weak detections by ensuring the confidence is
		# greater than the minimum confidence
		if confidence > minConf:
			# compute the (x, y)-coordinates of the bounding box for
			# the object
			box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
			(startX, startY, endX, endY) = box.astype("int")

			# extract the ROI of the face
			face = frame[startY:endY, startX:endX]

			# ensure the face ROI is sufficiently large
			if face.shape[0] < 20 or face.shape[1] < 20:
				continue

You should recognize Lines 29-43 — they loop over detections, ensure high confidence, and extract a face ROI.

Lines 46 and 47 are new — they ensure that a face ROI is sufficiently large in our stream for two reasons:

• First, we want to filter out false-positive face detections in the frame.
• Second, age classification results won’t be accurate for faces that are far away from the camera (i.e., perceivably small).

To finish out our helper utility, we’ll perform age recognition and return our results:

			# construct a blob from *just* the face ROI
			faceBlob = cv2.dnn.blobFromImage(face, 1.0, (227, 227),
				(78.4263377603, 87.7689143744, 114.895847746),
				swapRB=False)

			# make predictions on the age and find the age bucket with
			# the largest corresponding probability
			ageNet.setInput(faceBlob)
			preds = ageNet.forward()
			i = preds[0].argmax()
			age = AGE_BUCKETS[i]
			ageConfidence = preds[0][i]

			# construct a dictionary consisting of both the face
			# bounding box location along with the age prediction,
			# then update our results list
			d = {
				"loc": (startX, startY, endX, endY),
				"age": (age, ageConfidence)
			}
			results.append(d)

	# return our results to the calling function
	return results

Here, we predict the age of the face and extract the age bucket and ageConfidence (Lines 56-60).

Lines 65-68 arrange face localization and predicted age in a dictionary. The last step of the detection processing loop is to add the dictionary to the results list (Line 69).

Once all detections have been processed and any results are ready, we return the results to the caller.

With our helper function defined, now we can get back to working with our video stream. But first, we need to define command line arguments:

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-f", "--face", required=True,
	help="path to face detector model directory")
ap.add_argument("-a", "--age", required=True,
	help="path to age detector model directory")
ap.add_argument("-c", "--confidence", type=float, default=0.5,
	help="minimum probability to filter weak detections")
args = vars(ap.parse_args())

Our script requires three command line arguments:

• --face: The path to our pre-trained face detector model directory
• --age: Our pre-trained age detector model directory
• --confidence: The minimum probability threshold in order to filter weak detections

From here, we’ll load our models and initialize our video stream:

# load our serialized face detector model from disk
print("[INFO] loading face detector model...")
prototxtPath = os.path.sep.join([args["face"], "deploy.prototxt"])
weightsPath = os.path.sep.join([args["face"],
	"res10_300x300_ssd_iter_140000.caffemodel"])
faceNet = cv2.dnn.readNet(prototxtPath, weightsPath)

# load our serialized age detector model from disk
print("[INFO] loading age detector model...")
prototxtPath = os.path.sep.join([args["age"], "age_deploy.prototxt"])
weightsPath = os.path.sep.join([args["age"], "age_net.caffemodel"])
ageNet = cv2.dnn.readNet(prototxtPath, weightsPath)

# initialize the video stream and allow the camera sensor to warm up
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
time.sleep(2.0)

Lines 86-89 load and initialize our face detector, while Lines 93-95 load our age detector.

We then use the VideoStream class to initialize our webcam (Lines 99 and 100).

Once our webcam is warmed up, we’ll begin processing frames:

# loop over the frames from the video stream
while True:
	# grab the frame from the threaded video stream and resize it
	# to have a maximum width of 400 pixels
	frame = vs.read()
	frame = imutils.resize(frame, width=400)

	# detect faces in the frame, and for each face in the frame,
	# predict the age
	results = detect_and_predict_age(frame, faceNet, ageNet,
		minConf=args["confidence"])

	# loop over the results
	for r in results:
		# draw the bounding box of the face along with the associated
		# predicted age
		text = "{}: {:.2f}%".format(r["age"][0], r["age"][1] * 100)
		(startX, startY, endX, endY) = r["loc"]
		y = startY - 10 if startY - 10 > 10 else startY + 10
		cv2.rectangle(frame, (startX, startY), (endX, endY),
			(0, 0, 255), 2)
		cv2.putText(frame, text, (startX, y),
			cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 0, 255), 2)

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the `q` key was pressed, break from the loop
	if key == ord("q"):
		break

# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()

Inside our loop, we:

• Grab the next frame, and resize it to a known width (Lines 106 and 107)
• Send the frame through our detect_and_predict_age convenience function to (1) detect faces and (2) determine ages (Lines 111 and 112)
• Annotate the results on the frame (Lines 115-124)
• Display and capture keypresses (Lines 127 and 128)
• Exit and clean up if the q key was pressed (Lines 131-136)

In the next section, we’ll fire up our age detector and see if it works!

Real-time age detection with OpenCV results

Let’s now apply age detection with OpenCV to real-time video stream.

Make sure you’ve used the “Downloads” section of this tutorial to download the source code and pre-trained age detector.

From there, open up a terminal, and issue the following command:

$ python detect_age_video.py --face face_detector --age age_detector
[INFO] loading face detector model...
[INFO] loading age detector model...
[INFO] starting video stream...

Here, you can see that our OpenCV age detector is accurately predicting my age range as 25-32 (I am currently 31 at the time of this writing).

How can I improve age prediction results?

One of the biggest issues with the age prediction model trained by Levi and Hassner is that it’s heavily biased toward the age group 25-32, as shown by the following confusion matrix table from their original publication:

That unfortunately means that our model may predict the 25-32 age group when in fact the actual age belongs to a different age bracket — I noticed this a handful of times when gathering results for this tutorial as well as in my own applications of age prediction.

You can combat this bias by:

1. Gathering additional training data for the other age groups to help balance out the dataset
2. Applying class weighting to handle class imbalance
3. Being more aggressive with data augmentation
4. Implementing additional regularization when training the model

Secondly, age prediction results can typically be improved by using face alignment.

Face alignment identifies the geometric structure of faces and then attempts to obtain a canonical alignment of the face based on translation, scale, and rotation.

In many cases (but not always), face alignment can improve face application results, including face recognition, age prediction, etc.

As a matter of simplicity, we did not apply face alignment in this tutorial, but you can follow this tutorial to learn more about face alignment and then apply it to your own age prediction applications.

What about gender prediction?

I have chosen to purposely not cover gender prediction in this tutorial.

While using computer vision and deep learning to identify the gender of a person may seem like an interesting classification problem, it’s actually one wrought with moral implications.

Just because someone visually looks, dresses, or appears a certain way does not imply they identify with that (or any) gender.

Software that attempts to distill gender into binary classification only further chains us to antiquated notions of what gender is. Therefore, I would encourage you to not utilize gender recognition in your own applications if at all possible.

If you must perform gender recognition, make sure you are holding yourself accountable, and ensure you are not building applications that attempt to conform others to gender stereotypes (e.g., customizing user experiences based on perceived gender).

There is little value in gender recognition, and it truly just causes more problems than it solves. Try to avoid it if at all possible.

Do you want to train your own deep learning models?

In the blog post, I showed you how to use a pre-trained age detector — if you instead want to learn how to train the age detector from scratch, I would recommend you check out my book, Deep Learning for Computer Vision with Python.

Inside the book, you’ll find:

• Super-practical walkthroughs that present solutions to actual real-world image classification (ResNet, VGG, etc.), object detection (Faster R-CNN, SSDs, RetinaNet, etc.), and segmentation (Mask R-CNN) problems
• Hands-on tutorials (with lots of code) that show you not only the algorithms behind deep learning for computer vision but their implementations as well
• A no-nonsense teaching style that is guaranteed to help you master deep learning for image understanding and visual recognition

If you’re interested in learning more about the book, I’d be happy to send you a PDF containing the Table of Contents and a few sample chapters:

Summary

In this tutorial, you learned how to perform age detection with OpenCV and Deep Learning.

To do so, we utilized a pre-trained model from Levi and Hassner in their 2015 publication, Age and Gender Classification using Convolutional Neural Networks. This model allowed us to predict eight different age groups with reasonably high accuracy; however, we must recognize that age prediction is a challenging problem.

There are a number of factors that determine how old a person visually appears, including their lifestyle, work/job, smoking habits, and most importantly, genetics. Secondly, keep in mind that people purposely try to hide their age — if a human struggles to accurately predict someone’s age, then surely a machine learning model will struggle as well.

Therefore, you must assess all age prediction results in terms of perceived age rather than actual age. Keep this in mind when implementing age detection into your own computer vision projects.

I hope you enjoyed this tutorial!

To download the source code to this post (including the pre-trained age detector model), just enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post OpenCV Age Detection with Deep Learning appeared first on PyImageSearch.

In this tutorial, you will learn how to detect and remove duplicate images from a dataset for deep learning.

Over the past few weeks, I’ve been working on a project with Victor Gevers, the esteemed ethical hacker from the GDI.Foundation, an organization that is famous for responsibly disclosing data leaks and reporting security vulnerabilities.

I can’t go into details of the project (yet), but one of the tasks required me to train a custom deep neural network to detect specific patterns in images.

The dataset I was using was crafted by combining images from multiple sources. I knew there were going to be duplicate images in the dataset — I therefore needed a method to detect and remove these duplicate images from my dataset.

As I was working on the project, I just so happened to receive an email from Dahlia, a university student who also had a question on image duplicates and how to handle them:

Hi Adrian, my name is Dahlia. I’m an undergraduate working on my final year graduation project and have been tasked with building an image dataset by scraping Google Images, Bing, etc. and then training a deep neural network on the dataset.

My professor told me to be careful when building the dataset, stating that I needed to remove duplicate images.

That caused me some doubts: 

Why are duplicate images in a dataset a problem? Secondly, how do I detect the image duplicates? 

Trying to do so manually sounds like an error-prone process. I don’t want to make any mistakes. 

Is there a way that I can automatically detect and remove the duplicates from my image dataset? 

Thank you.

Dahlia asks some great questions.

Having duplicate images in your dataset creates a problem for two reasons:

1. It introduces bias into your dataset, giving your deep neural network additional opportunities to learn patterns specific to the duplicates
2. It hurts the ability of your model to generalize to new images outside of what it was trained on

While we often assume that data points in a dataset are independent and identically distributed, that’s rarely (if ever) the case when working with a real-world dataset. When training a Convolutional Neural Network, we typically want to remove those duplicate images before training the model.

Secondly, trying to manually detect duplicate images in a dataset is extremely time-consuming and error-prone — it also doesn’t scale to large image datasets. We therefore need a method to automatically detect and remove duplicate images from our deep learning dataset.

Is such a method possible?

It certainly is — and I’ll be covering it in the remainder of today’s tutorial.

To learn how to detect and remove duplicate images from your deep learning dataset, just keep reading!

Looking for the source code to this post?

Jump Right To The Downloads Section

Detect and remove duplicate images from a dataset for deep learning

In the first part of this tutorial, you’ll learn why detecting and removing duplicate images from your dataset is typically a requirement before you attempt to train a deep neural network on top of your data.

From there, we’ll review the example dataset I created so we can practice detecting duplicate images in a dataset.

We’ll then implement our image duplicate detector using a method called image hashing.

Finally, we’ll review the results of our work and:

1. Perform a dry run to validate that our image duplicate detector is working properly
2. Run our duplicate detector a second time, this time removing the actual duplicates from our dataset

Why bother removing duplicate images from a dataset when training a deep neural network?

If you’ve ever attempted to build your own image dataset by hand, you know it’s a likely possibility (if not an inevitability) that you’ll have duplicate images in your dataset.

Typically, you end up with duplicate images in your dataset by:

1. Scraping images from multiple sources (e.g., Google, Bing, etc.)
2. Combining existing datasets (ex., combining ImageNet with Sun397 and Indoor Scenes)

When this happens you need a way to:

1. Detect that there are duplicate images in your dataset
2. Remove the duplicates

But that raises the question — why bother caring about duplicates in the first place?

The usual assumption for supervised machine learning methods is that:

1. Data points are independent
2. They are identically distributed
3. Training and testing data are sampled from the same distribution

The problem is that these assumptions rarely (if ever) hold in practice.

What you really need to be afraid of is your model’s ability to generalize.

If you include multiple identical images in your dataset, your neural network is allowed to see and learn patterns from that image multiple times per epoch.

Your network could become biased toward patterns in those duplicate images, making it less likely to generalize to new images.

Bias and ability to generalize are a big deal in machine learning — they can be hard enough to combat when working with an “ideal” dataset.

Take the time to remove duplicates from your image dataset so you don’t accidentally introduce bias or hurt the ability of your model to generalize.

Our example duplicate-images dataset

To help us learn how to detect and remove duplicate images from a deep learning dataset, I created a “practice” dataset we can use based on the Stanford Dogs Dataset.

This dataset consists of 20,580 images of dog breeds, including Beagles, Newfoundlands, and Pomeranians, just to name a few.

To create our duplicate image dataset, I:

1. Downloaded the Stanford Dogs Dataset
2. Sampled three images that I would duplicate
3. Duplicated each of these three images a total of N times
4. Then randomly sampled the Stanford Dogs Dataset further until I obtained 1,000 total images

The following figure shows the number of duplicates per image:

Our goal is to create a Python script that can detect and remove these duplicates prior to training a deep learning model.

Project structure

I’ve included the duplicate image dataset along with the code in the “Downloads” section of this tutorial.

Once you extract the .zip, you’ll be presented with the following directory structure:

$ tree --dirsfirst --filelimit 10
.
├── dataset [1000 entries]
└── detect_and_remove.py

1 directory, 1 file

As you can see, our project structure is quite simple. We have a dataset/ of 1,000 images (duplicates included). Additionally, we have our detect_and_remove.py Python script, which is the basis of today’s tutorial.

Implementing our image duplicate detector

We are now ready to implement our image duplicate detector.

Open up the detect_and_remove.py script in your project directory, and let’s get to work:

# import the necessary packages
from imutils import paths
import numpy as np
import argparse
import cv2
import os

Imports for our script include my paths implementation from imutils so we can grab the filepaths to all images in our dataset, NumPy for image stacking, and OpenCV for image I/O, manipulation, and display. Both os and argparse are built-in to Python.

If you do not have OpenCV or imutils installed on your machine, I recommend following my pip install opencv guide which will show you how to install both.

The primary component of this tutorial is the dhash function:

def dhash(image, hashSize=8):
	# convert the image to grayscale and resize the grayscale image,
	# adding a single column (width) so we can compute the horizontal
	# gradient
	gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
	resized = cv2.resize(gray, (hashSize + 1, hashSize))

	# compute the (relative) horizontal gradient between adjacent
	# column pixels
	diff = resized[:, 1:] > resized[:, :-1]

	# convert the difference image to a hash and return it
	return sum([2 ** i for (i, v) in enumerate(diff.flatten()) if v])

• Convert the image to a single-channel grayscale image (Line 12)
• Resize the image according to the hashSize (Line 13). The algorithm requires that the width of the image have exactly 1 more column than the height as is evident by the dimension tuple.
• Compute the relative horizontal gradient between adjacent column pixels (Line 17). This is now known as the “difference image.”
• Apply our hashing calculation and return the result (Line 20).

I’ve covered image hashing in these previous articles. In particular, you should read my Image hashing with OpenCV and Python guide to understand the concept of image hashing using my dhash function.

With our hashing function defined, we’re now ready to define and parse command line arguments:

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d", "--dataset", required=True,
	help="path to input dataset")
ap.add_argument("-r", "--remove", type=int, default=-1,
	help="whether or not duplicates should be removed (i.e., dry run)")
args = vars(ap.parse_args())

Our script handles two command line arguments, which you can pass via your terminal or command prompt:

• --dataset: The path to your input dataset, which contains duplicates that you’d like to prune out of the dataset
• --remove: Indicates whether duplicates should be removed (deleted permanently) or whether you want to conduct a “dry run” so you can visualize the duplicates on your screen and see the hashes in your terminal

We’re now ready to begin computing hashes:

# grab the paths to all images in our input dataset directory and
# then initialize our hashes dictionary
print("[INFO] computing image hashes...")
imagePaths = list(paths.list_images(args["dataset"]))
hashes = {}

# loop over our image paths
for imagePath in imagePaths:
	# load the input image and compute the hash
	image = cv2.imread(imagePath)
	h = dhash(image)

	# grab all image paths with that hash, add the current image
	# path to it, and store the list back in the hashes dictionary
	p = hashes.get(h, [])
	p.append(imagePath)
	hashes[h] = p

• Load an image (Line 39)
• Compute the hash, h, using the dhash convenience function (Line 40)
• Grab all image paths, p, with the same hash, h (Line 44).
• Append the latest imagePath to p (Line 45). At this point, p represents our set of duplicate images (i.e., images with the same hash value)
• Add all of these duplicates to our hashes dictionary (Line 46)

{
	...
	7054210665732718398: ['dataset/00000005.jpg', 'dataset/00000071.jpg', 'dataset/00000869.jpg'],
	8687501631902372966: ['dataset/00000011.jpg'],
	1321903443018050217: ['dataset/00000777.jpg'],
	...
}

Notice how the first hash key example has three associated image paths (indicating duplicates) and the next two hash keys have only one path entry (indicating no duplicates).

At this point, with all of our hashes computed, we need to loop over the hashes and handle the duplicates:

# loop over the image hashes
for (h, hashedPaths) in hashes.items():
	# check to see if there is more than one image with the same hash
	if len(hashedPaths) > 1:
		# check to see if this is a dry run
		if args["remove"] <= 0:
			# initialize a montage to store all images with the same
			# hash
			montage = None

			# loop over all image paths with the same hash
			for p in hashedPaths:
				# load the input image and resize it to a fixed width
				# and heightG
				image = cv2.imread(p)
				image = cv2.resize(image, (150, 150))

				# if our montage is None, initialize it
				if montage is None:
					montage = image

				# otherwise, horizontally stack the images
				else:
					montage = np.hstack([montage, image])

			# show the montage for the hash
			print("[INFO] hash: {}".format(h))
			cv2.imshow("Montage", montage)
			cv2.waitKey(0)

If not, we ignore the hash and continue to check the next hash.

On the other hand, if there are, in fact, two or more hashedPaths, they are duplicates!

Therefore, we start an if/ else block to check whether this is a “dry run” or not; if the --remove flag is not a positive value, we are conducting a dry run (Line 53).

A dry run means that we aren’t ready to delete duplicates yet. Rather, we just want to check to see if any duplicates are present.

# otherwise, we'll be removing the duplicate images
		else:
			# loop over all image paths with the same hash *except*
			# for the first image in the list (since we want to keep
			# one, and only one, of the duplicate images)
			for p in hashedPaths[1:]:
				os.remove(p)

In this case, we are actually deleting the duplicate images from our system.

In this scenario, we simply loop over all image paths with the same hash except for the first image in the list — we want to keep one, and only one, of the example images and delete all other identical images.

Great job implementing your very own duplicate image detection and removal system.

Running our image duplicate detector for our deep learning dataset

Let’s put our image duplicate detector to work.

Start by making sure you have used the “Downloads” section of this tutorial to download the source code and example dataset.

From there, open up a terminal, and execute the following command just to verify there are 1,000 images in our dataset/ directory:

$ ls -l dataset/*.jpg | wc -l
    1000

Let’s now perform a dry run, which will allow us to visualize the duplicates in our dataset:

$ python detect_and_remove.py --dataset dataset
[INFO] computing image hashes...
[INFO] hash: 7054210665732718398
[INFO] hash: 15443501585133582635
[INFO] hash: 13344784005636363614

The following figure shows the output of our script, demonstrating that we have been able to successfully find the duplicates as detailed in “Our example duplicate image dataset” section above.

To actually remove the duplicates from our system, we need to execute detect_and_remove.py again, this time supplying the --remove 1 command line argument:

$ python detect_and_remove.py --dataset dataset --remove 1
[INFO] computing image hashes...

We can verify that the duplicates have been removed by counting the number of JPEG Images in the dataset directory:

$ ls -l dataset/*.jpg | wc -l
     993

Originally, there were 1,000 images in dataset, but now there are 993, implying that we removed the 7 duplicate images.

At this point, you could proceed to train a deep neural network on this dataset.

How do I create my own image dataset?

I’ve created a sample dataset for today’s tutorial — it is included with the “Downloads” so that you can begin learning the concept of deduplication immediately.

However, you may be wondering:

“How do I create a dataset in the first place?”

There isn’t a “one size fits all” approach for creating a dataset. Rather, you need to consider the problem and design your data collection around it.

You may determine that you need automation and a camera to collect data. Or you may determine that you need to combine existing datasets to save a lot of time.

Let’s first consider datasets for the purpose of face applications. If you’d like to create a custom face dataset, you can use any of three methods:

1. Enrolling faces via OpenCV and a webcam
2. Downloading face images programmatically
3. Manually collecting face images

From there, you can apply face applications, including facial recognition, facial landmarks, etc.

But what if you want to harness the power of the internet and an existing search engine or scraping tool? Is there hope?

In fact there is. I have written three tutorials to help you get started.

1. How to create a deep learning dataset using Google Images
2. How to (quickly) build a deep learning image dataset (using Bing)
3. Scraping images with Python and Scrapy

Use these blog posts to help create your datasets, keeping in mind the copyrights of the image owners. As a general rule, you should only use copyrighted images for educational purposes. For commercial purposes, you need to contact the owner of each image for permission.

Collecting images online nearly always results in duplicates — be sure to do a quick inspection. After you have created your dataset, follow today’s deduplication tutorial to compute hashes and prune the duplicates automatically.

Recall the two important reasons for pruning duplicates from your machine learning and deep learning datasets:

1. Duplicate images in your dataset introduce bias into your dataset, giving your deep neural network additional opportunities to learn patterns specific to the duplicates.
2. Additionally, duplicates impact the ability of your model to generalize to new images outside of what it was trained on.

From there, you can train your very own deep learning model on your newly formed dataset and deploy it!

What’s next?

In this tutorial, you learned how to deduplicate images from your dataset. The next step is to train a Convolutional Neural Network on your dataset.

Training your own deep neural networks can be challenging if you’re new to machine learning or Python.

I’ll be honest — It wasn’t easy for me either when I first started, even with years of machine learning research and teaching under my belt.

But it doesn’t have to be like that for you.

Rather than juggling issues with deep learning APIs, searching in places like StackOverflow and GitHub Issues, and begging for help on AI and DL Facebook groups, why not read the best, most comprehensive deep learning book?

Don’t go on a wild goose chase searching for answers online to your academic, work, or hobby deep learning projects. Instead, pick up a copy of the text, and find answers and proven code recipes to:

• Create and prepare your own custom image datasets for image classification, object detection, and segmentation
• Understand how popular network architectures work, including ResNet, Inception, Faster R-CNN, Single Shot Detectors (SSD), RetinaNet, and Mask R-CNN
• Train these architectures on your own custom datasets
• Learn my tips, suggestions, and best practices to ensure you maximize the accuracy of your models

Okay, I’ll admit — I’m a bit biased, since I wrote Deep Learning for Computer Vision with Python, but if you visit PyImageSearch tutorials often, then you know that the quality of my content speaks for itself.

You don’t have to take my word for it either. Take a look at success stories of PyImageSearch students, where my books and courses are helping students in their careers as developers or CV/DL practitioners, allowing them to land high-paying jobs, publish research papers, and win academic research grants.

If you’re interested in learning more about my deep learning book, I’d be happy to send you a free PDF containing the Table of Contents and a few sample chapters:

Grab my free deep learning PDF!

Summary

In this tutorial, you learned how to detect and remove duplicate images from a deep learning dataset.

Typically, you’ll want to remove duplicate images from your dataset to ensure each data point (i.e., image) is represented only a single time — if there are multiple identical images in your dataset, your convolutional neural network may learn to be biased toward the images, making it less likely for your model to generalize to new images.

To help prevent this type of bias, we implemented our duplicate detector using a method called image hashing.

Image hashing works by:

1. Examining the contents of an image
2. Constructing a hash value (i.e., an integer) that uniquely quantifies an input image based on the contents of the image alone

Using our hashing function we then:

1. Looped over all images in our image dataset
2. Computed an image hash for each image
3. Checked for “hash collisions,” implying that if two images had the same hash, they must be duplicates
4. Removed duplicate images from our dataset

Using the technique covered in this tutorial, you can detect and remove duplicate images from your own datasets — from there, you’ll be able to train a deep neural network on top of your newly deduplicated dataset!

I hope you enjoyed this tutorial.

To download the source code and example dataset for this post (and be notified when future tutorials are published here on PyImageSearch), just enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Detect and remove duplicate images from a dataset for deep learning appeared first on PyImageSearch.

In this tutorial, you will learn how to fine-tune ResNet using Keras, TensorFlow, and Deep Learning.

A couple of months ago, I posted on Twitter asking my followers for help creating a dataset of camouflage vs. noncamouflage clothes:

This dataset was to be used on a special project that Victor Gevers, an esteemed ethical hacker from the GDI.Foundation, and I were working on (more on that in two weeks, when I’ll reveal the details on what we’ve built).

Two PyImageSearch readers, Julia Riede and Nitin Rai, not only stepped up to the plate to help out but hit a home run!

Both of them spent a couple of days downloading images for each class, organizing the files, and then uploading them so Victor and I could train a model on them — thank you so much, Julia and Nitin; we couldn’t have done it without you!

A few days after I started working with the camouflage vs. noncamouflage dataset, I received an email from PyImageSearch reader Lucas:

Hi Adrian, I’m big fan of the PyImageSearch blog. It’s helped me tremendously with my undergrad project.

I have a question for you:

Do you have any tutorials on how to fine-tune ResNet?

I’ve been going through your archives and it seems like you’ve covered fine-tuning other architectures (ex. VGGNet) but I couldn’t find anything on ResNet. I’ve been trying to fine-tune ResNet with Keras/TensorFlow for the past few days and I just keep running into errors.

If you can help me out I would appreciate it.

I was already planning on fine-tuning a model on top of the camouflage vs. noncamouflage clothes dataset, so helping Lucas seemed like a natural fit.

Inside the remainder of this this tutorial you will:

1. Discover the seminal ResNet architecture
2. Learn how to fine-tune it using Keras and TensorFlow
3. Fine-tune ResNet for camouflage vs. noncamouflage clothes detection

And in two weeks, I’ll show you the practical, real-world use case that Victor and I applied camouflage detection to — it’s a great story, and you won’t want to miss it!

To learn how to fine-tune ResNet with Keras and TensorFlow, just keep reading!

Looking for the source code to this post?

Jump Right To The Downloads Section

Fine-tuning ResNet with Keras, TensorFlow, and Deep Learning

In the first part of this tutorial, you will learn about the ResNet architecture, including how we can fine-tune ResNet using Keras and TensorFlow.

From there, we’ll discuss our camouflage clothing vs. normal clothing image dataset in detail.

We’ll then review our project directory structure and proceed to:

1. Implement our configuration file
2. Create a Python script to build/organize our image dataset
3. Implement a second Python script used to fine-tune ResNet with Keras and TensorFlow
4. Execute the training script and fine-tune ResNet on our dataset

Let’s get started!

What is ResNet?

ResNet was first introduced by He et al. in their seminal 2015 paper, Deep Residual Learning for Image Recognition — that paper has been cited an astonishing 43,064 times!

A follow-up paper in 2016, Identity Mappings in Deep Residual Networks, performed a series of ablation experiments, playing with the inclusion, removal, and ordering of various components in the residual module, ultimately resulting in a variation of ResNet that:

1. Is easier to train
2. Is more tolerant of hyperparameters, including regularization and initial learning rate
3. Generalizes better

ResNet is arguably the most important network architecture since:

• AlexNet — which reignited researcher interest in deep neural networks back in 2012
• VGGNet — which demonstrated how deeper neural networks could be trained successfully using only 3×3 convolutions (2014)
• GoogLeNet — which introduced the inception module/micro-architecture (2014)

In fact, the techniques that ResNet employs have been successfully applied to noncomputer vision tasks, including audio classification and Natural Language Processing (NLP)!

How does ResNet work?

Note: The following section was adapted from Chapter 12 of my book, Deep Learning for Computer Vision with Python (Practitioner Bundle).

The original residual module introduced by He et al. relies on the concept of identify mappings, the process of taking the original input to the module and adding it to the output of a series of operations:

At the top of the module, we accept an input to the module (i.e., the previous layer in the network). The right branch is a “linear shortcut” — it connects the input to an addition operation at the bottom of the model. Then, on the left branch of the residual module, we apply a series of convolutions (both of which are 3×3), activations, and batch normalizations. This is a standard pattern to follow when constructing Convolutional Neural Networks.

But what makes ResNet interesting is that He et al. suggested adding the original input to the output of the CONV, RELU, and BN layers.

We call this addition an identity mapping since the input (the identity) is added to the output of a series of operations.

It’s also way the term residual is used — the “residual” input is added to the output of a series of layer operations. The connection between the input and addition node is called the shortcut.

While traditional neural networks can be seen as learning a function y = f(x), a residual layer attempts to approximate y via f(x) + id(x) = f(x) + x where id(x) is the identity function.

These residual layers start at the identity function and evolve to become more complex as the network learns. This type of residual learning framework allows us to train networks that are substantially deeper than previously proposed architectures.

Furthermore, since the input is included in every residual module, it turns out the network can learn faster and with larger learning rates.

In the original 2015 paper, He et al. also included an extension to the original residual module called bottlenecks:

Here we can see that the same identity mapping is taking place, only now the CONV layers in the left branch of the residual module have been updated:

1. We are utilizing three CONV layers rather than just two
2. The first and last CONV layers are 1×1 convolutions
3. The number of filters learned in the first two CONV layers are 1/4 the number of filters learned in the final CONV

This variation of the residual module serves as a form of dimensionality reduction, thereby reducing the total number of parameters in the network (and doing so without sacrificing accuracy). This form of dimensionality reduction is called the bottleneck.

He et al.’s 2016 publication on Identity Mappings in Deep Residual Networks performed a series of ablation studies, playing with the inclusion, removal, and ordering of various components in the residual module, ultimately resulting in the concept of pre-activation:

Without going into too much detail, the pre-activation residual module rearranges the order in which convolution, batch normalization, and activation are performed.

The original residual module (with bottleneck) accepts an input (i.e., a RELU activation map) and then applies a series of (CONV => BN => RELU) * 2 => CONV => BN before adding this output to the original input and applying a final RELU activation.

Their 2016 study demonstrated that instead, applying a series of (BN => RELU => CONV) * 3 led to higher accuracy models that were easier to train.

We call this method of layer ordering pre-activation as our RELUs and batch normalizations are placed before the convolutions, which is in contrast to the typical approach of applying RELUs and batch normalizations after the convolutions.

For a more complete review of ResNet, including how to implement it from scratch using Keras/TensorFlow, be sure to refer to my book, Deep Learning for Computer Vision with Python.

How can we fine-tune it with Keras and TensorFlow?

In order to fine-tune ResNet with Keras and TensorFlow, we need to load ResNet from disk using the pre-trained ImageNet weights but leaving off the fully-connected layer head.

We can do so using the following code:

>>> baseModel = ResNet50(weights="imagenet", include_top=False,
	input_tensor=Input(shape=(224, 224, 3)))

Inspecting the baseModel.summary(), you’ll see the following:

...
conv5_block3_3_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block3_3_bn (BatchNormali (None, 7, 7, 2048)   8192        conv5_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_add (Add)          (None, 7, 7, 2048)   0           conv5_block2_out[0][0]           
                                                                 conv5_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_out (Activation)   (None, 7, 7, 2048)   0           conv5_block3_add[0][0]           
==================================================================================================

Here, we can observe that the final layer in the ResNet architecture (again, without the fully-connected layer head) is an Activation layer that is 7 x 7 x 2048.

We can construct a new, freshly initialized layer head by accepting the baseModel.output and then applying a 7×7 average pooling, followed by our fully-connected layers:

headModel = baseModel.output
headModel = AveragePooling2D(pool_size=(7, 7))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(256, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(len(config.CLASSES), activation="softmax")(headModel)

With the headModel constructed, we simply need to append it to the body of the ResNet model:

model = Model(inputs=baseModel.input, outputs=headModel)

Now, if we take a look at the model.summary(), we can conclude that we have successfully added a new fully-connected layer head to ResNet, making the architecture suitable for fine-tuning:

conv5_block3_3_conv (Conv2D)    (None, 7, 7, 2048)   1050624     conv5_block3_2_relu[0][0]        
__________________________________________________________________________________________________
conv5_block3_3_bn (BatchNormali (None, 7, 7, 2048)   8192        conv5_block3_3_conv[0][0]        
__________________________________________________________________________________________________
conv5_block3_add (Add)          (None, 7, 7, 2048)   0           conv5_block2_out[0][0]           
                                                                 conv5_block3_3_bn[0][0]          
__________________________________________________________________________________________________
conv5_block3_out (Activation)   (None, 7, 7, 2048)   0           conv5_block3_add[0][0]           
__________________________________________________________________________________________________
average_pooling2d (AveragePooli (None, 1, 1, 2048)   0           conv5_block3_out[0][0]           
__________________________________________________________________________________________________
flatten (Flatten)               (None, 2048)         0           average_pooling2d[0][0]          
__________________________________________________________________________________________________
dense (Dense)                   (None, 256)          524544      flatten[0][0]                    
__________________________________________________________________________________________________
dropout (Dropout)               (None, 256)          0           dense[0][0]                      
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 2)            514         dropout[0][0]                    
==================================================================================================

In the remainder of this tutorial, I will provide you with a fully working example of fine-tuning ResNet using Keras and TensorFlow.

Our camouflage vs. normal clothing dataset

In this tutorial, we will be training a camouflage clothes vs. normal clothes detector.

I’ll be discussing exactly why we’re building a camouflage clothes detector in two weeks, but for the time being, let this serve as a standalone example of how to fine-tune ResNet with Keras and TensorFlow.

The dataset we’re using here was curated by PyImageSearch readers, Julia Riede and Nitin Rai.

The dataset consists of two classes, each with an equal number of images:

• camouflage_clothes: 7,949 images
• normal_clothes: 7,949 images

A sample of the images for each class can be seen in Figure 6.

In the remainder of this tutorial, you’ll learn how to fine-tune ResNet to predict both of these classes — the knowledge that you gain will enable you to fine-tune ResNet on your own datasets as well.

Downloading our camouflage vs. normal clothing dataset

The camouflage clothes vs. normal clothes dataset can be downloaded directly from Kaggle:

https://www.kaggle.com/imneonizer/normal-vs-camouflage-clothes

Simply click the “Download” button (Figure 7) to download a .zip archive of the dataset.

Project structure

Be sure to grab and unzip the code from the “Downloads” section of this blog post. Let’s take a moment to inspect the organizational structure of our project:

$ tree --dirsfirst --filelimit 10
.
├── 8k_normal_vs_camouflage_clothes_images
│   ├── camouflage_clothes [7949 entries]
│   └── normal_clothes [7949 entries]
├── pyimagesearch
│   ├── __init__.py
│   └── config.py
├── build_dataset.py
├── camo_detector.model
├── normal-vs-camouflage-clothes.zip
├── plot.png
└── train_camo_detector.py

4 directories, 7 files

As you can see, I’ve placed the dataset (normal-vs-camouflage-clothes.zip) in the root directory of our project and extracted the files. The images therein now reside in the 8k_normal_vs_camouflage_clothes_images directory.

Today’s pyimagesearch module comes with a single Python configuration file (config.py) that houses our important paths and variables. We’ll review this file in the next section.

Our Python driver scripts consist of:

• build_dataset.py: Splits our data into training, testing, and validation subdirectories
• train_camo_detector.py: Trains a camouflage classifier with Python, TensorFlow/Keras, and fine-tuning

Our configuration file

Before we can (1) build our camouflage vs. noncamouflage image dataset and (2) fine-tune ResNet on our image dataset, let’s first create a simple configuration file to store all our important image paths and variables.

Open up the config.py file in your project, and insert the following code:

# import the necessary packages
import os

# initialize the path to the *original* input directory of images
ORIG_INPUT_DATASET = "8k_normal_vs_camouflage_clothes_images"

# initialize the base path to the *new* directory that will contain
# our images after computing the training and testing split
BASE_PATH = "camo_not_camo"

# derive the training, validation, and testing directories
TRAIN_PATH = os.path.sep.join([BASE_PATH, "training"])
VAL_PATH = os.path.sep.join([BASE_PATH, "validation"])
TEST_PATH = os.path.sep.join([BASE_PATH, "testing"])

The os module import allows us to build dynamic paths directly in our configuration file.

Our existing input dataset path should be placed on Line 5 (the Kaggle dataset you should have downloaded by this point).

The path to our new dataset directory that will contain our training, testing, and validation splits is shown on Line 9. This path will be created by the build_dataset.py script.

Three subdirectories per class (we have two classes) will also be created (Lines 12-14) — the paths to our training, validation, and testing dataset splits. Each will be populated with a subset of the images from our dataset.

Next, we’ll define our split percentages and classes:

# define the amount of data that will be used training
TRAIN_SPLIT = 0.75

# the amount of validation data will be a percentage of the
# *training* data
VAL_SPLIT = 0.1

# define the names of the classes
CLASSES = ["camouflage_clothes", "normal_clothes"]

Training data will be represented by 75% of all the data available (Line 17), 10% of which will be marked for validation (Line 21).

Our camouflage and normal clothes classes are defined on Line 24.

We’ll wrap up with a few hyperparameters and our output model path:

# initialize the initial learning rate, batch size, and number of
# epochs to train for
INIT_LR = 1e-4
BS = 32
NUM_EPOCHS = 20

# define the path to the serialized output model after training
MODEL_PATH = "camo_detector.model"

The initial learning rate, batch size, and number of epochs to train for are set on Lines 28-30.

The path to the output serialized ResNet-based camouflage classification model after fine-tuning will be stored at the path defined on Line 33.

Implementing our camouflage dataset builder script

With our configuration file implemented, let’s move on to creating our dataset builder, which will:

1. Split our dataset into training, validation, and testing sets, respectively
2. Organize our images on disk so we can use Keras’ ImageDataGenerator class and associated flow_from_directory function to easily fine-tune ResNet

Open up build_dataset.py, and let’s get started:

# import the necessary packages
from pyimagesearch import config
from imutils import paths
import random
import shutil
import os

We begin by importing our config from the previous section along with the paths module, which will help us to find the image files on disk. Three modules built into Python will be used for shuffling paths and creating directories/subdirectories.

Let’s go ahead and grab the paths to all original images in our dataset:

# grab the paths to all input images in the original input directory
# and shuffle them
imagePaths = list(paths.list_images(config.ORIG_INPUT_DATASET))
random.seed(42)
random.shuffle(imagePaths)

# compute the training and testing split
i = int(len(imagePaths) * config.TRAIN_SPLIT)
trainPaths = imagePaths[:i]
testPaths = imagePaths[i:]

# we'll be using part of the training data for validation
i = int(len(trainPaths) * config.VAL_SPLIT)
valPaths = trainPaths[:i]
trainPaths = trainPaths[i:]

# define the datasets that we'll be building
datasets = [
	("training", trainPaths, config.TRAIN_PATH),
	("validation", valPaths, config.VAL_PATH),
	("testing", testPaths, config.TEST_PATH)
]

Lines 25-29 define the dataset splits we’ll be building in the remainder of this script. Let’s proceed:

# loop over the datasets
for (dType, imagePaths, baseOutput) in datasets:
	# show which data split we are creating
	print("[INFO] building '{}' split".format(dType))

	# if the output base output directory does not exist, create it
	if not os.path.exists(baseOutput):
		print("[INFO] 'creating {}' directory".format(baseOutput))
		os.makedirs(baseOutput)

	# loop over the input image paths
	for inputPath in imagePaths:
		# extract the filename of the input image along with its
		# corresponding class label
		filename = inputPath.split(os.path.sep)[-1]
		label = inputPath.split(os.path.sep)[-2]

		# build the path to the label directory
		labelPath = os.path.sep.join([baseOutput, label])

		# if the label output directory does not exist, create it
		if not os.path.exists(labelPath):
			print("[INFO] 'creating {}' directory".format(labelPath))
			os.makedirs(labelPath)

		# construct the path to the destination image and then copy
		# the image itself
		p = os.path.sep.join([labelPath, filename])
		shutil.copy2(inputPath, p)

This last block of code handles copying images from their original location into their destination path; directories and subdirectories are created in the process. Let’s review in more detail:

• We loop over each of the datasets, creating the directory if it doesn’t exist (Lines 32-39)
• For each of our imagePaths, we proceed to:
  • Extract the filename and class label (Lines 45 and 46)
  • Build the path to the label directory (Line 49) and create the subdirectory, if required (Lines 52-54)
  • Copy the image from the source directory into its destination (Lines 58 and 59)

In the next section, we’ll build our dataset accordingly.

Building the camouflage image dataset

Let’s now build and organize our image camouflage dataset.

Make sure you have:

1. Used the “Downloads” section of this tutorial to download the source code
2. Followed the “Downloading our camouflage vs. normal clothing dataset” section above to download the dataset

From there, open a terminal, and execute the following command:

$ python build_dataset.py
[INFO] building 'training' split
[INFO] 'creating camo_not_camo/training' directory
[INFO] 'creating camo_not_camo/training/normal_clothes' directory
[INFO] 'creating camo_not_camo/training/camouflage_clothes' directory
[INFO] building 'validation' split
[INFO] 'creating camo_not_camo/validation' directory
[INFO] 'creating camo_not_camo/validation/camouflage_clothes' directory
[INFO] 'creating camo_not_camo/validation/normal_clothes' directory
[INFO] building 'testing' split
[INFO] 'creating camo_not_camo/testing' directory
[INFO] 'creating camo_not_camo/testing/normal_clothes' directory
[INFO] 'creating camo_not_camo/testing/camouflage_clothes' directory

You can then use the tree command to inspect camo_not_camo directory to validate that each of the training, testing, and validation splits was created:

$ tree camo_not_camo --filelimit 20
camo_not_camo
├── testing
│   ├── camouflage_clothes [2007 entries]
│   └── normal_clothes [1968 entries]
├── training
│   ├── camouflage_clothes [5339 entries]
│   └── normal_clothes [5392 entries]
└── validation
    ├── camouflage_clothes [603 entries]
    └── normal_clothes [589 entries]

9 directories, 0 files

Implementing our ResNet fine-tuning script with Keras and TensorFlow

With our dataset created and properly organized on disk, let’s learn how we can fine-tune ResNet using Keras and TensorFlow.

Open the train_camo_detector.py file, and insert the following code:

# set the matplotlib backend so figures can be saved in the background
import matplotlib
matplotlib.use("Agg")

# import the necessary packages
from pyimagesearch import config
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.layers import AveragePooling2D
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Input
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.applications import ResNet50
from sklearn.metrics import classification_report
from imutils import paths
import matplotlib.pyplot as plt
import numpy as np
import argparse

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-p", "--plot", type=str, default="plot.png",
	help="path to output loss/accuracy plot")
args = vars(ap.parse_args())

# determine the total number of image paths in training, validation,
# and testing directories
totalTrain = len(list(paths.list_images(config.TRAIN_PATH)))
totalVal = len(list(paths.list_images(config.VAL_PATH)))
totalTest = len(list(paths.list_images(config.TEST_PATH)))

We have a single command line argument --plot, the path to an image file that will have our accuracy/loss training curves. Our other configurations are in the Python configuration file we reviewed previously.

Lines 30-32 determine the total number of training, validation, and testing images, respectively.

Next, we’ll prepare for data augmentation:

# initialize the training training data augmentation object
trainAug = ImageDataGenerator(
	rotation_range=25,
	zoom_range=0.1,
	width_shift_range=0.1,
	height_shift_range=0.1,
	shear_range=0.2,
	horizontal_flip=True,
	fill_mode="nearest")

# initialize the validation/testing data augmentation object (which
# we'll be adding mean subtraction to)
valAug = ImageDataGenerator()

# define the ImageNet mean subtraction (in RGB order) and set the
# the mean subtraction value for each of the data augmentation
# objects
mean = np.array([123.68, 116.779, 103.939], dtype="float32")
trainAug.mean = mean
valAug.mean = mean

Data augmentation allows for training time mutations of our images including random rotations, zooms, shifts, shears, flips, and mean subtraction. Lines 35-42 initialize our training data augmentation object with a selection of these parameters. Similarly, Line 46 initializes the validation/testing data augmentation object (it will only be used for mean subtraction).

Both of our data augmentation objects are set up to perform mean subtraction on-the-fly (Lines 51-53).

We’ll now instantiate three Python generators from our data augmentation objects:

# initialize the training generator
trainGen = trainAug.flow_from_directory(
	config.TRAIN_PATH,
	class_mode="categorical",
	target_size=(224, 224),
	color_mode="rgb",
	shuffle=True,
	batch_size=config.BS)

# initialize the validation generator
valGen = valAug.flow_from_directory(
	config.VAL_PATH,
	class_mode="categorical",
	target_size=(224, 224),
	color_mode="rgb",
	shuffle=False,
	batch_size=config.BS)

# initialize the testing generator
testGen = valAug.flow_from_directory(
	config.TEST_PATH,
	class_mode="categorical",
	target_size=(224, 224),
	color_mode="rgb",
	shuffle=False,
	batch_size=config.BS)

Let’s load our ResNet50 classification model and prepare it for fine-tuning:

# load the ResNet-50 network, ensuring the head FC layer sets are left
# off
print("[INFO] preparing model...")
baseModel = ResNet50(weights="imagenet", include_top=False,
	input_tensor=Input(shape=(224, 224, 3)))

# construct the head of the model that will be placed on top of the
# the base model
headModel = baseModel.output
headModel = AveragePooling2D(pool_size=(7, 7))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(256, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(len(config.CLASSES), activation="softmax")(headModel)

# place the head FC model on top of the base model (this will become
# the actual model we will train)
model = Model(inputs=baseModel.input, outputs=headModel)

# loop over all layers in the base model and freeze them so they will
# *not* be updated during the training process
for layer in baseModel.layers:
	layer.trainable = False

The final step for fine-tuning is to ensure that the weights of the base of our CNN are frozen (Lines 103 and 104) — we only want to train (i.e., fine-tune) the head of the network.

If you need to brush up on the concept of fine-tuning, please refer to my fine-tuning articles, in particular Fine-tuning with Keras and Deep Learning.

We’re now ready to fine-tune our ResNet-based camouflage detector with TensorFlow, Keras, and deep learning:

# compile the model
opt = Adam(lr=config.INIT_LR, decay=config.INIT_LR / config.NUM_EPOCHS)
model.compile(loss="binary_crossentropy", optimizer=opt,
	metrics=["accuracy"])

# train the model
print("[INFO] training model...")
H = model.fit_generator(
	trainGen,
	steps_per_epoch=totalTrain // config.BS,
	validation_data=valGen,
	validation_steps=totalVal // config.BS,
	epochs=config.NUM_EPOCHS)

First, we compile our model with learning rate decay and the Adam optimizer using "binary_crossentropy" loss, since this is a two-class problem (Lines 107-109). If you are training with more than two classes of data, be sure to set your loss to "categorical_crossentropy".

Lines 113-118 then train our model using our training and validation data generators.

Upon the completion of training, we’ll evaluate our model on the testing set:

# reset the testing generator and then use our trained model to
# make predictions on the data
print("[INFO] evaluating network...")
testGen.reset()
predIdxs = model.predict_generator(testGen,
	steps=(totalTest // config.BS) + 1)

# for each image in the testing set we need to find the index of the
# label with corresponding largest predicted probability
predIdxs = np.argmax(predIdxs, axis=1)

# show a nicely formatted classification report
print(classification_report(testGen.classes, predIdxs,
	target_names=testGen.class_indices.keys()))

# serialize the model to disk
print("[INFO] saving model...")
model.save(config.MODEL_PATH, save_format="h5")

# plot the training loss and accuracy
N = config.NUM_EPOCHS
plt.style.use("ggplot")
plt.figure()
plt.plot(np.arange(0, N), H.history["loss"], label="train_loss")
plt.plot(np.arange(0, N), H.history["val_loss"], label="val_loss")
plt.plot(np.arange(0, N), H.history["accuracy"], label="train_acc")
plt.plot(np.arange(0, N), H.history["val_accuracy"], label="val_acc")
plt.title("Training Loss and Accuracy on Dataset")
plt.xlabel("Epoch #")
plt.ylabel("Loss/Accuracy")
plt.legend(loc="lower left")
plt.savefig(args["plot"])

Once the plot is generated, Line 151 saves it to disk in the location specified by our --plot command line argument.

Fine-tuning ResNet with Keras and TensorFlow results

We are now ready to fine-tune ResNet with Keras and TensorFlow.

Make sure you have:

• Used the “Downloads” section of this tutorial to download the source code
• Followed the “Downloading our camouflage vs. normal clothing dataset” section above to download the dataset
• Executed the build_dataset.py script to organize the dataset into the project directory structure for training

From there, open up a terminal, and run the train_camo_detector.py script:

$ python train_camo_detector.py
Found 10731 images belonging to 2 classes.
Found 1192 images belonging to 2 classes.
Found 3975 images belonging to 2 classes.
[INFO] preparing model...
[INFO] training model...
Epoch 1/20
335/335 [==============================] - 311s 929ms/step - loss: 0.1736 - accuracy: 0.9326 - val_loss: 0.1050 - val_accuracy: 0.9671
Epoch 2/20
335/335 [==============================] - 305s 912ms/step - loss: 0.0997 - accuracy: 0.9632 - val_loss: 0.1028 - val_accuracy: 0.9586
Epoch 3/20
335/335 [==============================] - 305s 910ms/step - loss: 0.0729 - accuracy: 0.9753 - val_loss: 0.0951 - val_accuracy: 0.9730
...
Epoch 18/20
335/335 [==============================] - 298s 890ms/step - loss: 0.0336 - accuracy: 0.9878 - val_loss: 0.0854 - val_accuracy: 0.9696
Epoch 19/20
335/335 [==============================] - 298s 891ms/step - loss: 0.0296 - accuracy: 0.9896 - val_loss: 0.0850 - val_accuracy: 0.9679
Epoch 20/20
335/335 [==============================] - 299s 894ms/step - loss: 0.0275 - accuracy: 0.9905 - val_loss: 0.0955 - val_accuracy: 0.9679
[INFO] evaluating network...
                   precision    recall  f1-score   support

    normal_clothes       0.95      0.99      0.97      2007
camouflage_clothes       0.99      0.95      0.97      1968

          accuracy                           0.97      3975
         macro avg       0.97      0.97      0.97      3975
      weighted avg       0.97      0.97      0.97      3975

[INFO] saving model...

Here, you can see that we are obtaining ~97% accuracy on our normal clothes vs. camouflage clothes detector.

Our training plot is shown below:

Our training loss decreases at a much sharper rate than our validation loss; furthermore, it appears that validation loss may be rising toward the end of training, indicating that the model may be overfitting.

Future experiments should look into applying additional regularization to the model as well as gathering additional training data.

In two weeks, I’ll show you how to take this fine-tuned ResNet model and use it in a practical, real-world application!

Stay tuned for the post; you won’t want to miss it!

Credits

This tutorial would not be possible without:

• Victor Gevers of the GDI.Foundation, who brought this project to my attention
• Nitin Rai who curated the normal clothes vs. camouflage clothes and posted the dataset on Kaggle
• Julia Riede who curated a variation of the dataset

Additionally, I’d like to credit Han et al. for the ResNet-152 visualization used in the header of this image.

What’s next?

Inside today’s tutorial we covered fine-tuning ResNet, but if you want a deeper dive into transfer learning and fine-tuning, I would recommend reading my book, Deep Learning for Computer Vision with Python.

Inside the book I cover:

• The theory behind transfer learning and fine-tuning
• How to take any pre-trained model and prepare it for transfer learning/fine-tuning
• How to perform transfer learning and fine-tuning with Keras and TensorFlow on your own datasets

You’ll also find answers and proven code recipes to:

• Create and prepare your own custom image datasets for image classification, object detection, and segmentation
• Hands-on tutorials (with lots of code) that not only show you the algorithms behind deep learning for computer vision but their implementations as well
• Use my tips, suggestions, and best practices to ensure you maximize the accuracy of your models

Readers of mine, enjoy my no-nonsense teaching style that is guaranteed to help you master deep learning for image understanding and visual recognition.

If you’re ready to dive in, just click here. You can fill out the form to grab free sample chapters and the entire table of contents.

Summary

In this tutorial you learned how to fine-tune ResNet with Keras and TensorFlow.

Fine-tuning is the process of:

1. Taking a pre-trained deep neural network (in this case, ResNet)
2. Removing the fully-connected layer head from the network
3. Placing a new, freshly initialized layer head on top of the body of the network
4. Optionally freezing the weights for the layers in the body
5. Training the model, using the pre-trained weights as a starting point to help the model learn faster

Using fine-tune we can obtain a higher accuracy model, typically with much less effort, data, and training time.

As a practical application, we fine-tuned ResNet on a dataset of camouflage vs. noncamouflage clothes images.

This dataset was curated and put together for us by PyImageSearch readers, Julia Riede and Nitin Rai — without them, this tutorial, as well as the project Victor Gevers and I were working on, would not have been possible! Please thank both Julia and Nitin if you see them online.

In two weeks, I’ll go into the details of the project that Victor Gevers and I have been working on, which wraps a nice a little bow on the following topics that we’ve recently covered on PyImageSearch:

• Face detection
• Age detection
• Removing duplicates from a deep learning dataset
• Fine-tuning a model for camouflage clothes vs. noncamouflage clothes detection

It’s a great post with very real applications to make the world a better place with computer vision and deep learning — you won’t want to miss it!

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Fine-tuning ResNet with Keras, TensorFlow, and Deep Learning appeared first on PyImageSearch.

In this tutorial, you will learn how to train a COVID-19 face mask detector with OpenCV, Keras/TensorFlow, and Deep Learning.

Last month, I authored a blog post on detecting COVID-19 in X-ray images using deep learning.

Readers really enjoyed learning from the timely, practical application of that tutorial, so today we are going to look at another COVID-related application of computer vision, this one on detecting face masks with OpenCV and Keras/TensorFlow.

I was inspired to author this tutorial after:

1. Receiving numerous requests from PyImageSearch readers asking that I write such a blog post
2. Seeing others implement their own solutions (my favorite being Prajna Bhandary’s, which we are going to build from today)

If deployed correctly, the COVID-19 mask detector we’re building here today could potentially be used to help ensure your safety and the safety of others (but I’ll leave that to the medical professionals to decide on, implement, and distribute in the wild).

To learn how to create a COVID-19 face mask detector with OpenCV, Keras/TensorFlow, and Deep Learning, just keep reading!

Looking for the source code to this post?

Jump Right To The Downloads Section

COVID-19: Face Mask Detector with OpenCV, Keras/TensorFlow, and Deep Learning

In this tutorial, we’ll discuss our two-phase COVID-19 face mask detector, detailing how our computer vision/deep learning pipeline will be implemented.

From there, we’ll review the dataset we’ll be using to train our custom face mask detector.

I’ll then show you how to implement a Python script to train a face mask detector on our dataset using Keras and TensorFlow.

We’ll use this Python script to train a face mask detector and review the results.

Given the trained COVID-19 face mask detector, we’ll proceed to implement two more additional Python scripts used to:

1. Detect COVID-19 face masks in images
2. Detect face masks in real-time video streams

We’ll wrap up the post by looking at the results of applying our face mask detector.

I’ll also provide some additional suggestions for further improvement.

Two-phase COVID-19 face mask detector

In order to train a custom face mask detector, we need to break our project into two distinct phases, each with its own respective sub-steps (as shown by Figure 1 above):

1. Training: Here we’ll focus on loading our face mask detection dataset from disk, training a model (using Keras/TensorFlow) on this dataset, and then serializing the face mask detector to disk
2. Deployment: Once the face mask detector is trained, we can then move on to loading the mask detector, performing face detection, and then classifying each face as with_mask or without_mask

We’ll review each of these phases and associated subsets in detail in the remainder of this tutorial, but in the meantime, let’s take a look at the dataset we’ll be using to train our COVID-19 face mask detector.

Our COVID-19 face mask detection dataset

The dataset we’ll be using here today was created by PyImageSearch reader Prajna Bhandary.

This dataset consists of 1,376 images belonging to two classes:

• with_mask: 690 images
• without_mask: 686 images

Our goal is to train a custom deep learning model to detect whether a person is or is not wearing a mask.

Note: For convenience, I have included the dataset created by Prajna in the “Downloads” section of this tutorial.

How was our face mask dataset created?

Prajna, like me, has been feeling down and depressed about the state of the world — thousands of people are dying each day, and for many of us, there is very little (if anything) we can do.

To help keep her spirts up, Prajna decided to distract herself by applying computer vision and deep learning to solve a real-world problem:

• Best case scenario — she could use her project to help others
• Worst case scenario — it gave her a much needed mental escape

Either way, it’s win-win!

As programmers, developers, and computer vision/deep learning practitioners, we can all take a page from Prajna’s book — let your skills become your distraction and your haven.

To create this dataset, Prajna had the ingenious solution of:

1. Taking normal images of faces
2. Then creating a custom computer vision Python script to add face masks to them, thereby creating an artificial (but still real-world applicable) dataset

This method is actually a lot easier than it sounds once you apply facial landmarks to the problem.

Facial landmarks allow us to automatically infer the location of facial structures, including:

• Eyes
• Eyebrows
• Nose
• Mouth
• Jawline

To use facial landmarks to build a dataset of faces wearing face masks, we need to first start with an image of a person not wearing a face mask:

From there, we apply face detection to compute the bounding box location of the face in the image:

Once we know where in the image the face is, we can extract the face Region of Interest (ROI):

And from there, we apply facial landmarks, allowing us to localize the eyes, nose, mouth, etc.:

Next, we need an image of a mask (with a transparent background) such as the one below:

This mask will be automatically applied to the face by using the facial landmarks (namely the points along the chin and nose) to compute where the mask will be placed.

The mask is then resized and rotated, placing it on the face:

We can then repeat this process for all of our input images, thereby creating our artificial face mask dataset:

However, there is a caveat you should be aware of when using this method to artificially create a dataset!

If you use a set of images to create an artificial dataset of people wearing masks, you cannot “re-use” the images without masks in your training set — you still need to gather non-face mask images that were not used in the artificial generation process!

If you include the original images used to generate face mask samples as non-face mask samples, your model will become heavily biased and fail to generalize well. Avoid that at all costs by taking the time to gather new examples of faces without masks.

Covering how to use facial landmarks to apply a mask to a face is outside the scope of this tutorial, but if you want to learn more about it, I would suggest:

1. Referring to Prajna’s GitHub repository
2. Reading this tutorial on the PyImageSearch blog where I discuss how to use facial landmarks to automatically apply sunglasses to a face

The same principle from my sunglasses post applies to building an artificial face mask dataset — use the facial landmarks to infer the facial structures, rotate and resize the mask, and then apply it to the image.

Project structure

Once you grab the files from the “Downloads” section of this article, you’ll be presented with the following directory structure:

$ tree --dirsfirst --filelimit 10
.
├── dataset
│   ├── with_mask [690 entries]
│   └── without_mask [686 entries]
├── examples
│   ├── example_01.png
│   ├── example_02.png
│   └── example_03.png
├── face_detector
│   ├── deploy.prototxt
│   └── res10_300x300_ssd_iter_140000.caffemodel
├── detect_mask_image.py
├── detect_mask_video.py
├── mask_detector.model
├── plot.png
└── train_mask_detector.py

5 directories, 10 files

• train_mask_detector.py: Accepts our input dataset and fine-tunes MobileNetV2 upon it to create our mask_detector.model. A training history plot.png containing accuracy/loss curves is also produced
• detect_mask_image.py: Performs face mask detection in static images
• detect_mask_video.py: Using your webcam, this script applies face mask detection to every frame in the stream

In the next two sections, we will train our face mask detector.

Implementing our COVID-19 face mask detector training script with Keras and TensorFlow

Now that we’ve reviewed our face mask dataset, let’s learn how we can use Keras and TensorFlow to train a classifier to automatically detect whether a person is wearing a mask or not.

To accomplish this task, we’ll be fine-tuning the MobileNet V2 architecture, a highly efficient architecture that can be applied to embedded devices with limited computational capacity (ex., Raspberry Pi, Google Coral, NVIDIA Jetson Nano, etc.).

Note: If your interest is embedded computer vision, be sure to check out my Raspberry Pi for Computer Vision book which covers working with computationally limited devices for computer vision and deep learning.

Deploying our face mask detector to embedded devices could reduce the cost of manufacturing such face mask detection systems, hence why we choose to use this architecture.

Let’s get started!

Open up the train_mask_detector.py file in your directory structure, and insert the following code:

# import the necessary packages
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications import MobileNetV2
from tensorflow.keras.layers import AveragePooling2D
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Input
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.applications.mobilenet_v2 import preprocess_input
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.utils import to_categorical
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from imutils import paths
import matplotlib.pyplot as plt
import numpy as np
import argparse
import os

The imports for our training script may look intimidating to you either because there are so many or you are new to deep learning. If you are new, I would recommend reading both my Keras tutorial and fine-tuning tutorial before moving forward.

Our set of tensorflow.keras imports allow for:

• Data augmentation
• Loading the MobilNetV2 classifier (we will fine-tune this model with pre-trained ImageNet weights)
• Building a new fully-connected (FC) head
• Pre-processing
• Loading image data

To install the necessary software so that these imports are available to you, be sure to follow either one of my Tensorflow 2.0+ installation guides:

• How to install TensorFlow 2.0 on Ubuntu
• How to install TensorFlow 2.0 on macOS

Let’s go ahead and parse a few command line arguments that are required to launch our script from a terminal:

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d", "--dataset", required=True,
	help="path to input dataset")
ap.add_argument("-p", "--plot", type=str, default="plot.png",
	help="path to output loss/accuracy plot")
ap.add_argument("-m", "--model", type=str,
	default="mask_detector.model",
	help="path to output face mask detector model")
args = vars(ap.parse_args())

Our command line arguments include:

• --dataset: The path to the input dataset of faces and and faces with masks
• --plot: The path to your output training history plot, which will be generated using matplotlib
• --model: The path to the resulting serialized face mask classification model

I like to define my deep learning hyperparameters in one place:

# initialize the initial learning rate, number of epochs to train for,
# and batch size
INIT_LR = 1e-4
EPOCHS = 20
BS = 32

# grab the list of images in our dataset directory, then initialize
# the list of data (i.e., images) and class images
print("[INFO] loading images...")
imagePaths = list(paths.list_images(args["dataset"]))
data = []
labels = []

# loop over the image paths
for imagePath in imagePaths:
	# extract the class label from the filename
	label = imagePath.split(os.path.sep)[-2]

	# load the input image (224x224) and preprocess it
	image = load_img(imagePath, target_size=(224, 224))
	image = img_to_array(image)
	image = preprocess_input(image)

	# update the data and labels lists, respectively
	data.append(image)
	labels.append(label)

# convert the data and labels to NumPy arrays
data = np.array(data, dtype="float32")
labels = np.array(labels)

In this block, we are:

• Grabbing all of the imagePaths in the dataset (Line 44)
• Initializing data and labels lists (Lines 45 and 46)
• Looping over the imagePaths and loading + pre-processing images (Lines 49-60). Pre-processing steps include resizing to 224×224 pixels, conversion to array format, and scaling the pixel intensities in the input image to the range [-1, 1] (via the preprocess_input convenience function)
• Appending the pre-processed image and associated label to the data and labels lists, respectively (Lines 59 and 60)
• Ensuring our training data is in NumPy array format (Lines 63 and 64)

The above lines of code assume that your entire dataset is small enough to fit into memory. If your dataset is larger than the memory you have available, I suggest using HDF5, a strategy I cover in Deep Learning for Computer Vision with Python (Practitioner Bundle Chapters 9 and 10).

Our data preparation work isn’t done yet. Next, we’ll encode our labels, partition our dataset, and prepare for data augmentation:

# perform one-hot encoding on the labels
lb = LabelBinarizer()
labels = lb.fit_transform(labels)
labels = to_categorical(labels)

# partition the data into training and testing splits using 75% of
# the data for training and the remaining 25% for testing
(trainX, testX, trainY, testY) = train_test_split(data, labels,
	test_size=0.20, stratify=labels, random_state=42)

# construct the training image generator for data augmentation
aug = ImageDataGenerator(
	rotation_range=20,
	zoom_range=0.15,
	width_shift_range=0.2,
	height_shift_range=0.2,
	shear_range=0.15,
	horizontal_flip=True,
	fill_mode="nearest")

Lines 67-69 one-hot encode our class labels, meaning that our data will be in the following format:

$ python  train_mask_detector.py --dataset  dataset 
[INFO] loading images...
-> (trainX, testX, trainY, testY) = train_test_split(data, labels,
(Pdb) labels[500:]
array([[1., 0.],
       [1., 0.],
       [1., 0.],
       ...,
       [0., 1.],
       [0., 1.],
       [0., 1.]], dtype=float32)
(Pdb)

# load the MobileNetV2 network, ensuring the head FC layer sets are
# left off
baseModel = MobileNetV2(weights="imagenet", include_top=False,
	input_tensor=Input(shape=(224, 224, 3)))

# construct the head of the model that will be placed on top of the
# the base model
headModel = baseModel.output
headModel = AveragePooling2D(pool_size=(7, 7))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(128, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(2, activation="softmax")(headModel)

# place the head FC model on top of the base model (this will become
# the actual model we will train)
model = Model(inputs=baseModel.input, outputs=headModel)

# loop over all layers in the base model and freeze them so they will
# *not* be updated during the first training process
for layer in baseModel.layers:
	layer.trainable = False

Fine-tuning setup is a three-step process:

1. Load MobileNet with pre-trained ImageNet weights, leaving off head of network (Lines 88 and 89)
2. Construct a new FC head, and append it to the base in place of the old head (Lines 93-102)
3. Freeze the base layers of the network (Lines 106 and 107). The weights of these base layers will not be updated during the process of backpropagation, whereas the head layer weights will be tuned.

Fine-tuning is a strategy I nearly always recommend to establish a baseline model while saving considerable time. To learn more about the theory, purpose, and strategy, please refer to my fine-tuning blog posts and Deep Learning for Computer Vision with Python (Practitioner Bundle Chapter 5).

With our data prepared and model architecture in place for fine-tuning, we’re now ready to compile and train our face mask detector network:

# compile our model
print("[INFO] compiling model...")
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="binary_crossentropy", optimizer=opt,
	metrics=["accuracy"])

# train the head of the network
print("[INFO] training head...")
H = model.fit(
	aug.flow(trainX, trainY, batch_size=BS),
	steps_per_epoch=len(trainX) // BS,
	validation_data=(testX, testY),
	validation_steps=len(testX) // BS,
	epochs=EPOCHS)

Once training is complete, we’ll evaluate the resulting model on the test set:

# make predictions on the testing set
print("[INFO] evaluating network...")
predIdxs = model.predict(testX, batch_size=BS)

# for each image in the testing set we need to find the index of the
# label with corresponding largest predicted probability
predIdxs = np.argmax(predIdxs, axis=1)

# show a nicely formatted classification report
print(classification_report(testY.argmax(axis=1), predIdxs,
	target_names=lb.classes_))

# serialize the model to disk
print("[INFO] saving mask detector model...")
model.save(args["model"], save_format="h5")

Our last step is to plot our accuracy and loss curves:

# plot the training loss and accuracy
N = EPOCHS
plt.style.use("ggplot")
plt.figure()
plt.plot(np.arange(0, N), H.history["loss"], label="train_loss")
plt.plot(np.arange(0, N), H.history["val_loss"], label="val_loss")
plt.plot(np.arange(0, N), H.history["accuracy"], label="train_acc")
plt.plot(np.arange(0, N), H.history["val_accuracy"], label="val_acc")
plt.title("Training Loss and Accuracy")
plt.xlabel("Epoch #")
plt.ylabel("Loss/Accuracy")
plt.legend(loc="lower left")
plt.savefig(args["plot"])

Once our plot is ready, Line 152 saves the figure to disk using the --plot filepath.

Training the COVID-19 face mask detector with Keras/TensorFlow

We are now ready to train our face mask detector using Keras, TensorFlow, and Deep Learning.

Make sure you have used the “Downloads” section of this tutorial to download the source code and face mask dataset.

From there, open up a terminal, and execute the following command:

$ python train_mask_detector.py --dataset dataset
[INFO] loading images...
[INFO] compiling model...
[INFO] training head...
Train for 34 steps, validate on 276 samples
Epoch 1/20
34/34 [==============================] - 30s 885ms/step - loss: 0.6431 - accuracy: 0.6676 - val_loss: 0.3696 - val_accuracy: 0.8242
Epoch 2/20
34/34 [==============================] - 29s 853ms/step - loss: 0.3507 - accuracy: 0.8567 - val_loss: 0.1964 - val_accuracy: 0.9375
Epoch 3/20
34/34 [==============================] - 27s 800ms/step - loss: 0.2792 - accuracy: 0.8820 - val_loss: 0.1383 - val_accuracy: 0.9531
Epoch 4/20
34/34 [==============================] - 28s 814ms/step - loss: 0.2196 - accuracy: 0.9148 - val_loss: 0.1306 - val_accuracy: 0.9492
Epoch 5/20
34/34 [==============================] - 27s 792ms/step - loss: 0.2006 - accuracy: 0.9213 - val_loss: 0.0863 - val_accuracy: 0.9688
...
Epoch 16/20
34/34 [==============================] - 27s 801ms/step - loss: 0.0767 - accuracy: 0.9766 - val_loss: 0.0291 - val_accuracy: 0.9922
Epoch 17/20
34/34 [==============================] - 27s 795ms/step - loss: 0.1042 - accuracy: 0.9616 - val_loss: 0.0243 - val_accuracy: 1.0000
Epoch 18/20
34/34 [==============================] - 27s 796ms/step - loss: 0.0804 - accuracy: 0.9672 - val_loss: 0.0244 - val_accuracy: 0.9961
Epoch 19/20
34/34 [==============================] - 27s 793ms/step - loss: 0.0836 - accuracy: 0.9710 - val_loss: 0.0440 - val_accuracy: 0.9883
Epoch 20/20
34/34 [==============================] - 28s 838ms/step - loss: 0.0717 - accuracy: 0.9710 - val_loss: 0.0270 - val_accuracy: 0.9922
[INFO] evaluating network...
              precision    recall  f1-score   support

   with_mask       0.99      1.00      0.99       138
without_mask       1.00      0.99      0.99       138

    accuracy                           0.99       276
   macro avg       0.99      0.99      0.99       276
weighted avg       0.99      0.99      0.99       276

As you can see, we are obtaining ~99% accuracy on our test set.

Looking at Figure 10, we can see there are little signs of overfitting, with the validation loss lower than the training loss (a phenomenon I discuss in this blog post).

Given these results, we are hopeful that our model will generalize well to images outside our training and testing set.

Implementing our COVID-19 face mask detector for images with OpenCV

Now that our face mask detector is trained, let’s learn how we can:

1. Load an input image from disk
2. Detect faces in the image
3. Apply our face mask detector to classify the face as either with_mask or without_mask

Open up the detect_mask_image.py file in your directory structure, and let’s get started:

# import the necessary packages
from tensorflow.keras.applications.mobilenet_v2 import preprocess_input
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.models import load_model
import numpy as np
import argparse
import cv2
import os

Our driver script requires three TensorFlow/Keras imports to (1) load our MaskNet model and (2) pre-process the input image.

OpenCV is required for display and image manipulations.

The next step is to parse command line arguments:

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
	help="path to input image")
ap.add_argument("-f", "--face", type=str,
	default="face_detector",
	help="path to face detector model directory")
ap.add_argument("-m", "--model", type=str,
	default="mask_detector.model",
	help="path to trained face mask detector model")
ap.add_argument("-c", "--confidence", type=float, default=0.5,
	help="minimum probability to filter weak detections")
args = vars(ap.parse_args())

Our four command line arguments include:

• --image: The path to the input image containing faces for inference
• --face: The path to the face detector model directory (we need to localize faces prior to classifying them)
• --model: The path to the face mask detector model that we trained earlier in this tutorial
• --confidence: An optional probability threshold can be set to override 50% to filter weak face detections

Next, we’ll load both our face detector and face mask classifier models:

# load our serialized face detector model from disk
print("[INFO] loading face detector model...")
prototxtPath = os.path.sep.join([args["face"], "deploy.prototxt"])
weightsPath = os.path.sep.join([args["face"],
	"res10_300x300_ssd_iter_140000.caffemodel"])
net = cv2.dnn.readNet(prototxtPath, weightsPath)

# load the face mask detector model from disk
print("[INFO] loading face mask detector model...")
model = load_model(args["model"])

With our deep learning models now in memory, our next step is to load and pre-process an input image:

# load the input image from disk, clone it, and grab the image spatial
# dimensions
image = cv2.imread(args["image"])
orig = image.copy()
(h, w) = image.shape[:2]

# construct a blob from the image
blob = cv2.dnn.blobFromImage(image, 1.0, (300, 300),
	(104.0, 177.0, 123.0))

# pass the blob through the network and obtain the face detections
print("[INFO] computing face detections...")
net.setInput(blob)
detections = net.forward()

Upon loading our --image from disk (Line 37), we make a copy and grab frame dimensions for future scaling and display purposes (Lines 38 and 39).

Pre-processing is handled by OpenCV’s blobFromImage function (Lines 42 and 43). As shown in the parameters, we resize to 300×300 pixels and perform mean subtraction.

Lines 47 and 48 then perform face detection to localize where in the image all faces are.

Once we know where each face is predicted to be, we’ll ensure they meet the --confidence threshold before we extract the faceROIs:

# loop over the detections
for i in range(0, detections.shape[2]):
	# extract the confidence (i.e., probability) associated with
	# the detection
	confidence = detections[0, 0, i, 2]

	# filter out weak detections by ensuring the confidence is
	# greater than the minimum confidence
	if confidence > args["confidence"]:
		# compute the (x, y)-coordinates of the bounding box for
		# the object
		box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
		(startX, startY, endX, endY) = box.astype("int")

		# ensure the bounding boxes fall within the dimensions of
		# the frame
		(startX, startY) = (max(0, startX), max(0, startY))
		(endX, endY) = (min(w - 1, endX), min(h - 1, endY))

Next, we’ll run the face ROI through our MaskNet model:

		# extract the face ROI, convert it from BGR to RGB channel
		# ordering, resize it to 224x224, and preprocess it
		face = image[startY:endY, startX:endX]
		face = cv2.cvtColor(face, cv2.COLOR_BGR2RGB)
		face = cv2.resize(face, (224, 224))
		face = img_to_array(face)
		face = preprocess_input(face)
		face = np.expand_dims(face, axis=0)

		# pass the face through the model to determine if the face
		# has a mask or not
		(mask, withoutMask) = model.predict(face)[0]

In this block, we:

• Extract the face ROI via NumPy slicing (Line 71)
• Pre-process the ROI the same way we did during training (Lines 72-76)
• Perform mask detection to predict with_mask or without_mask (Line 80)

From here, we will annotate and display the result!

		# determine the class label and color we'll use to draw
		# the bounding box and text
		label = "Mask" if mask > withoutMask else "No Mask"
		color = (0, 255, 0) if label == "Mask" else (0, 0, 255)

		# include the probability in the label
		label = "{}: {:.2f}%".format(label, max(mask, withoutMask) * 100)

		# display the label and bounding box rectangle on the output
		# frame
		cv2.putText(image, label, (startX, startY - 10),
			cv2.FONT_HERSHEY_SIMPLEX, 0.45, color, 2)
		cv2.rectangle(image, (startX, startY), (endX, endY), color, 2)

# show the output image
cv2.imshow("Output", image)
cv2.waitKey(0)

COVID-19 face mask detection in images with OpenCV

Let’s put our COVID-19 face mask detector to work!

Make sure you have used the “Downloads” section of this tutorial to download the source code, example images, and pre-trained face mask detector.

From there, open up a terminal, and execute the following command:

$ python detect_mask_image.py --image examples/example_01.png 
[INFO] loading face detector model...
[INFO] loading face mask detector model...
[INFO] computing face detections...

As you can see, our face mask detector correctly labeled this image as Mask.

Let’s try another image, this one of a person not wearing a face mask:

$ python detect_mask_image.py --image examples/example_02.png 
[INFO] loading face detector model...
[INFO] loading face mask detector model...
[INFO] computing face detections...

Our face mask detector has correctly predicted No Mask.

Let’s try one final image:

$ python detect_mask_image.py --image examples/example_03.png 
[INFO] loading face detector model...
[INFO] loading face mask detector model...
[INFO] computing face detections...

What happened here?

Why is it that we were able to detect the faces of the two gentlemen in the background and correctly classify mask/no mask for them, but we could not detect the woman in the foreground?

I discuss the reason for this issue in the “Suggestions for further improvement” section later in this tutorial, but the gist is that we’re too reliant on our two-stage process.

Keep in mind that in order to classify whether or not a person is wearing in mask, we first need to perform face detection — if a face is not found (which is what happened in this image), then the mask detector cannot be applied!

The reason we cannot detect the face in the foreground is because:

1. It’s too obscured by the mask
2. The dataset used to train the face detector did not contain example images of people wearing face masks

Therefore, if a large portion of the face is occluded, our face detector will likely fail to detect the face.

Again, I discuss this problem in more detail, including how to improve the accuracy of our mask detector, in the “Suggestions for further improvement” section of this tutorial.

Implementing our COVID-19 face mask detector in real-time video streams with OpenCV

At this point, we know we can apply face mask detection to static images — but what about real-time video streams?

Is our COVID-19 face mask detector capable of running in real-time?

Let’s find out.

Open up the detect_mask_video.py file in your directory structure, and insert the following code:

# import the necessary packages
from tensorflow.keras.applications.mobilenet_v2 import preprocess_input
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.models import load_model
from imutils.video import VideoStream
import numpy as np
import argparse
import imutils
import time
import cv2
import os

The algorithm for this script is the same, but it is pieced together in such a way to allow for processing every frame of your webcam stream.

def detect_and_predict_mask(frame, faceNet, maskNet):
	# grab the dimensions of the frame and then construct a blob
	# from it
	(h, w) = frame.shape[:2]
	blob = cv2.dnn.blobFromImage(frame, 1.0, (300, 300),
		(104.0, 177.0, 123.0))

	# pass the blob through the network and obtain the face detections
	faceNet.setInput(blob)
	detections = faceNet.forward()

	# initialize our list of faces, their corresponding locations,
	# and the list of predictions from our face mask network
	faces = []
	locs = []
	preds = []

By defining this convenience function here, our frame processing loop will be a little easier to read later.

This function detects faces and then applies our face mask classifier to each face ROI. Such a function consolidates our code — it could even be moved to a separate Python file if you so choose.

	# loop over the detections
	for i in range(0, detections.shape[2]):
		# extract the confidence (i.e., probability) associated with
		# the detection
		confidence = detections[0, 0, i, 2]

		# filter out weak detections by ensuring the confidence is
		# greater than the minimum confidence
		if confidence > args["confidence"]:
			# compute the (x, y)-coordinates of the bounding box for
			# the object
			box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
			(startX, startY, endX, endY) = box.astype("int")

			# ensure the bounding boxes fall within the dimensions of
			# the frame
			(startX, startY) = (max(0, startX), max(0, startY))
			(endX, endY) = (min(w - 1, endX), min(h - 1, endY))

Inside the loop, we filter out weak detections (Lines 34-38) and extract bounding boxes while ensuring bounding box coordinates do not fall outside the bounds of the image (Lines 41-47).

Next, we’ll add face ROIs to two of our corresponding lists:

			# extract the face ROI, convert it from BGR to RGB channel
			# ordering, resize it to 224x224, and preprocess it
			face = frame[startY:endY, startX:endX]
			face = cv2.cvtColor(face, cv2.COLOR_BGR2RGB)
			face = cv2.resize(face, (224, 224))
			face = img_to_array(face)
			face = preprocess_input(face)
			face = np.expand_dims(face, axis=0)

			# add the face and bounding boxes to their respective
			# lists
			faces.append(face)
			locs.append((startX, startY, endX, endY))

	# only make a predictions if at least one face was detected
	if len(faces) > 0:
		# for faster inference we'll make batch predictions on *all*
		# faces at the same time rather than one-by-one predictions
		# in the above `for` loop
		preds = maskNet.predict(faces)

	# return a 2-tuple of the face locations and their corresponding
	# locations
	return (locs, preds)

Line 72 returns our face bounding box locations and corresponding mask/not mask predictions to the caller.

Next, we’ll define our command line arguments:

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-f", "--face", type=str,
	default="face_detector",
	help="path to face detector model directory")
ap.add_argument("-m", "--model", type=str,
	default="mask_detector.model",
	help="path to trained face mask detector model")
ap.add_argument("-c", "--confidence", type=float, default=0.5,
	help="minimum probability to filter weak detections")
args = vars(ap.parse_args())

Our command line arguments include:

# load our serialized face detector model from disk
print("[INFO] loading face detector model...")
prototxtPath = os.path.sep.join([args["face"], "deploy.prototxt"])
weightsPath = os.path.sep.join([args["face"],
	"res10_300x300_ssd_iter_140000.caffemodel"])
faceNet = cv2.dnn.readNet(prototxtPath, weightsPath)

# load the face mask detector model from disk
print("[INFO] loading face mask detector model...")
maskNet = load_model(args["model"])

# initialize the video stream and allow the camera sensor to warm up
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
time.sleep(2.0)

Here we have initialized our:

• Face detector
• COVID-19 face mask detector
• Webcam video stream

Let’s proceed to loop over frames in the stream:

# loop over the frames from the video stream
while True:
	# grab the frame from the threaded video stream and resize it
	# to have a maximum width of 400 pixels
	frame = vs.read()
	frame = imutils.resize(frame, width=400)

	# detect faces in the frame and determine if they are wearing a
	# face mask or not
	(locs, preds) = detect_and_predict_mask(frame, faceNet, maskNet)

We begin looping over frames on Line 103. Inside, we grab a frame from the stream and resize it (Lines 106 and 107).

From there, we put our convenience utility to use; Line 111 detects and predicts whether people are wearing their masks or not.

Let’s post-process (i.e., annotate) the COVID-19 face mask detection results:

	# loop over the detected face locations and their corresponding
	# locations
	for (box, pred) in zip(locs, preds):
		# unpack the bounding box and predictions
		(startX, startY, endX, endY) = box
		(mask, withoutMask) = pred

		# determine the class label and color we'll use to draw
		# the bounding box and text
		label = "Mask" if mask > withoutMask else "No Mask"
		color = (0, 255, 0) if label == "Mask" else (0, 0, 255)

		# include the probability in the label
		label = "{}: {:.2f}%".format(label, max(mask, withoutMask) * 100)

		# display the label and bounding box rectangle on the output
		# frame
		cv2.putText(frame, label, (startX, startY - 10),
			cv2.FONT_HERSHEY_SIMPLEX, 0.45, color, 2)
		cv2.rectangle(frame, (startX, startY), (endX, endY), color, 2)

Inside our loop over the prediction results (beginning on Line 115), we:

• Unpack a face bounding box and mask/not mask prediction (Lines 117 and 118)
• Determine the label and color (Lines 122-126)
• Annotate the label and face bounding box (Lines 130-132)

Finally, we display the results and perform cleanup:

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the `q` key was pressed, break from the loop
	if key == ord("q"):
		break

# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()

After the frame is displayed, we capture key presses. If the user presses q (quit), we break out of the loop and perform housekeeping.

Great job implementing your real-time face mask detector with Python, OpenCV, and deep learning with TensorFlow/Keras!

Detecting COVID-19 face masks with OpenCV in real-time

To see our real-time COVID-19 face mask detector in action, make sure you use the “Downloads” section of this tutorial to download the source code and pre-trained face mask detector model.

You can then launch the mask detector in real-time video streams using the following command:

$ python detect_mask_video.py
[INFO] loading face detector model...
[INFO] loading face mask detector model...
[INFO] starting video stream...

Here, you can see that our face mask detector is capable of running in real-time (and is correct in its predictions as well).

Suggestions for improvement

As you can see from the results sections above, our face mask detector is working quite well despite:

1. Having limited training data
2. The with_mask class being artificially generated (see the “How was our face mask dataset created?” section above).

To improve our face mask detection model further, you should gather actual images (rather than artificially generated images) of people wearing masks.

While our artificial dataset worked well in this case, there’s no substitute for the real thing.

Secondly, you should also gather images of faces that may “confuse” our classifier into thinking the person is wearing a mask when in fact they are not — potential examples include shirts wrapped around faces, bandana over the mouth, etc.

All of these are examples of something that could be confused as a face mask by our face mask detector.

Finally, you should consider training a dedicated two-class object detector rather than a simple image classifier.

Our current method of detecting whether a person is wearing a mask or not is a two-step process:

1. Step #1: Perform face detection
2. Step #2: Apply our face mask detector to each face

The problem with this approach is that a face mask, by definition, obscures part of the face. If enough of the face is obscured, the face cannot be detected, and therefore, the face mask detector will not be applied.

First, the object detector will be able to naturally detect people wearing masks that otherwise would have been impossible for the face detector to detect due to too much of the face being obscured.

Secondly, this approach reduces our computer vision pipeline to a single step — rather than applying face detection and then our face mask detector model, all we need to do is apply the object detector to give us bounding boxes for people both with_mask and without_mask in a single forward pass of the network.

Not only is such a method more computationally efficient, it’s also more “elegant” and end-to-end.

What’s next?

Inside today’s tutorial, we covered training a face mask detector. If you’re inspired to create your own deep learning projects, I would recommend reading my book, Deep Learning for Computer Vision with Python.

I crafted my book so that it perfectly balances theory with implementation, ensuring you properly master:

• Deep learning fundamentals and theory without unnecessary mathematical fluff. I present the basic equations and back them up with code walkthroughs that you can implement and easily understand. You don’t need a degree in advanced mathematics to understand this book
• How to implement your own custom neural network architectures. Not only will you learn how to implement state-of-the-art architectures, including ResNet, SqueezeNet, etc., but you’ll also learn how to create your own custom CNNs
• How to train CNNs on your own datasets. Most deep learning tutorials don’t teach you how to work with your own custom datasets. Mine do. You’ll be training CNNs on your own datasets in no time
• Object detection (Faster R-CNNs, Single Shot Detectors, and RetinaNet) and instance segmentation (Mask R-CNN). Use these chapters to create your own custom object detectors and segmentation networks

You’ll also find answers and proven code recipes to:

• Create and prepare your own custom image datasets for image classification, object detection, and segmentation
• Hands-on tutorials (with lots of code) that not only show you the algorithms behind deep learning for computer vision but their implementations as well
• Put my tips, suggestions, and best practices into action, ensuring you maximize the accuracy of your models

My readers enjoy my no-nonsense teaching style that is guaranteed to help you master deep learning for image understanding and visual recognition.

If you’re ready to dive in, just click here. And if you aren’t convinced yet, just grab my free PDF of sample chapters and the entire table of contents by filling the form in the lower right of this page.

Summary

In this tutorial, you learned how to create a COVID-19 face mask detector using OpenCV, Keras/TensorFlow, and Deep Learning.

To create our face mask detector, we trained a two-class model of people wearing masks and people not wearing masks.

We fine-tuned MobileNetV2 on our mask/no mask dataset and obtained a classifier that is ~99% accurate.

We then took this face mask classifier and applied it to both images and real-time video streams by:

1. Detecting faces in images/video
2. Extracting each individual face
3. Applying our face mask classifier

Our face mask detector is accurate, and since we used the MobileNetV2 architecture, it’s also computationally efficient, making it easier to deploy the model to embedded systems (Raspberry Pi, Google Coral, Jetosn, Nano, etc.).

I hope you enjoyed this tutorial!

To download the source code to this post (including the pre-trained COVID-19 face mask detector model), just enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post COVID-19: Face Mask Detector with OpenCV, Keras/TensorFlow, and Deep Learning appeared first on PyImageSearch.

In this tutorial, we will learn how to apply Computer Vision, Deep Learning, and OpenCV to identify potential child soldiers through automatic age detection and military fatigue recognition.

Military service is something of personal importance to me, something I consider honorable and admirable. That’s precisely the reason why this project, leveraging technology to identify child soldiers, is something I feel strongly about — nobody should be forced to serve, and especially young children.

You see, the military has always been a big part of my family growing up, even though I did not personally serve.

• My great-grandfather was an infantryman during WWI
• My grandfather served as a cook/baker in the Army for over 25 years
• My dad served in the U.S. Army for eight years, studying infectious diseases toward the end/immediately following the Vietnam War
• My cousin joined the Marines right out of high school and did two tours in Afghanistan before he was honorably discharged

Even outside my direct family, the military was still part of my life and community. I went to high school in a rural area of Maryland. There wasn’t much opportunity post-high school, with only three real paths:

1. Become a farmer — which many did, working on their respective family farms until they ultimately inherited it
2. Try to make it in college — a reasonable chunk of people choose this path, but either they or their families incur massive debt in the process
3. Join the military — get paid, learn practical skills that can transfer to real-world jobs, have up to $50K to pay for college through the GI Bill (which could also be partly transferred to your spouse or kids), and if deployed, get additional benefits (of course, at the risk of loss of life)

If you didn’t want to become a farmer or work in agriculture, that really only left two options — go to college or join the military. And for some, college just didn’t make sense. It wasn’t worth the expense.

If I’m recalling correctly, before I graduated from high school, at least 10 kids from my class enlisted, some of whom I knew personally and had classes with.

Growing up like I did, I have an immense amount of respect for the military, especially those who have served their country, regardless of what country they may be from. Whether you served in the United States, Finland, Japan, etc., serving your country is a big deal, and I respect all those that have.

I don’t have many regrets in my life, but truly, looking back, not serving is one of them. I really wish I had spent four years in the military, and each time I reflect on it, I still get pangs of regret and guilt.

That said, choosing not to serve was my choice.

The majority of us have a choice in our service — though of course there are complex social issues, such as poverty or ethical considerations, that go beyond the scope of this blog post. However, the bottom line is that young children should never be forced into enlisting.

There are parts of the world where kids don’t get to choose. Due to extreme poverty, terrorism, government propaganda, and/or manipulation, children are forced to fight.

This fighting doesn’t always involve a weapon either. In war, children can be used as spies/informants, messengers, human shields, and even as bargaining pieces.

Whether it be firing a weapon or serving as a pawn in a larger game, child soldiers incur lasting health effects, including but not limited to (source):

• Mental illness (chronic stress, anxiety, PTSD, etc.)
• Poor literacy
• Higher risk of poverty
• Unemployment as adults
• Alcohol and drug abuse
• Higher risk of suicide

Children serving in the military isn’t a new phenomenon either. It’s a tale as old as time:

• The Children’s Crusade in 1212 was infamous for enlisting children. Some died, but many others were sold into slavery
• Napoleon enlisted children in his army
• Children were used throughout WWI and WWII
• Enemy at the Gates, the 1973 nonfiction book and later 2001 film, told the story of The Battle of Stalingrad, and more specifically, a fictionalized Vasily Zaitsev, a famous Soviet sniper. In that book/movie, Sasha Filippov, a child, was used as a spy and informant. Sasha would routinely befriend Nazis and then feed information back to the Soviets. Sasha was later caught by the Nazis and killed
• And in the modern day, we are all too familiar with terrorist organizations such as Al-Qaeda and ISIS enlisting vulnerable kids into their efforts

While many of us would agree that using children during a war is unacceptable, the fact that children still end up participating in wars is a more complicated matter. When your life is on the line, when your family is hungry, when those around you are dying, it becomes a matter of life and death.

Fight or die.

It’s a sad reality, but it’s something that we can improve (and ideally resolve) through proper education, and slowly, incrementally, make the world a safer, better place.

In the meantime, we can use a bit of Computer Vision and Deep Learning to help identify potential child soldiers, both on the battlefield and in less than savory countries or organizations where they are being educated/indoctrinated.

Today’s post is the culmination of my past few months’ work after I was introduced to Victor Gevers (an esteemed ethical hacker) from the GDI.Foundation.

Victor and his team identified leaks in classroom facial recognition software that was being used to verify children were in attendance. When examining those photos, it appeared that some of these children were receiving military education and training (i.e., kids wearing military fatigues and other evidence that I’m not comfortable posting).

I’m not going to discuss the specifics of the politics, countries, or organizations involved — that’s not my place, and it’s entirely up to Victor and his team on how they will handle that particular situation.

Instead, I’m here to report on the science and the algorithms.

The field of Computer Vision and Deep Learning rightfully receives some deserved criticism for allowing powerful governments and organizations to create “Big Brother”-like police states where a watchful eye is always present.

That said, CV/DL can be used to “watch the watchers.” There will always be organizations and countries that try to surveil us. We can use CV/DL in turn as a form of accountability, keeping them responsible for their actions. And yes, it can be used to save lives when applied correctly.

To learn about an ethical application of Computer Vision and Deep Learning, specifically identifying child soldiers through automatic age and military fatigue detection, just keep reading!

Looking for the source code to this post?

Jump Right To The Downloads Section

An Ethical Application of Computer Vision and Deep Learning — Identifying Child Soldiers Through Automatic Age and Military Fatigue Detection

In the first part of this tutorial, we’ll discuss how I became involved in this project.

From there, we’ll take a look at four steps to identifying potential child soldiers in images and video streams.

Once we understand our basic algorithm, we’ll then implement our child soldier detection method using Python, OpenCV, and Keras/TensorFlow.

We’ll wrap up the tutorial by examining the results of our work.

Children in the military, child soldiers, and human rights — my attempt to help the cause

I first became involved in this project back in mid-January when I was connected with Victor Gevers from the GDI.Foundation.

Victor and his team discovered a data leak in software used for classroom facial recognition (i.e., a “smart attendance system” that automatically takes attendance based on face recognition).

This data leak exposed millions of children’s records that included ID card numbers, GPS locations, and yes, even the face photos themselves.

Any type of data leak is a concern, but a leak that exposes children is severe to say the least.

Unfortunately, the matter got worse.

Upon inspecting the photos from the leak, a concentration of children wearing military fatigues was found.

That immediately raised some eyebrows.

Victor and I connected and briefly exchanged emails regarding using my knowledge and expertise to help out.

Victor and his team needed a method to automatically detect faces, determine their age, and determine if the person was wearing military fatigues or not.

I agreed, provided that I could ethically share the results of the science (not the politics, countries, or organizations involved) as a form of education to help others learn from real-world problems.

There are organizations, coalitions, and individuals well more suited than me to properly handle the humanitarian side — and while I’m an expert in CV/DL, I am not an expert in politics or humanitarian efforts (although I do try my best to educate myself and make the best decisions I possibly can).

I hope you treat this article as a form of education. It is by no means a disclosure of countries or organizations involved, and I have made sure that none of the original training data or example images is provided in this tutorial. All original data has either been removed or properly anonymized.

How can we detect and identify potential child soldiers with computer vision and deep learning?

Detecting and identifying potential child soldiers is a four-step process:

1. Step #1 – Face Detection: Apply face detection to localize faces in the input images/video streams
2. Step #2 – Age Detection: Utilize deep learning-based age detectors to determine the age of the person detected via Step #1
3. Step #3 – Military Fatigue Detection: Apply deep learning to automatically detect camouflage or other indications of a military uniform
4. Step #4 – Combine Results: Take the results from Step #2 and Step #3 to determine if a child is potentially wearing military fatigues, which may be an indication of a child soldier, depending on the origin and context of the original image

If you’ve noticed, I’ve been purposely covering these topics on the PyImageSearch blog over the past 1-1.5 months, building up to this blog post.

We’ll do a quick review of each of the four steps below, but I suggest you use the links above to gain more detail on each of the steps involved.

Step #1: Detect faces in images or video streams

Before we can determine if a child is in an image or video stream, we first need to detect faces.

Face detection is the process of automatically locating where in an image a face is.

We’ll be using OpenCV’s deep learning-based face detector in this tutorial, but you could just as easily swap in Haar cascades, HOG + Linear SVM, or any number of other face detection methods.

Step #2: Take the face ROIs and perform age detection

Once we have localized each of the faces in the image/video stream, we can determine their age.

We’ll be using the age detector trained by Levi and Hassner in their 2015 publication, Age and Gender Classification using Convolutional Neural Networks.

This age detection model is compatible with OpenCV, as discussed in this tutorial.

Step #3: Train a camouflage/military fatigue detector, and apply it to the image

An indicator of a potential child soldier could be wearing military fatigues, which typically includes some sort of camouflage-like pattern.

Training a camouflage detector was covered in a previous tutorial — we’ll be using the trained model here today.

Step #4: Combine results of models, and look for children under 18 wearing military fatigues

The final step is to combine the results from our age detector with our military fatigue/camouflage detector.

If we (1) detect a person under the age of 18 in the photo, and (2) there also appears to be camouflage in the image, we’ll log that image to disk for further review.

Configuring your development environment

To configure your system for this tutorial, I first recommend following either of these tutorials:

• How to install TensorFlow 2.0 on Ubuntu
• How to install TensorFlow 2.0 on macOS

Either tutorial will help you configure you system with all the necessary software for this blog post with one exception. You also need to install the progressbar2 package into your virtual environment via:

$ workon dl4cv
$ pip install progressbar2

Once your system is configured, you are ready to move on with the rest of the tutorial.

Project structure

Be sure to grab the files for today’s tutorial from the “Downloads” section. Our project is organized as follows:

$ tree --dirsfirst
.
├── models
│   ├── age_detector
│   │   ├── age_deploy.prototxt
│   │   └── age_net.caffemodel
│   ├── camo_detector
│   │   └── camo_detector.model
│   └── face_detector
│       ├── deploy.prototxt
│       └── res10_300x300_ssd_iter_140000.caffemodel
├── output
│   ├── ages.csv
│   └── camo.csv
├── pyimagesearch
│   ├── __init__.py
│   ├── config.py
│   └── helpers.py
├── parse_results.py
└── process_dataset.py

6 directories, 12 files

The models/ directory contains each of our pre-trained deep learning models:

• Face detector
• Age classifier
• Camouflage classifier

Note: I cannot supply the original dataset used in this tutorial in the “Downloads” section of the guide as I normally would. That dataset is sensitive and cannot be distributed under any means.

Our configuration file

Before we get too far into our implementation, let’s first define a simple configuration file to store file paths to our face detector, age detector, and camouflage detector models, respectively.

Open up the config.py file in your project directory structure, and insert the following code:

# import the necessary packages
import os

# define the path to our face detector model
FACE_PROTOTXT = os.path.sep.join(["models", "face_detector",
	"deploy.prototxt"])
FACE_WEIGHTS = os.path.sep.join(["models", "face_detector",
	"res10_300x300_ssd_iter_140000.caffemodel"])

# define the path to our age detector model
AGE_PROTOTXT = os.path.sep.join(["models", "age_detector",
	"age_deploy.prototxt"])
AGE_WEIGHTS = os.path.sep.join(["models", "age_detector",
	"age_net.caffemodel"])

# define the path to our camo detector model
CAMO_MODEL = os.path.sep.join(["models", "camo_detector",
	"camo_detector.model"])

By making our config a Python file and by using the os module, we are able to build OS-agnostic paths directly.

Our config contains:

• Face detector model paths (Lines 5-8); be sure to read my deep learning face detector tutorial
• Age detector paths (Lines 11-14); take a moment to read my deep learning age detection tutorial in which this model was introduced
• Camouflage detector model path (Lines 17 and 18); read all about camouflage clothing classification with deep learning

With each of these paths defined, we’re ready to define convenience functions in a separate Python file in the next section.

Convenience functions for face detection, age prediction, camouflage detection, and face anonymization

To complete this project, we’ll be using a number of computer vision/deep learning techniques covered in previous tutorials, including:

• Face detection
• Age detection
• Camouflage detection
• Face anonymization

Let’s now define convenience functions for each technique in a central place for our child soldier detection project.

Note: For a more detailed review of face detection, face anonymization, age detection, and camouflage clothing detection, be sure to click on the corresponding link above.

Open up the helpers.py file in the pyimagesearch module, and insert the following code used to detect faces and predict age in the input image:

# import the necessary packages
import numpy as np
import cv2

def detect_and_predict_age(image, faceNet, ageNet, minConf=0.5):
	# define the list of age buckets our age detector will predict
	# and then initialize our results list
	AGE_BUCKETS = ["(0-2)", "(4-6)", "(8-12)", "(15-20)", "(25-32)",
		"(38-43)", "(48-53)", "(60-100)"]
	results = []

Our helper utilities only require OpenCV and NumPy (Lines 2 and 3).

Let’s go ahead and perform face detection:

	# grab the dimensions of the image and then construct a blob
	# from it
	(h, w) = image.shape[:2]
	blob = cv2.dnn.blobFromImage(image, 1.0, (300, 300),
		(104.0, 177.0, 123.0))

	# pass the blob through the network and obtain the face detections
	faceNet.setInput(blob)
	detections = faceNet.forward()

	# loop over the detections
	for i in range(0, detections.shape[2]):
		# extract the confidence (i.e., probability) associated with
		# the prediction
		confidence = detections[0, 0, i, 2]

		# filter out weak detections by ensuring the confidence is
		# greater than the minimum confidence
		if confidence > minConf:
			# compute the (x, y)-coordinates of the bounding box for
			# the object
			box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
			(startX, startY, endX, endY) = box.astype("int")

			# extract the ROI of the face
			face = image[startY:endY, startX:endX]

			# ensure the face ROI is sufficiently large
			if face.shape[0] < 20 or face.shape[1] < 20:
				continue

Lines 23-37 loop over detections, ensure high confidence, and extract a face ROI while ensuring it is sufficiently large for two reasons:

• First, we want to filter out false-positive face detections in the image
• Second, age classification results won’t be accurate for faces that are far away from the camera (i.e., perceivably small)

To finish out our face detection and age prediction helper utility, we’ll perform face prediction:

			# construct a blob from *just* the face ROI
			faceBlob = cv2.dnn.blobFromImage(face, 1.0, (227, 227),
				(78.4263377603, 87.7689143744, 114.895847746),
				swapRB=False)

			# make predictions on the age and find the age bucket with
			# the largest corresponding probability
			ageNet.setInput(faceBlob)
			preds = ageNet.forward()
			i = preds[0].argmax()
			age = AGE_BUCKETS[i]
			ageConfidence = preds[0][i]

			# construct a dictionary consisting of both the face
			# bounding box location along with the age prediction,
			# then update our results list
			d = {
				"loc": (startX, startY, endX, endY),
				"age": (age, ageConfidence)
			}
			results.append(d)

	# return our results to the calling function
	return results

def detect_camo(image, camoNet):
	# initialize (1) the class labels the camo detector can predict
	# and (2) the ImageNet means (in RGB order)
	CLASS_LABELS = ["camouflage_clothes", "normal_clothes"]
	MEANS = np.array([123.68, 116.779, 103.939], dtype="float32")

	# resize the image to 224x224 (ignoring aspect ratio), convert
	# the image from BGR to RGB ordering, and then add a batch
	# dimension to the volume
	image = cv2.resize(image, (224, 224))
	image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
	image = np.expand_dims(image, axis=0).astype("float32")

	# perform mean subtraction
	image -= MEANS

	# make predictions on the input image and find the class label
	# with the largest corresponding probability
	preds = camoNet.predict(image)[0]
	i = np.argmax(preds)

	# return the class label and corresponding probability
	return (CLASS_LABELS[i], preds[i])

def anonymize_face_pixelate(image, blocks=3):
	# divide the input image into NxN blocks
	(h, w) = image.shape[:2]
	xSteps = np.linspace(0, w, blocks + 1, dtype="int")
	ySteps = np.linspace(0, h, blocks + 1, dtype="int")

	# loop over the blocks in both the x and y direction
	for i in range(1, len(ySteps)):
		for j in range(1, len(xSteps)):
			# compute the starting and ending (x, y)-coordinates
			# for the current block
			startX = xSteps[j - 1]
			startY = ySteps[i - 1]
			endX = xSteps[j]
			endY = ySteps[i]

			# extract the ROI using NumPy array slicing, compute the
			# mean of the ROI, and then draw a rectangle with the
			# mean RGB values over the ROI in the original image
			roi = image[startY:endY, startX:endX]
			(B, G, R) = [int(x) for x in cv2.mean(roi)[:3]]
			cv2.rectangle(image, (startX, startY), (endX, endY),
				(B, G, R), -1)

	# return the pixelated blurred image
	return image

For face anonymization, we’ll use a pixelated type of face blurring. This method is typically what most people think of when they hear “face blurring” — it’s the same type of face blurring you’ll see on the evening news, mainly because it’s a bit more “aesthetically pleasing” to the eye than a simpler Gaussian blur (which is indeed a bit “jarring”).

Implementing our potential child soldier detector using OpenCV and Keras/TensorFlow

With both our configuration file and helper functions in place, let’s move on to applying them to a dataset of images that potentially contains child soldiers.

Open up the process_dataset.py script, and insert the following code:

# import the necessary packages
from pyimagesearch.helpers import detect_and_predict_age
from pyimagesearch.helpers import detect_camo
from pyimagesearch import config
from tensorflow.keras.models import load_model
from imutils import paths
import progressbar
import argparse
import cv2
import os

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d", "--dataset", required=True,
	help="path to input directory of images to process")
ap.add_argument("-o", "--output", required=True,
	help="path to output directory where CSV files will be stored")
args = vars(ap.parse_args())

# initialize a dictionary that will store output file pointers for
# our age and camo predictions, respectively
FILES = {}

# loop over our two types of output predictions
for k in ("ages", "camo"):
	# construct the output file path for the CSV file, open a path to
	# the file pointer, and then store it in our files dictionary
	p = os.path.sep.join([args["output"], "{}.csv".format(k)])
	f = open(p, "w")
	FILES[k] = f

Both file pointers are opened for writing in the process.

At this point, we’ll initialize three deep learning models:

# load our serialized face detector, age detector, and camo detector
# from disk
print("[INFO] loading trained models...")
faceNet = cv2.dnn.readNet(config.FACE_PROTOTXT, config.FACE_WEIGHTS)
ageNet = cv2.dnn.readNet(config.AGE_PROTOTXT, config.AGE_WEIGHTS)
camoNet = load_model(config.CAMO_MODEL)

# grab the paths to all images in our dataset
imagePaths = sorted(list(paths.list_images(args["dataset"])))
print("[INFO] processing {} images".format(len(imagePaths)))

# initialize the progress bar
widgets = ["Processing Images: ", progressbar.Percentage(), " ",
	progressbar.Bar(), " ", progressbar.ETA()]
pbar = progressbar.ProgressBar(maxval=len(imagePaths),
	widgets=widgets).start()

We’re now to the heart of our dataset processing script. We’ll begin looping over all images to detect faces, predict ages, and determine if there is camouflage present:

# loop over the image paths
for (i, imagePath) in enumerate(imagePaths):
	# load the image from disk
	image = cv2.imread(imagePath)

	# if the image is 'None', then it could not be properly read from
	# disk (so we should just skip it)
	if image is None:
		continue

	# detect all faces in the input image and then predict their
	# perceived age based on the face ROI
	ageResults = detect_and_predict_age(image, faceNet, ageNet)

	# use our camo detection model to detect whether camouflage exists in
	# the image or not
	camoResults = detect_camo(image, camoNet)

	# loop over the age detection results
	for r in ageResults:
		# the output row for the ages CSV consists of (1) the image
		# file path, (2) bounding box coordinates of the face, (3)
		# the predicted age, and (4) the corresponding probability
		# of the age prediction
		row = [imagePath, *r["loc"], r["age"][0], r["age"][1]]
		row = ",".join([str(x) for x in row])

		# write the row to the age prediction CSV file
		FILES["ages"].write("{}\n".format(row))
		FILES["ages"].flush()

	# check to see if our camouflage predictor was triggered
	if camoResults[0] == "camouflage_clothes":
		# the output row for the camo CSV consists of (1) the image
		# file path and (2) the probability of the camo prediction
		row = [imagePath, camoResults[1]]
		row = ",".join([str(x) for x in row])

		# write the row to the camo prediction CSV file
		FILES["camo"].write("{}\n".format(row))
		FILES["camo"].flush()

	# update the progress bar
	pbar.update(i)

# stop the progress bar
pbar.finish()
print("[INFO] cleaning up...")

# loop over the open file pointers and close them
for f in FILES.values():
	f.close()

Line 92 updates our progress bar, at which point, we’ll process the next image in the dataset from the top of the loop.

Lines 95-100 stop the progress bar and close the CSV file pointers.

Great job implementing your dataset processing script. In the next section we’ll put it to work!

Processing our dataset of potential child soldiers

$ time python process_dataset.py --dataset VictorGevers_Dataset --output output
[INFO] loading trained models...
[INFO] processing 56037 images
Processing Images: 100% |############################| Time:  1:49:48
[INFO] cleaning up...

real	109m53.034s
user	428m1.900s
sys   306m23.741s

This dataset was supplied by Victor Gevers (i.e., the dataset that was obtained during the data leakage).

Processing the entire dataset took almost two hours on my 3 GHz Intel Xeon W processor — a GPU would have made it even faster.

Of course, I cannot supply the original dataset used in this tutorial in the “Downloads” section of the guide as I normally would. That dataset is private, sensitive, and cannot be distributed under any means.

After the script finished executing, I had two CSV files in my output directory:

$ ls output/
ages.csv	camo.csv

Here is a sample output of ages.csv:

$ tail output/ages.csv 
rBIABl3RztuAVy6gAAMSpLwFcC0051.png,661,1079,1081,1873,(48-53),0.6324904
rBIABl3RzuuAbzmlAAUsBPfvHNA217.png,546,122,1081,1014,(8-12),0.59567857
rBIABl3RzxKAaJEoAAdr1POcxbI556.png,4,189,105,349,(48-53),0.49577188
rBIABl3RzxmAM6nvAABRgKCu0g4069.png,104,76,317,346,(8-12),0.31842607
rBIABl3RzxmAM6nvAABRgKCu0g4069.png,236,246,449,523,(60-100),0.9929517
rBIABl3RzxqAbJZVAAA7VN0gGzg369.png,41,79,258,360,(38-43),0.63570714
rBIABl3RzxyABhCxAAav3PMc9eo739.png,632,512,1074,1419,(48-53),0.5355053
rBIABl3RzzOAZ-HuAAZQoGUjaiw399.png,354,56,1089,970,(60-100),0.48260492
rBIABl3RzzOAZ-HuAAZQoGUjaiw399.png,820,475,1540,1434,(4-6),0.6595153
rBIABl3RzzeAb1lkAAdmVBqVDho181.png,258,994,826,2542,(15-20),0.3086191

As you can see, each row contains:

1. The image file path
2. Bounding box coordinates of a particular face
3. The age range prediction for that face and associated probability

And below we have a sample of the output from camo.csv:

$ tail output/camo.csv 
rBIABl3RY-2AYS0RAAaPGGXk-_A001.png,0.9579516
rBIABl3Ra4GAScPBAABEYEkNOcQ818.png,0.995684
rBIABl3Rb36AMT9WAABN7PoYIew817.png,0.99894327
rBIABl3Rby-AQv5MAAB8CPkzp58351.png,0.9577539
rBIABl3Re6OALgO5AABY5AH5hJc735.png,0.7973979
rBIABl3RvkuAXeryAABlfL8vLL4072.png,0.7121747
rBIABl3RwaOAFX21AABy6JNWkVY010.png,0.97816855
rBIABl3Rz-2AUOD0AAQ3eMMg8gg856.png,0.8256913
rBIABl3RztOAeFb1AABG-K96F_c092.png,0.50594944
rBIABl3RzxeAGI5XAAfg5J_Svmc027.png,0.98626024

This CSV file has less information, containing only:

1. The image file path
2. The probability indicating whether the image contains camouflage

We now have both our age and camouflage predictions.

But how do we combine these predictions to determine whether a particular image has a potential child soldier?

I’ll answer that question in the next section.

Implementing a Python script to parse the results of our detections

You could easily output the child soldier data to another CSV file and provide it to a reporting agency if you were doing this type of work.

Rather than that, the script we’re going to develop simply anonymizes faces (i.e., applies our pixelated blur method) in the suspected child soldier image and displays the results on screen.

Let’s take a look at parse_results.py now:

# import the necessary packages
from pyimagesearch.helpers import anonymize_face_pixelate
import numpy as np
import argparse
import imutils
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-a", "--ages", required=True,
	help="path to input ages CSV file")
ap.add_argument("-c", "--camo", required=True,
	help="path to input camo CSV file")
args = vars(ap.parse_args())

# load the contents of the ages and camo CSV files
ageRows = open(args["ages"]).read().strip().split("\n")
camoRows = open(args["camo"]).read().strip().split("\n")

# initialize two dictionaries, one to store the age results and the
# other to store the camo results, respectively
ages = {}
camo = {}

# loop over the age rows
for row in ageRows:
	# parse the row
	row = row.split(",")
	imagePath = row[0]
	bbox = [int(x) for x in row[1:5]]
	age = row[5]
	ageProb = float(row[6])

	# construct a tuple that consists of the bounding box coordinates,
	# age, and age probability
	t = (bbox, age, ageProb)

	# update our ages dictionary to use the image path as the key and
	# the detection information as a tuple
	l = ages.get(imagePath, [])
	l.append(t)
	ages[imagePath] = l

# loop over the camo rows
for row in camoRows:
	# parse the row
	row = row.split(",")
	imagePath = row[0]
	camoProb = float(row[1])

	# update our camo dictionary to use the image path as the key and
	# the camouflage probability as the value
	camo[imagePath] = camoProb

# find all image paths that exist in *BOTH* the age dictionary and
# camo dictionary
inter = sorted(set(ages.keys()).intersection(camo.keys()))

# loop over all image paths in the intersection
for imagePath in inter:
	# load the input image and grab its dimensions
	image = cv2.imread(imagePath)
	(h, w) = image.shape[:2]

	# if the width is greater than the height, resize along the width
	# dimension
	if w > h:
		image = imutils.resize(image, width=600)

	# otherwise, resize the image along the height
	else:
		image = imutils.resize(image, height=600)

	# compute the resize ratio, which is the ratio between the *new*
	# image dimensions to the *old* image dimensions
	ratio = image.shape[1] / float(w)

Computing the ratio between the new image dimensions and the old image dimensions (Line 76) allows us the scale our face bounding boxes in the next code block.

Let’s loop over the age predictions for this particular image:

	# loop over the age predictions for this particular image
	for (bbox, age, ageProb) in ages[imagePath]:
		# extract the bounding box coordinates of the face detection
		bbox = [int(x) for x in np.array(bbox) * ratio]
		(startX, startY, endX, endY) = bbox

		# anonymize the face
		face = image[startY:endY, startX:endX]
		face = anonymize_face_pixelate(face, blocks=5)
		image[startY:endY, startX:endX] = face

		# set the color for the annotation to *green*
		color = (0, 255, 0)

		# override the color to *red* they are potential child soldier
		if age in ["(0-2)", "(4-6)", "(8-12)", "(15-20)"]:
			color = (0, 0,  255)

		# draw the bounding box of the face along with the associated
		# predicted age
		text = "{}: {:.2f}%".format(age, ageProb * 100)
		y = startY - 10 if startY - 10 > 10 else startY + 10
		cv2.rectangle(image, (startX, startY), (endX, endY), color, 2)
		cv2.putText(image, text, (startX, y),
			cv2.FONT_HERSHEY_SIMPLEX, 0.45, color, 2)

Let’s take our visualization a step further and also annotate the probability of camouflage in the top left corner of the image:

	# draw the camouflage prediction probability on the image
	label = "camo: {:.2f}%".format(camo[imagePath] * 100)
	cv2.rectangle(image, (0, 0), (300, 40), (0, 0, 0), -1)
	cv2.putText(image, label, (10, 25), cv2.FONT_HERSHEY_SIMPLEX,
		0.8, (255, 255, 255), 2)

	# show the output image
	cv2.imshow("Image", image)
	cv2.waitKey(0)

Results: Using computer vision and deep learning for good

We are now ready to combine the results from our age prediction and camouflage output to determine if a particular image contains a potential child soldier.

To execute this script, I used the following command:

$ python parse_results.py --ages output/ages.csv --camo output/camo.csv

Note: For privacy concerns, even with face anonymization, I do not feel comfortable sharing the original images from the dataset Victor Gevers provided me with. I’ve included samples from other images online to demonstrate that the script is working properly. I hope you understand and appreciate why I made this decision.

Below is an example of an image containing a potential child soldier:

Here is a second image of our method detecting a potential child soldier:

The age prediction is a bit off here.

I would estimate this young lady (refer to Figure 1 for the original image) to be somewhere in the range of 12-16; however, our age predictor model is predicts 4-6 — the limitation of the age prediction model is discussed in the “Summary” section below.

What’s next?

This tutorial was the culmination of our series on face anonymization, age prediction, and camouflage detection on the PyImageSearch blog.

As you read this tutorial and looked at the figures, you may have a complex mixture of emotions, including sadness, remorse, and even anger.

I felt the same way as I was writing the code, running the experiments, and authoring this tutorial.

However, it is worth sharing so that people combating the challenging problem of children in unfortunate situations (be it child labor, child pornography, or child soldiers) have software tools and knowledge to do their jobs. Let the government and humanitarian organizations combat the problem with the aid of artificial intelligence.

As previously stated, the field of Computer Vision and Deep Learning rightfully receives some deserved criticism for allowing powerful governments and organizations to create “Big Brother”-like police states where a watchful eye is always present. That said, CV/DL can be used to “watch the watchers.” There will always be organizations and countries that try to surveil us. We can use CV/DL in turn as a form of accountability, keeping them responsible for their actions. And yes, it can be used to save lives when applied correctly.

If you are interested in solving complex visual problems that today’s world faces, my book Deep Learning for Computer Vision with Python is a great place to start.

I crafted my book so that it perfectly balances theory with implementation, ensuring you properly master:

• Deep learning fundamentals and theory without unnecessary mathematical fluff. I present the basic equations and back them up with code walkthroughs that you can implement and easily understand. You don’t need a degree in advanced mathematics to understand this book
• How to implement your own custom neural network architectures. Not only will you learn how to implement state-of-the-art architectures, including ResNet, SqueezeNet, etc., but you’ll also learn how to create your own custom CNNs
• How to train CNNs on your own datasets. Most deep learning tutorials don’t teach you how to work with your own custom datasets. Mine do. You’ll be training CNNs on your own datasets in no time
• Object detection (Faster R-CNNs, Single Shot Detectors, and RetinaNet) and instance segmentation (Mask R-CNN). Use these chapters to create your own custom object detectors and segmentation networks

You’ll also find answers and proven code recipes to:

• Create and prepare your own custom image datasets for image classification, object detection, and segmentation
• Work with hands-on tutorials (with lots of code) that not only show you the algorithms behind deep learning for computer vision but their implementations as well.
• Put my tips, suggestions, and best practices into action, ensuring you maximize the accuracy of your models

If you’re interested in grabbing the free table of contents and browsing a few sample chapters, just fill the form available by clicking here:

Summary

In this tutorial, you learned an ethical application of Computer Vision and Deep learning — identifying potential child soldiers.

To accomplish this task, we applied:

1. Age detection — used to detect the age of a person in an image
2. Camouflage/fatigue detection — used to detect whether camouflage was in the image, indicating that the person was likely wearing military fatigues

Our system is fairly accurate, but as I discuss in my age detection post, as well as the camouflage detection tutorial, results can be improved by:

1. Training a more accurate age detector with a balanced dataset
2. Gathering additional images of children to better identify age brackets for kids
3. Training a more accurate camouflage detector by applying more aggressive data augmentation and regularization techniques
4. Building a better military fatigue/uniform detector through clothing segmentation

I hope you enjoyed this tutorial — and I also hope you didn’t find the subject matter of this post too upsetting.

Computer vision and deep learning, just like nearly any product or science, can be used for good or evil. Try to stay on the good side however you can. The world is a scary place — let’s all work together to make a better.

To download the source code to this post (including the pre-trained face detector, age detector, and camouflage detector models), just enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post An Ethical Application of Computer Vision and Deep Learning — Identifying Child Soldiers Through Automatic Age and Military Fatigue Detection appeared first on PyImageSearch.

In this tutorial, you will learn how to perform image inpainting with OpenCV and Python.

Image inpainting is a form of image conservation and image restoration, dating back to the 1700s when Pietro Edwards, director of the Restoration of the Public Pictures in Venice, Italy, applied this scientific methodology to restore and conserve famous works (source).

Technology has advanced image painting significantly, allowing us to:

• Restore old, degraded photos
• Repair photos with missing areas due to damage and aging
• Mask out and remove particular objects from an image (and do so in an aesthetically pleasing way)

Today, we’ll be looking at two image inpainting algorithms that OpenCV ships “out-of-the-box.”

To learn how to perform image inpainting with OpenCV and Python, just keep reading!

Looking for the source code to this post?

Jump Right To The Downloads Section

Image inpainting with OpenCV and Python

In the first part of this tutorial, you’ll learn about OpenCV’s inpainting algorithms.

From there, we’ll implement an inpainting demo using OpenCV’s built-in algorithms, and then apply inpainting to a set of images.

Finally, we’ll review the results and discuss the next steps.

I’ll also be upfront and say that this tutorial is an introduction to inpainting including its basics, how it works, and what kind of results we can expect.

While this tutorial doesn’t necessarily “break new ground” in terms of inpainting results, it is an essential prerequisite to future tutorials because:

1. It shows you how to use inpainting with OpenCV
2. It provides you with a baseline that we can improve on
3. It shows some of the manual input required by traditional inpainting algorithms, which deep learning methods can now automate

OpenCV’s inpainting algorithms

To quote the OpenCV documentation, the Telea method:

… is based on Fast Marching Method. Consider a region in the image to be inpainted. Algorithm starts from the boundary of this region and goes inside the region gradually filling everything in the boundary first. It takes a small neighbourhood around the pixel on the neighbourhood to be inpainted. This pixel is replaced by normalized weighted sum of all the known pixels in the neighbourhood. Selection of the weights is an important matter. More weightage is given to those pixels lying near to the point, near to the normal of the boundary and those lying on the boundary contours. Once a pixel is inpainted, it moves to next nearest pixel using Fast Marching Method. FMM ensures those pixels near the known pixels are inpainted first, so that it just works like a manual heuristic operation.

The second method, Navier-Stokes, is based on fluid dynamics.

Again, quoting the OpenCV documentation:

This algorithm is based on fluid dynamics and utilizes partial differential equations. Basic principle is heurisitic [sic]. It first travels along the edges from known regions to unknown regions (because edges are meant to be continuous). It continues isophotes (lines joining points with same intensity, just like contours joins points with same elevation) while matching gradient vectors at the boundary of the inpainting region. For this, some methods from fluid dynamics are used. Once they are obtained, color is filled to reduce minimum variance in that area.

In the rest of this tutorial you will learn how to apply both the cv2.INPAINT_TELEA and cv2.INPAINT_NS methods using OpenCV.

How does inpainting work with OpenCV?

When applying inpainting with OpenCV, we need to provide two images:

1. The input image we wish to inpaint and restore. Presumably, this image is “damaged” in some manner, and we need to apply inpainting algorithms to fix it
2. The mask image, which indicates where in the image the damage is. This image should have the same spatial dimensions (width and height) as the input image. Non-zero pixels correspond to areas that should be inpainted (i.e., fixed), while zero pixels are considered “normal” and do not need inpainting

An example of these images can be seen in Figure 2 above.

The image on the left is our original input image. Notice how this image is old, faded, and damaged/ripped.

The image on the right is our mask image. Notice how white pixels in the mask mark where the damage is in the input image (left).

Finally, on the bottom, we have our output image after applying inpainting with OpenCV. Our old, faded, damaged image has now been partially restored.

How do we create the mask for inpainting with OpenCV?

At this point, the big question is:

“Adrian, how did you create the mask? Was that created programmatically? Or did you manually create it?”

For Figure 2 above (in the previous section), I had to manually create the mask. To do so, I opened up Photoshop (GIMP or another photo editing/manipulation tool would work just as well), and then used the Magic Wand tool and manual selection tool to select the damaged areas of the image.

I then flood-filled the selection area with white, left the background as black, and saved the mask to disk.

Doing so was a manual, tedious process — you may be able to programmatically define masks for your own images using image processing techniques such as thresholding, edge detection, and contours to mark damaged reasons, but realistically, there will likely be some sort of manual intervention.

The manual intervention is one of the primary limitations of using OpenCV’s built-in inpainting algorithms.

I discuss how we can improve upon OpenCV’s inpainting algorithms, including deep learning-based methods, in the “How can we improve OpenCV inpainting results?” section later in this tutorial.

Project structure

Scroll to the “Downloads” section of this tutorial and grab the .zip containing our code and images. The files are organized as follows:

$ tree --dirsfirst
.
├── examples
│   ├── example01.png
│   ├── example02.png
│   ├── example03.png
│   ├── mask01.png
│   ├── mask02.png
│   └── mask03.png
└── opencv_inpainting.py

1 directory, 7 files

Implementing inpainting with OpenCV and Python

# import the necessary packages
import argparse
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", type=str, required=True,
	help="path input image on which we'll perform inpainting")
ap.add_argument("-m", "--mask", type=str, required=True,
	help="path input mask which corresponds to damaged areas")
ap.add_argument("-a", "--method", type=str, default="telea",
	choices=["telea", "ns"],
	help="inpainting algorithm to use")
ap.add_argument("-r", "--radius", type=int, default=3,
	help="inpainting radius")
args = vars(ap.parse_args())

# initialize the inpainting algorithm to be the Telea et al. method
flags = cv2.INPAINT_TELEA

# check to see if we should be using the Navier-Stokes (i.e., Bertalmio
# et al.) method for inpainting
if args["method"] == "ns":
	flags = cv2.INPAINT_NS

# load the (1) input image (i.e., the image we're going to perform
# inpainting on) and (2) the  mask which should have the same input
# dimensions as the input image -- zero pixels correspond to areas
# that *will not* be inpainted while non-zero pixels correspond to
# "damaged" areas that inpainting will try to correct
image = cv2.imread(args["image"])
mask = cv2.imread(args["mask"])
mask = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)

# perform inpainting using OpenCV
output = cv2.inpaint(image, mask, args["radius"], flags=flags)

# show the original input image, mask, and output image after
# applying inpainting
cv2.imshow("Image", image)
cv2.imshow("Mask", mask)
cv2.imshow("Output", output)
cv2.waitKey(0)

We display three images on-screen: (1) our original damaged photograph, (2) our mask which highlights the damaged areas, and (3) the inpainted (i.e., restored) output photograph. Each of these images will remain on your screen until any key is pressed while one of the GUI windows is in focus.

OpenCV inpainting results

We are now ready to apply inpainting using OpenCV.

Make sure you have used the “Downloads” section of this tutorial to download the source code and example images.

From there, open a terminal, and execute the following command:

$ python opencv_inpainting.py --image examples/example01.png \
	--mask examples/mask01.png

On the left, you can see the original input image of my dog Janie, sporting an ultra punk/ska jean jacket.

I have purposely added the text “Adrian wuz here” to the image, the mask of which is shown in the middle.

$ python opencv_inpainting.py --image examples/example02.png \
	--mask examples/mask02.png --method ns

On the top, you can see an old photograph, which has been damaged. I then manually created a mask for the damaged areas on the right (using Photoshop as explained in the “How do we create the mask for inpainting with OpenCV?” section).

The bottom shows the output of the Navier-Stokes inpainting method. By applying this method of OpenCV inpainting, we have been able to partially repair the old, damaged photo.

Let’s try one final image:

$ python opencv_inpainting.py --image examples/example03.png \
	--mask examples/mask03.png

On the left, we have the original image, while on the right, we have the corresponding mask.

Notice that the mask has two areas that we’ll be trying to “repair”:

1. The watermark on the bottom-right
2. The circular area corresponds to one of the trees

In this example, we’re treating OpenCV inpainting as a method of removing objects from an image, the results of which can be seen on the bottom.

Unfortunately, results are not as good as we would have hoped for. The tree we wish to have removed appears as a circular blur, while the watermark is blurry as well.

That begs the question — what can we do to improve our results?

How can we improve OpenCV inpainting results?

One of the biggest problems with OpenCV’s built-in inpainting algorithms is that they require manual intervention, meaning that we have to manually supply the masked region we wish to fix and restore.

Manually supplying the mask is tedious — isn’t there a better way?

In fact, there is.

Using deep learning-based approaches, including fully-convolutional neural networks and Generative Adversarial Networks (GANs), we can “learn to inpaint.”

These networks:

• Require zero manual intervention
• Can generate their own training data
• Generate results that are more aesthetically pleasing than traditional computer vision inpainting algorithms

Deep learning-based inpainting algorithms are outside the scope of this tutorial but will be covered in a future blog post.

What’s next?

Are you interested in learning more about image processing, computer vision, and machine/deep learning?

If so, you’ll want to take a look at the PyImageSearch Gurus course.

I didn’t have the luxury of such a course in college.

I learned computer vision the hard way — a tale much like the one your grandparents tell in which they walked uphill both ways in four feet of snow each day on their way to school.

Back then, there weren’t great image processing blogs like PyImageSearch online to learn from. Of course there were theory and math intensive text books, complex research papers, and the occasional sit-down in my advisor’s office. But none of these resources taught computer vision systematically via practical use cases and Python code examples.

So what did I do?

I took what I learned and came up with my own examples and projects to learn from. It wasn’t easy, but by the end of it, I was confident that I knew computer vision well enough to consult for the NIH and build/deploy a couple iPhone apps to the App Store.

Now what does that mean for you?

You have the golden opportunity to learn from me in a central place with other motivated students. I’ve developed a course using my personal arsenal of code and my years of knowledge. You will learn concepts and code the way I wish I had earlier in my career.

Inside PyImageSearch Gurus, you’ll find:

• An actionable, real-world course on Computer Vision, Deep Learning, and OpenCV. Each lesson in PyImageSearch Gurus is taught in the same hands-on, easy-to-understand PyImageSearch style that you know and love
• The most comprehensive computer vision education online today. The PyImageSearch Gurus course covers 13 modules broken out into 168 lessons, with over 2,161 pages of content. You won’t find a more detailed computer vision course anywhere else online; I guarantee it
• A community of like-minded developers, researchers, and students just like you, who are eager to learn computer vision, level-up their skills, and collaborate on projects. I participate in the forums nearly every day. These forums are a great way to get expert advice, both from me as well as the more advanced students

Take a look at these previous students’ success stories — each of these students invested in themselves and have achieved success. You can too in a short time after you take the plunge by enrolling today.

If you’re on the fence, grab the course syllabus and 10 free sample lessons. If that sounds interesting to you, simply click this link:

Summary

These methods are traditional computer vision algorithms and do not rely on deep learning, making them easy and efficient to utilize.

However, while these algorithms are easy to use (since they are baked into OpenCV), they leave a lot to be desired in terms of accuracy.

Not to mention, having to manually supply the mask image, marking the damaged areas of the original photograph, is quite tedious.

In a future tutorial, we’ll look at deep learning-based inpainting algorithms — these methods require more computation and are a bit harder to code, but ultimately lead to better results (plus, there’s no mask image requirement).

To download the source code to this pots (and be notified when future tutorials are published here on PyImageSearch), just enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Image inpainting with OpenCV and Python appeared first on PyImageSearch.

In this tutorial, you will learn how to utilize Tesseract to detect, localize, and OCR text, all within a single, efficient function call.

Back in September, I showed you how to use OpenCV to detect and OCR text. This method was a three stage process:

1. Use OpenCV’s EAST text detection model to detect the presence of text in an image
2. Extract the text Region of Interest (ROI) from the image using basic image cropping/NumPy array slicing
3. Take the text ROI, and then pass it into Tesseract to actually OCR the text

Our method worked quite well but was a bit complicated and less efficient due to the multistage process.

PyImageSearch reader Bryan wonders if there is a better, more streamlined way:

Hi Adrian,

I noticed that OpenCV’s uses the EAST text detection model. I assume text detection also exists inside Tesseract?

If so, is there anyway we can utilize Tesseract to both detect the text and OCR it without having to call additional OpenCV functions?

You’re in luck, Bryan. Tesseract does have the ability to perform text detection and OCR in a single function call — and as you’ll find out, it’s quite easy to do!

To learn how to detect, localize, and OCR text with Tesseract, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Tesseract OCR: Text localization and detection

In the first part of this tutorial, we’ll discuss the concept of text detection and localization.

From there, I will show you how to install Tesseract on your system.

We’ll then implement text localization, detection, and OCR using Tesseract and Python.

Finally, we’ll review our results.

What is text localization and detection?

Text detection is the process of localizing where an image text is.

You can think of text detection as a specialized form of object detection.

In object detection, our goal is to (1) detect and compute the bounding box of all objects in an image and (2) determine the class label for each bounding box, similar to the image below:

In text detection, our goal is to automatically compute the bounding boxes for every region of text in an image:

Once we have those regions, we can then OCR them.

How to install pytesseract for Tesseract OCR

I have provided instructions for installing the Tesseract OCR engine as well as pytesseract (the Python bindings used to interface with Tesseract) in my blog post OpenCV OCR and text recognition with Tesseract.

Follow the instructions in the “How to install Tesseract 4” section of that tutorial, confirm your Tesseract install, and then come back here to learn how to detect and localize text with Tesseract.

Project structure

Go ahead and grab today’s .zip from the “Downloads” section of this blog post. Once you extract the files, you’ll be presented with an especially simple project layout:

% tree
.
├── apple_support.png
└── localize_text_tesseract.py

0 directories, 2 files

Implementing text localization, text detection, and OCR with Tesseract

# import the necessary packages
from pytesseract import Output
import pytesseract
import argparse
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
	help="path to input image to be OCR'd")
ap.add_argument("-c", "--min-conf", type=int, default=0,
	help="mininum confidence value to filter weak text detection")
args = vars(ap.parse_args())

We begin by importing packages, namely pytesseract and OpenCV. Be sure to refer to the “How to install pytesseract for Tesseract OCR” section above for installation links.

Next, we parse two command line arguments:

# load the input image, convert it from BGR to RGB channel ordering,
# and use Tesseract to localize each area of text in the input image
image = cv2.imread(args["image"])
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
results = pytesseract.image_to_data(rgb, output_type=Output.DICT)

# loop over each of the individual text localizations
for i in range(0, len(results["text"])):
	# extract the bounding box coordinates of the text region from
	# the current result
	x = results["left"][i]
	y = results["top"][i]
	w = results["width"][i]
	h = results["height"][i]

	# extract the OCR text itself along with the confidence of the
	# text localization
	text = results["text"][i]
	conf = int(results["conf"][i])

# filter out weak confidence text localizations
	if conf > args["min_conf"]:
		# display the confidence and text to our terminal
		print("Confidence: {}".format(conf))
		print("Text: {}".format(text))
		print("")

		# strip out non-ASCII text so we can draw the text on the image
		# using OpenCV, then draw a bounding box around the text along
		# with the text itself
		text = "".join([c if ord(c) < 128 else "" for c in text]).strip()
		cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
		cv2.putText(image, text, (x, y - 10), cv2.FONT_HERSHEY_SIMPLEX,
			1.2, (0, 0, 255), 3)

# show the output image
cv2.imshow("Image", image)
cv2.waitKey(0)

Great job performing OCR with Tesseract and pytesseract.

Tesseract text localization, text detection, and OCR results

We are now ready to perform text detection and localization with Tesseract!

Make sure you use the “Downloads” section of this tutorial to download the source code and example image.

From there, open up a terminal, and execute the following command:

$ python localize_text_tesseract.py --image apple_support.png
Confidence: 26
Text: a

Confidence: 96
Text: Apple

Confidence: 96
Text: Support

Confidence: 96

$ python localize_text_tesseract.py --image apple_support.png --min-conf 50
Confidence: 96
Text: Apple

Confidence: 96
Text: Support

Confidence: 96
Text: 1-800-275-2273

What’s next?

Today’s blog post was admittedly simple and straightforward, and my hope is that it gives you a little inspiration and confidence.

I’ll be blunt: Computer vision apps and services can be quite complex — much more so than today’s tutorial.

If you are interested in learning more about image processing, computer vision, and machine/deep learning, look no further than the PyImageSearch Gurus course and community.

Inside PyImageSearch Gurus, you’ll find:

• An actionable, real-world course on Computer Vision, Deep Learning, and OpenCV. Each lesson in PyImageSearch Gurus is taught in the same hands-on, easy-to-understand PyImageSearch style that you know and love
• The most comprehensive computer vision education online today. The PyImageSearch Gurus course covers 13 modules broken out into 168 lessons, with over 2,161 pages of content. You won’t find a more detailed computer vision course anywhere else online; I guarantee it
• A community of like-minded developers, researchers, and students just like you, who are eager to learn computer vision, level-up their skills, and collaborate on projects. I participate in the forums nearly every day. These forums are a great way to get expert advice, both from me as well as the more advanced students

Spend a moment reviewing these previous students’ success stories — each of these students invested in themselves and has achieved success. I have no doubt the same will be true for you once you enroll.

If you’d like more information, simply click here:

Summary

In this tutorial, you learned how to use Tesseract to detect text, localize it, and then OCR it.

The benefit of using Tesseract to perform text detection and OCR is that we can do so in just a single function call, making it easier than the multistage OpenCV OCR process.

That said, OCR is still an area of computer vision that is far from solved.

Whenever confronted with an OCR project, be sure to apply both methods and see which method gives you the best results — let your empirical results guide you.

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Tesseract OCR: Text localization and detection appeared first on PyImageSearch.

In this tutorial, you will learn how to implement a COVID-19 social distancing detector using OpenCV, Deep Learning, and Computer Vision.

Today’s tutorial is inspired by PyImageSearch reader Min-Jun, who emailed in asking:

Hi Adrian,

I’ve seen a number of people in the computer vision community implementing “social distancing detectors”, but I’m not sure how they work.

Would you consider writing a tutorial on the topic?

Thank you.

Min-Jun is correct — I’ve seen a number of social distancing detector implementations on social media, my favorite ones being from reddit user danlapko and Rohit Kumar Srivastava’s implementation.

Today, I’m going to provide you with a starting point for your own social distancing detector. You can then extend it as you see fit to develop your own projects.

To learn how to implement a social distancing detector with OpenCV, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

OpenCV Social Distancing Detector

In the first part of this tutorial, we’ll briefly discuss what social distancing is and how OpenCV and deep learning can be used to implement a social distancing detector.

We’ll then review our project directory structure including:

1. Our configuration file used to keep our implementation neat and tidy
2. Our detect_people utility function, which detects people in video streams using the YOLO object detector
3. Our Python driver script, which glues all the pieces together into a full-fledged OpenCV social distancing detector

We’ll wrap up the post by reviewing the results, including a brief discussion on limitations and future improvements.

What is social distancing?

Social distancing is a method used to control the spread of contagious diseases.

As the name suggests, social distancing implies that people should physically distance themselves from one another, reducing close contact, and thereby reducing the spread of a contagious disease (such as coronavirus):

Social distancing is not a new concept, dating back to the fifth century (source), and has even been referenced in religious texts such as the Bible:

And the leper in whom the plague is … he shall dwell alone; [outside] the camp shall his habitation be. — Leviticus 13:46

Social distancing is arguably the most effective nonpharmaceutical way to prevent the spread of a disease — by definition, if people are not close together, they cannot spread germs.

Using OpenCV, computer vision, and deep learning for social distancing

We can use OpenCV, computer vision, and deep learning to implement social distancing detectors.

The steps to build a social distancing detector include:

1. Apply object detection to detect all people (and only people) in a video stream (see this tutorial on building an OpenCV people counter)
2. Compute the pairwise distances between all detected people
3. Based on these distances, check to see if any two people are less than N pixels apart

For the most accurate results, you should calibrate your camera through intrinsic/extrinsic parameters so that you can map pixels to measurable units.

An easier alternative (but less accurate) method would be to apply triangle similarity calibration (as discussed in this tutorial).

Both of these methods can be used to map pixels to measurable units.

Finally, if you do not want/cannot apply camera calibration, you can still utilize a social distancing detector, but you’ll have to rely strictly on the pixel distances, which won’t necessarily be as accurate.

For the sake of simplicity, our OpenCV social distancing detector implementation will rely on pixel distances — I will leave it as an exercise for you, the reader, to extend the implementation as you see fit.

Project structure

Be sure to grab the code from the “Downloads” section of this blog post. From there, extract the files, and use the tree command to see how our project is organized:

$ tree --dirsfirst
.
├── pyimagesearch
│   ├── __init__.py
│   ├── detection.py
│   └── social_distancing_config.py
├── yolo-coco
│   ├── coco.names
│   ├── yolov3.cfg
│   └── yolov3.weights
├── output.avi
├── pedestrians.mp4
└── social_distance_detector.py

2 directories, 9 files

Let’s dive into the Python configuration file in the next section.

Our configuration file

To help keep our code tidy and organized, we’ll be using a configuration file to store important variables.

Let’s take a look at them now — open up the social_distancing_config.py file inside the pyimagesearch module, and take a peek:

# base path to YOLO directory
MODEL_PATH = "yolo-coco"

# initialize minimum probability to filter weak detections along with
# the threshold when applying non-maxima suppression
MIN_CONF = 0.3
NMS_THRESH = 0.3

Here, we have the path to the YOLO object detection model (Line 2). We also define the minimum object detection confidence and non-maxima suppression threshold.

We have two more configuration constants to define:

# boolean indicating if NVIDIA CUDA GPU should be used
USE_GPU = False

# define the minimum safe distance (in pixels) that two people can be
# from each other
MIN_DISTANCE = 50

The USE_GPU boolean on Line 10 indicates whether your NVIDIA CUDA-capable GPU will be used to speed up inference (requires that OpenCV’s “dnn” module be installed with NVIDIA GPU support).

Line 14 defines the minimum distance (in pixels) that people must stay from each other in order to adhere to social distancing protocols.

Detecting people in images and video streams with OpenCV

# import the necessary packages
from .social_distancing_config import NMS_THRESH
from .social_distancing_config import MIN_CONF
import numpy as np
import cv2

def detect_people(frame, net, ln, personIdx=0):
	# grab the dimensions of the frame and  initialize the list of
	# results
	(H, W) = frame.shape[:2]
	results = []

	# construct a blob from the input frame and then perform a forward
	# pass of the YOLO object detector, giving us our bounding boxes
	# and associated probabilities
	blob = cv2.dnn.blobFromImage(frame, 1 / 255.0, (416, 416),
		swapRB=True, crop=False)
	net.setInput(blob)
	layerOutputs = net.forward(ln)

	# initialize our lists of detected bounding boxes, centroids, and
	# confidences, respectively
	boxes = []
	centroids = []
	confidences = []

	# loop over each of the layer outputs
	for output in layerOutputs:
		# loop over each of the detections
		for detection in output:
			# extract the class ID and confidence (i.e., probability)
			# of the current object detection
			scores = detection[5:]
			classID = np.argmax(scores)
			confidence = scores[classID]

			# filter detections by (1) ensuring that the object
			# detected was a person and (2) that the minimum
			# confidence is met
			if classID == personIdx and confidence > MIN_CONF:
				# scale the bounding box coordinates back relative to
				# the size of the image, keeping in mind that YOLO
				# actually returns the center (x, y)-coordinates of
				# the bounding box followed by the boxes' width and
				# height
				box = detection[0:4] * np.array([W, H, W, H])
				(centerX, centerY, width, height) = box.astype("int")

				# use the center (x, y)-coordinates to derive the top
				# and and left corner of the bounding box
				x = int(centerX - (width / 2))
				y = int(centerY - (height / 2))

				# update our list of bounding box coordinates,
				# centroids, and confidences
				boxes.append([x, y, int(width), int(height)])
				centroids.append((centerX, centerY))
				confidences.append(float(confidence))

	# apply non-maxima suppression to suppress weak, overlapping
	# bounding boxes
	idxs = cv2.dnn.NMSBoxes(boxes, confidences, MIN_CONF, NMS_THRESH)

	# ensure at least one detection exists
	if len(idxs) > 0:
		# loop over the indexes we are keeping
		for i in idxs.flatten():
			# extract the bounding box coordinates
			(x, y) = (boxes[i][0], boxes[i][1])
			(w, h) = (boxes[i][2], boxes[i][3])

			# update our results list to consist of the person
			# prediction probability, bounding box coordinates,
			# and the centroid
			r = (confidences[i], (x, y, x + w, y + h), centroids[i])
			results.append(r)

	# return the list of results
	return results

Implementing a social distancing detector with OpenCV and deep learning

# import the necessary packages
from pyimagesearch import social_distancing_config as config
from pyimagesearch.detection import detect_people
from scipy.spatial import distance as dist
import numpy as np
import argparse
import imutils
import cv2
import os

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--input", type=str, default="",
	help="path to (optional) input video file")
ap.add_argument("-o", "--output", type=str, default="",
	help="path to (optional) output video file")
ap.add_argument("-d", "--display", type=int, default=1,
	help="whether or not output frame should be displayed")
args = vars(ap.parse_args())

# load the COCO class labels our YOLO model was trained on
labelsPath = os.path.sep.join([config.MODEL_PATH, "coco.names"])
LABELS = open(labelsPath).read().strip().split("\n")

# derive the paths to the YOLO weights and model configuration
weightsPath = os.path.sep.join([config.MODEL_PATH, "yolov3.weights"])
configPath = os.path.sep.join([config.MODEL_PATH, "yolov3.cfg"])

Here, we load our load COCO labels (Lines 22 and 23) as well as define our YOLO paths (Lines 26 and 27).

Using the YOLO paths, now we can load the model into memory:

# load our YOLO object detector trained on COCO dataset (80 classes)
print("[INFO] loading YOLO from disk...")
net = cv2.dnn.readNetFromDarknet(configPath, weightsPath)

# check if we are going to use GPU
if config.USE_GPU:
	# set CUDA as the preferable backend and target
	print("[INFO] setting preferable backend and target to CUDA...")
	net.setPreferableBackend(cv2.dnn.DNN_BACKEND_CUDA)
	net.setPreferableTarget(cv2.dnn.DNN_TARGET_CUDA)

# determine only the *output* layer names that we need from YOLO
ln = net.getLayerNames()
ln = [ln[i[0] - 1] for i in net.getUnconnectedOutLayers()]

# initialize the video stream and pointer to output video file
print("[INFO] accessing video stream...")
vs = cv2.VideoCapture(args["input"] if args["input"] else 0)
writer = None

# loop over the frames from the video stream
while True:
	# read the next frame from the file
	(grabbed, frame) = vs.read()

	# if the frame was not grabbed, then we have reached the end
	# of the stream
	if not grabbed:
		break

	# resize the frame and then detect people (and only people) in it
	frame = imutils.resize(frame, width=700)
	results = detect_people(frame, net, ln,
		personIdx=LABELS.index("person"))

	# initialize the set of indexes that violate the minimum social
	# distance
	violate = set()

	# ensure there are *at least* two people detections (required in
	# order to compute our pairwise distance maps)
	if len(results) >= 2:
		# extract all centroids from the results and compute the
		# Euclidean distances between all pairs of the centroids
		centroids = np.array([r[2] for r in results])
		D = dist.cdist(centroids, centroids, metric="euclidean")

		# loop over the upper triangular of the distance matrix
		for i in range(0, D.shape[0]):
			for j in range(i + 1, D.shape[1]):
				# check to see if the distance between any two
				# centroid pairs is less than the configured number
				# of pixels
				if D[i, j] < config.MIN_DISTANCE:
					# update our violation set with the indexes of
					# the centroid pairs
					violate.add(i)
					violate.add(j)

What fun would our app be if we couldn’t visualize results?

No fun at all, I say! So let’s annotate our frame with rectangles, circles, and text:

	# loop over the results
	for (i, (prob, bbox, centroid)) in enumerate(results):
		# extract the bounding box and centroid coordinates, then
		# initialize the color of the annotation
		(startX, startY, endX, endY) = bbox
		(cX, cY) = centroid
		color = (0, 255, 0)

		# if the index pair exists within the violation set, then
		# update the color
		if i in violate:
			color = (0, 0, 255)

		# draw (1) a bounding box around the person and (2) the
		# centroid coordinates of the person,
		cv2.rectangle(frame, (startX, startY), (endX, endY), color, 2)
		cv2.circle(frame, (cX, cY), 5, color, 1)

	# draw the total number of social distancing violations on the
	# output frame
	text = "Social Distancing Violations: {}".format(len(violate))
	cv2.putText(frame, text, (10, frame.shape[0] - 25),
		cv2.FONT_HERSHEY_SIMPLEX, 0.85, (0, 0, 255), 3)

	# check to see if the output frame should be displayed to our
	# screen
	if args["display"] > 0:
		# show the output frame
		cv2.imshow("Frame", frame)
		key = cv2.waitKey(1) & 0xFF

		# if the `q` key was pressed, break from the loop
		if key == ord("q"):
			break

	# if an output video file path has been supplied and the video
	# writer has not been initialized, do so now
	if args["output"] != "" and writer is None:
		# initialize our video writer
		fourcc = cv2.VideoWriter_fourcc(*"MJPG")
		writer = cv2.VideoWriter(args["output"], fourcc, 25,
			(frame.shape[1], frame.shape[0]), True)

	# if the video writer is not None, write the frame to the output
	# video file
	if writer is not None:
		writer.write(frame)

OpenCV social distancing detector results

We are now ready to test our OpenCV social distancing detector.

Make sure you use the “Downloads” section of this tutorial to download the source code and example demo video.

From there, open up a terminal, and execute the following command:

$ time python social_distance_detector.py --input pedestrians.mp4  \
	--output output.avi --display 0
[INFO] loading YOLO from disk...
[INFO] accessing video stream...

real    3m43.120s
user    23m20.616s
sys     0m25.824s

Here, you can see that I was able to process the entire video in 3m43s on my CPU, and as the results show, our social distancing detector is correctly marking people who violate social distancing rules.

The problem with this current implementation is speed. Our CPU-based social distancing detector is obtaining ~2.3 FPS, which is far too slow for real-time processing.

You can obtain a higher frame processing rate by (1) utilizing an NVIDIA CUDA-capable GPU and (2) compiling/installing OpenCV’s “dnn” module with NVIDIA GPU support.

Provided you already have OpenCV installed with NVIDIA GPU support, all you need to do is set USE_GPU = True in your social_distancing_config.py file:

# boolean indicating if NVIDIA CUDA GPU should be used
USE_GPU = True

$ time python social_distance_detector.py --input pedestrians.mp4 \
	--output output.avi --display 0
[INFO] loading YOLO from disk...
[INFO] setting preferable backend and target to CUDA...
[INFO] accessing video stream...

real    0m56.008s
user    1m15.772s
sys     0m7.036s

Here, we processed the entire video in only 56 seconds, amounting to ~9.38 FPS, which is a 307% speedup!

Limitations and future improvements

As already mentioned earlier in this tutorial, our social distancing detector did not leverage a proper camera calibration, meaning that we could not (easily) map distances in pixels to actual measurable units (i.e., meters, feet, etc.).

Therefore, the first step to improving our social distancing detector is to utilize a proper camera calibration.

Doing so will yield better results and enable you to compute actual measurable units (rather than pixels).

Secondly, you should consider applying a top-down transformation of your viewing angle, as this implementation has done:

From there, you can apply the distance calculations to the top-down view of the pedestrians, leading to a better distance approximation.

My third recommendation is to improve the people detection process.

OpenCV’s YOLO implementation is quite slow not because of the model itself but because of the additional post-processing required by the model.

To further speedup the pipeline, consider utilizing a Single Shot Detector (SSD) running on your GPU — that will improve frame throughput rate considerably.

To wrap up, I’d like to mention that there are a number of social distancing detector implementations you’ll see online — the one I’ve covered here today should be considered a template and starting point that you can build off of.

If you would like to learn more about implementing social distancing detectors with computer vision, check out some of the following resources:

• Automatic social distance measurement
• Social distancing in the workplace
• Rohit Kumar Srivastava’s social distancing implementation
• Venkatagiri Ramesh’s social distancing project
• Mohan Morkel’s social distancing application (which I think may be based on Venkatagiri Ramesh’s)

If you have implemented your own OpenCV social distancing project and I have not linked to it, kindly accept my apologies — there are simply too many implementations for me to keep track of at this point.

What’s next?

This tutorial focused on a timely application of deep learning amid the worldwide COVID-19 emergency.

My hope is that you are inspired to pursue and solve complex project ideas of your own using artificial intelligence and computer vision.

But where should you begin?

If you don’t already know the fundamentals (let alone, advanced concepts) of deep learning, now would be a good time to learn them so that you can make your next app or service a reality.

To get a head start, you should read my book, Deep Learning for Computer Vision with Python.

Inside the book, you will learn:

• Deep learning fundamentals and theory without unnecessary mathematical fluff. I present the basic equations and back them up with code walkthroughs that you can implement and easily understand. You don’t need a degree in advanced mathematics to understand this book
• How to implement your own custom neural network architectures. Not only will you learn how to implement state-of-the-art architectures, including ResNet, SqueezeNet, etc., but you’ll also learn how to create your own custom CNNs
• How to train CNNs on your own datasets. Most deep learning tutorials don’t teach you how to work with your own custom datasets. Mine do. You’ll be training CNNs on your own datasets in no time
• Object detection (Faster R-CNNs, Single Shot Detectors, and RetinaNet) and instance segmentation (Mask R-CNN). Use these chapters to create your own custom object detectors and segmentation networks

If you’re interested in learning more, simply click the button below. I’m happy to send you the full table of contents and a few sample chapters so you can see if the book is for you.

Summary

In this tutorial, you learned how to implement a social distancing detector using OpenCV, computer vision, and deep learning.

Our implementation worked by:

1. Using the YOLO object detector to detect people in a video stream
2. Determining the centroids for each detected person
3. Computing the pairwise distances between all centroids
4. Checking to see if any pairwise distances were < N pixels apart, and if so, indicating that the pair of people violated social distancing rules

Furthermore, by using an NVIDIA CUDA-capable GPU, along with OpenCV’s dnn module compiled with NVIDIA GPU support, our method was able to run in real-time, making it usable as a proof-of-concept social distancing detector.

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post OpenCV Social Distancing Detector appeared first on PyImageSearch.

In this tutorial, you will learn how to use OpenCV and the Fast Fourier Transform (FFT) to perform blur detection in images and real-time video streams.

Today’s tutorial is an extension of my previous blog post on Blur Detection with OpenCV. The original blur detection method:

• Relied on computing the variance of the Laplacian operator
• Could be implemented in only a single line of code
• Was dead simple to use

The downside is that the Laplacian method required significant manual tuning to define the “threshold” at which an image was considered blurry or not. If you could control your lighting conditions, environment, and image capturing process, it worked quite well — but if not, you would obtain mixed results, to say the least.

The method we’ll be covering here today relies on computing the Fast Fourier Transform of the image. It still requires some manual tuning, but as we’ll find out, the FFT blur detector we’ll be covering is far more robust and reliable than the variance of the Laplacian method.

By the end of this tutorial, you’ll have a fully functioning FFT blur detector that you can apply to both images and video streams.

To learn how to use OpenCV and the Fast Fourier Transform (FFT) to perform blur detection, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

OpenCV Fast Fourier Transform (FFT) for Blur Detection

In the first part of this tutorial, we’ll briefly discuss:

• What blur detection is
• Why we may want to detect blur in an image/video stream
• And how the Fast Fourier Transform can enable us to detect blur.

From there, we’ll implement our FFT blur detector for both images and real-time video.

We’ll wrap up the tutorial by reviewing the results of our FFT blur detector.

What is blur detection and when would we want to detect blur?

Blur detection, as the name suggests, is the process of detecting whether an image is blurry or not.

Possible applications of blur detection include:

• Automatic image quality grading
• Helping professional photographers sort through 100s to 1000s of photos during a photo shoot by automatically discarding the blurry/low quality ones
• Applying OCR to real-time video streams, but only applying the expensive OCR computation to non-blurry frames

The key takeaway here is that it’s always easier to write computer vision code for images captured under ideal conditions.

Instead of trying to handle edge cases where image quality is extremely poor, simply detect and discard the poor quality images (such as ones with significant blur).

Such a blur detection procedure could either automatically discard the poor quality images or simply tell the end user “Hey bud, try again. Let’s capture a better image here.”

Keep in mind that computer vision applications are meant to be intelligent, hence the term, artificial intelligence — and sometimes, that “intelligence” can simply be detecting when input data is of poor quality or not rather than trying to make sense of it.

What is the Fast Fourier Transform (FFT)?

The Fast Fourier Transform is a convenient mathematical algorithm for computing the Discrete Fourier Transform. It is used for converting a signal from one domain into another.

The FFT is useful in many disciplines, ranging from music, mathematics, science, and engineering. For example, electrical engineers, particularly those working with wireless, power, and audio signals, need the FFT calculation to convert time-series signals into the frequency domain because some calculations are more easily made in the frequency domain. Conversely, a frequency domain signal could be converted back into the time domain using the FFT.

In terms of computer vision, we often think of the FFT as an image processing tool that represents an image in two domains:

1. Fourier (i.e., frequency) domain
2. Spatial domain

Therefore, the FFT represents the image in both real and imaginary components.

By analyzing these values, we can perform image processing routines such as blurring, edge detection, thresholding, texture analysis, and yes, even blur detection.

Reviewing the mathematical details of the Fast Fourier Transform is outside the scope of this blog post, so if you’re interested in learning more about it, I suggest you read this article on the FFT and its relation to image processing.

For readers who are academically inclined, take a look at Aaron Bobick’s fantastic slides from Georgia Tech’s computer vision course.

Finally, the Wikipedia page on the Fourier Transform goes into more detail on the mathematics including its applications to non-image processing tasks.

Project structure

Start by using the “Downloads” section of this tutorial to download the source code and example images. Once you extract the files, you’ll have a directory organized as follows:

$ tree --dirsfirst
.
├── images
│   ├── adrian_01.png
│   ├── adrian_02.png
│   ├── jemma.png
│   └── resume.png
├── pyimagesearch
│   ├── __init__.py
│   └── blur_detector.py
├── blur_detector_image.py
└── blur_detector_video.py

2 directories, 8 files

In the next section, we’ll implement our FFT-based blur detection algorithm.

Implementing our FFT blur detector with OpenCV

We are now ready to implement our Fast Fourier Transform blur detector with OpenCV.

The method we’ll be covering is based on the following implementation from Liu et al.’s 2008 CVPR publication, Image Partial Blur Detection and Classification.

Open up the blur_detector.py file in our directory structure, and insert the following code:

# import the necessary packages
import matplotlib.pyplot as plt
import numpy as np

def detect_blur_fft(image, size=60, thresh=10, vis=False):
	# grab the dimensions of the image and use the dimensions to
	# derive the center (x, y)-coordinates
	(h, w) = image.shape
	(cX, cY) = (int(w / 2.0), int(h / 2.0))

	# compute the FFT to find the frequency transform, then shift
	# the zero frequency component (i.e., DC component located at
	# the top-left corner) to the center where it will be more
	# easy to analyze
	fft = np.fft.fft2(image)
	fftShift = np.fft.fftshift(fft)

Here, using NumPy’s built-in algorithm, we compute the FFT (Line 15).

We then shift the zero frequency component (DC component) of the result to the center for easier analysis (Line 16).

Now that we have the FFT of our image in hand, let’s visualize the result if the vis flag has been set:

	# check to see if we are visualizing our output
	if vis:
		# compute the magnitude spectrum of the transform
		magnitude = 20 * np.log(np.abs(fftShift))

		# display the original input image
		(fig, ax) = plt.subplots(1, 2, )
		ax[0].imshow(image, cmap="gray")
		ax[0].set_title("Input")
		ax[0].set_xticks([])
		ax[0].set_yticks([])

		# display the magnitude image
		ax[1].imshow(magnitude, cmap="gray")
		ax[1].set_title("Magnitude Spectrum")
		ax[1].set_xticks([])
		ax[1].set_yticks([])

		# show our plots
		plt.show()

	# zero-out the center of the FFT shift (i.e., remove low
	# frequencies), apply the inverse shift such that the DC
	# component once again becomes the top-left, and then apply
	# the inverse FFT
	fftShift[cY - size:cY + size, cX - size:cX + size] = 0
	fftShift = np.fft.ifftshift(fftShift)
	recon = np.fft.ifft2(fftShift)

Here, we:

• Zero-out the center of our FFT shift (i.e., to remove low frequencies) via Line 43
• Apply the inverse shift to put the DC component back in the top-left (Line 44)
• Apply the inverse FFT (Line 45)

And from here, we have three more steps to determine if our image is blurry:

	# compute the magnitude spectrum of the reconstructed image,
	# then compute the mean of the magnitude values
	magnitude = 20 * np.log(np.abs(recon))
	mean = np.mean(magnitude)

	# the image will be considered "blurry" if the mean value of the
	# magnitudes is less than the threshold value
	return (mean, mean <= thresh)

Great job implementing an FFT-based blurriness detector algorithm. We aren’t done yet though. In the next section, we’ll apply our algorithm to static images to ensure it is performing to our expectations.

Detecting blur in images with FFT

# import the necessary packages
from pyimagesearch.blur_detector import detect_blur_fft
import numpy as np
import argparse
import imutils
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", type=str, required=True,
	help="path input image that we'll detect blur in")
ap.add_argument("-t", "--thresh", type=int, default=20,
	help="threshold for our blur detector to fire")
ap.add_argument("-v", "--vis", type=int, default=-1,
	help="whether or not we are visualizing intermediary steps")
ap.add_argument("-d", "--test", type=int, default=-1,
	help="whether or not we should progressively blur the image")
args = vars(ap.parse_args())

# load the input image from disk, resize it, and convert it to
# grayscale
orig = cv2.imread(args["image"])
orig = imutils.resize(orig, width=500)
gray = cv2.cvtColor(orig, cv2.COLOR_BGR2GRAY)

# apply our blur detector using the FFT
(mean, blurry) = detect_blur_fft(gray, size=60,
	thresh=args["thresh"], vis=args["vis"] > 0)

# draw on the image, indicating whether or not it is blurry
image = np.dstack([gray] * 3)
color = (0, 0, 255) if blurry else (0, 255, 0)
text = "Blurry ({:.4f})" if blurry else "Not Blurry ({:.4f})"
text = text.format(mean)
cv2.putText(image, text, (10, 25), cv2.FONT_HERSHEY_SIMPLEX, 0.7,
	color, 2)
print("[INFO] {}".format(text))

# show the output image
cv2.imshow("Output", image)
cv2.waitKey(0)

# check to see if are going to test our FFT blurriness detector using
# various sizes of a Gaussian kernel
if args["test"] > 0:
	# loop over various blur radii
	for radius in range(1, 30, 2):
		# clone the original grayscale image
		image = gray.copy()

		# check to see if the kernel radius is greater than zero
		if radius > 0:
			# blur the input image by the supplied radius using a
			# Gaussian kernel
			image = cv2.GaussianBlur(image, (radius, radius), 0)

			# apply our blur detector using the FFT
			(mean, blurry) = detect_blur_fft(image, size=60,
				thresh=args["thresh"], vis=args["vis"] > 0)

			# draw on the image, indicating whether or not it is
			# blurry
			image = np.dstack([image] * 3)
			color = (0, 0, 255) if blurry else (0, 255, 0)
			text = "Blurry ({:.4f})" if blurry else "Not Blurry ({:.4f})"
			text = text.format(mean)
			cv2.putText(image, text, (10, 25), cv2.FONT_HERSHEY_SIMPLEX,
				0.7, color, 2)
			print("[INFO] Kernel: {}, Result: {}".format(radius, text))

		# show the image
		cv2.imshow("Test Image", image)
		cv2.waitKey(0)

FFT blur detection in images results

We are now ready to use OpenCV and the Fast Fourier Transform to detect blur in images.

Start by making sure you use the “Downloads” section of this tutorial to download the source code and example images.

From there, open up a terminal, and execute the following command:

$ python blur_detector_image.py --image images/adrian_01.png
[INFO] Not Blurry (42.4630)

Here you can see an input image of me hiking The Subway in Zion National Park — the image is correctly marked as not blurry.

Let’s try another image, this one of my family’s dog, Jemma:

$ python blur_detector_image.py --image images/jemma.png
[INFO] Blurry (12.4738)

$ python blur_detector_image.py --image images/adrian_02.png --test 1
[INFO] Not Blurry (32.0934)
[INFO] Kernel: 1, Result: Not Blurry (32.0934)
[INFO] Kernel: 3, Result: Not Blurry (25.1770)
[INFO] Kernel: 5, Result: Not Blurry (20.5668)
[INFO] Kernel: 7, Result: Blurry (13.4830)
[INFO] Kernel: 9, Result: Blurry (7.8893)
[INFO] Kernel: 11, Result: Blurry (0.6506)
[INFO] Kernel: 13, Result: Blurry (-5.3609)
[INFO] Kernel: 15, Result: Blurry (-11.4612)
[INFO] Kernel: 17, Result: Blurry (-17.0109)
[INFO] Kernel: 19, Result: Blurry (-19.6464)
[INFO] Kernel: 21, Result: Blurry (-20.4758)
[INFO] Kernel: 23, Result: Blurry (-20.7365)
[INFO] Kernel: 25, Result: Blurry (-20.9362)
[INFO] Kernel: 27, Result: Blurry (-21.1911)
[INFO] Kernel: 29, Result: Blurry (-21.3853)

Here, you can see that as our image becomes more and more blurry, the mean FFT magnitude values decrease.

Our FFT blur detection method can be applied to non-natural scene images as well.

For example, let’s suppose we want to build an automatic document scanner application — such a computer vision project should automatically reject blurry images.

However, document images are very different from natural scene images and by their nature will be much more sensitive to blur.

Any type of blur will impact OCR accuracy significantly.

Therefore, we should increase our --thresh value (and I’ll also include the --vis argument so we can visualize how the FFT magnitude values change):

$ python blur_detector_image.py --image images/resume.png --thresh 27 --test 1 --vis 1
[INFO] Not Blurry (34.6735)
[INFO] Kernel: 1, Result: Not Blurry (34.6735)
[INFO] Kernel: 3, Result: Not Blurry (29.2539)
[INFO] Kernel: 5, Result: Blurry (26.2893)
[INFO] Kernel: 7, Result: Blurry (21.7390)
[INFO] Kernel: 9, Result: Blurry (18.3632)
[INFO] Kernel: 11, Result: Blurry (12.7235)
[INFO] Kernel: 13, Result: Blurry (9.1489)
[INFO] Kernel: 15, Result: Blurry (2.3377)
[INFO] Kernel: 17, Result: Blurry (-2.6372)
[INFO] Kernel: 19, Result: Blurry (-9.1908)
[INFO] Kernel: 21, Result: Blurry (-15.9808)
[INFO] Kernel: 23, Result: Blurry (-20.6240)
[INFO] Kernel: 25, Result: Blurry (-29.7478)
[INFO] Kernel: 27, Result: Blurry (-29.0728)
[INFO] Kernel: 29, Result: Blurry (-37.7561)

Here, you can see that our image quickly becomes blurry and unreadable, and as the output shows, our OpenCV FFT blur detector correctly marks these images as blurry.

Below is a visualization of the Fast Fourier Transform magnitude values as the image becomes progressively blurrier and blurrier:

Detecting blur in video with OpenCV and the FFT

So far, we’ve applied our Fast Fourier Transform blur detector to images.

But is it possible to apply FFT blur detection to video streams as well?

And can the entire process be accomplished in real-time as well?

Let’s find out — open up a new file, name it blur_detector_video.py, and insert the following code:

# import the necessary packages
from imutils.video import VideoStream
from pyimagesearch.blur_detector import detect_blur_fft
import argparse
import imutils
import time
import cv2

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-t", "--thresh", type=int, default=10,
	help="threshold for our blur detector to fire")
args = vars(ap.parse_args())

# initialize the video stream and allow the camera sensor to warm up
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
time.sleep(2.0)

# loop over the frames from the video stream
while True:
	# grab the frame from the threaded video stream and resize it
	# to have a maximum width of 400 pixels
	frame = vs.read()
	frame = imutils.resize(frame, width=500)

	# convert the frame to grayscale and detect blur in it
	gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
	(mean, blurry) = detect_blur_fft(gray, size=60,
		thresh=args["thresh"], vis=False)

Lines 17 and 18 initialize our webcam stream and allow time for the camera to warm up.

From there, we begin a frame processing loop on Line 21. Inside, we grab a frame and convert it to grayscale (Lines 24-28) just as in our single image blur detection script.

	# draw on the frame, indicating whether or not it is blurry
	color = (0, 0, 255) if blurry else (0, 255, 0)
	text = "Blurry ({:.4f})" if blurry else "Not Blurry ({:.4f})"
	text = text.format(mean)
	cv2.putText(frame, text, (10, 25), cv2.FONT_HERSHEY_SIMPLEX,
		0.7, color, 2)

	# show the output frame
	cv2.imshow("Frame", frame)
	key = cv2.waitKey(1) & 0xFF

	# if the `q` key was pressed, break from the loop
	if key == ord("q"):
		break

# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()

Our last code block should look very familiar at this point because this is the third time we’ve seen these lines of code. Here we:

• Annotate either blurry (red colored text) or not blurry (green colored text) as well as the mean value (Lines 33-37)
• Display the result (Line 40)
• Quit if the q key is pressed (Lines 41-45), and perform housekeeping cleanup (Lines 48 and 49)

Fast Fourier Transform video blur detection results

We’re now ready to find out if our OpenCV FFT blur detector can be applied to real-time video streams.

Make sure you use the “Downloads” section of this tutorial to download the source code.

From there, open up a terminal, and execute the following command:

$ python blur_detector_video.py
[INFO] starting video stream...

As I move my laptop, motion blur is introduced into the frame.

If we were implementing a computer vision system to automatically extract key, important frames, or creating an automatic video OCR system, we would want to discard these blurry frames — using our OpenCV FFT blur detector, we can do exactly that!

What’s next?

Are you interested in learning more about image processing, computer vision, and machine/deep learning?

If so, you’ll want to take a look at the PyImageSearch Gurus course.

I didn’t have the luxury of such a course in college.

I learned computer vision the hard way — a tale much like the one your grandparents tell, in which they walked uphill both ways in 4 feet of snow each day on their way to school.

Back then, there weren’t great image processing blogs like PyImageSearch online to learn from. Of course, there were theory and math intensive textbooks, complex research papers, and the occasional sit-down in my adviser’s office. But none of these resources taught computer vision systematically via practical use cases and Python code examples.

So what did I do?

I took what I learned and came up with my own examples and projects to learn from. It wasn’t easy, but by the end of it, I was confident that I knew computer vision well enough to consult for the National Institutes of Health (NIH) and build/deploy a couple of iPhone apps to the App Store.

Now what does that mean for you?

You have the golden opportunity to learn from me in a central place with other motivated students. I’ve developed a course using my personal arsenal of code and my years of knowledge. You will learn concepts and code the way I wish I had earlier in my career.

Inside PyImageSearch Gurus, you’ll find:

• An actionable, real-world course on Computer Vision, Deep Learning, and OpenCV. Each lesson in PyImageSearch Gurus is taught in the same hands-on, easy-to-understand PyImageSearch style that you know and love.
• The most comprehensive computer vision education online today. The PyImageSearch Gurus course covers 13 modules broken out into 168 lessons, with over 2,161 pages of content. You won’t find a more detailed computer vision course anywhere else online; I guarantee it.
• A community of like-minded developers, researchers, and students just like you, who are eager to learn computer vision, level-up their skills, and collaborate on projects. I participate in the forums nearly every day. These forums are a great way to get expert advice, both from me as well as the more advanced students.

Take a look at these previous students’ success stories — each of these students invested in themselves and achieved success. You can too in a short time after you take the plunge by enrolling today.

If you’re on the fence, grab the course syllabus and 10 free sample lessons. If that sounds interesting to you, simply click this link:

Summary

In today’s tutorial, you learned how to use OpenCV’s Fast Fourier Transform (FFT) implementation to perform blur detection in images and real-time video streams.

While not as simple as our variance of the Laplacian blur detector, the FFT blur detector is more robust and tends to provide better blur detection accuracy in real-life applications.

The problem is that the FFT method still requires us to set a manual threshold, specifically on the mean value of the FFT magnitudes.

An ideal blur detector would be able to detect blur in images and video streams without such a threshold.

To accomplish this task we’ll need a bit of machine learning — I’ll cover an automatic blur detector in a future tutorial.

To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post OpenCV Fast Fourier Transform (FFT) for blur detection in images and video streams appeared first on PyImageSearch.

In this tutorial, you will learn how to take any pre-trained deep learning image classifier and turn it into an object detector using Keras, TensorFlow, and OpenCV.

Today, we’re starting a four-part series on deep learning and object detection:

• Part 1: Turning any deep learning image classifier into an object detector with Keras and TensorFlow (today’s post)
• Part 2: OpenCV Selective Search for Object Detection
• Part 3: Region proposal for object detection with OpenCV, Keras, and TensorFlow
• Part 4: R-CNN object detection with Keras and TensorFlow

The goal of this series of posts is to obtain a deeper understanding of how deep learning-based object detectors work, and more specifically:

1. How traditional computer vision object detection algorithms can be combined with deep learning
2. What the motivations behind end-to-end trainable object detectors and the challenges associated with them are
3. And most importantly, how the seminal Faster R-CNN architecture came to be (we’ll be building a variant of the R-CNN architecture throughout this series)

Today, we’ll be starting with the fundamentals of object detection, including how to take a pre-trained image classifier and utilize image pyramids, sliding windows, and non-maxima suppression to build a basic object detector (think HOG + Linear SVM-inspired).

Over the coming weeks, we’ll learn how to build an end-to-end trainable network from scratch.

But for today, let’s start with the basics.

To learn how to take any Convolutional Neural Network image classifier and turn it into an object detector with Keras and TensorFlow, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

Turning any CNN image classifier into an object detector with Keras, TensorFlow, and OpenCV

In the first part of this tutorial, we’ll discuss the key differences between image classification and object detection tasks.

I’ll then show you how you can take any Convolutional Neural Network trained for image classification and then turn it into an object detector, all in ~200 lines of code.

From there, we’ll implement the code necessary to take an image classifier and turn it into an object detector using Keras, TensorFlow, and OpenCV.

Finally, we’ll review the results of our work, noting some of the problems and limitations with our implementation, including how we can improve this method.

Image classification vs. object detection

When performing image classification, given an input image, we present it to our neural network, and we obtain a single class label and a probability associated with the class label prediction (Figure 1, left).

This class label is meant to characterize the contents of the entire image, or at least the most dominant, visible contents of the image.

We can thus think of image classification as:

• One image in
• One class label out

Object detection, on the other hand, not only tells us what is in the image (i.e., class label) but also where in the image the object is via bounding box (x, y)-coordinates (Figure 1, right).

Therefore, object detection algorithms allow us to:

• Input one image
• Obtain multiple bounding boxes and class labels as output

At the very core, any object detection algorithm (regardless of traditional computer vision or state-of-the-art deep learning), follows the same pattern:

• 1. Input: An image that we wish to apply object detection to
• 2. Output: Three values, including:
  • 2a. A list of bounding boxes, or the (x, y)-coordinates for each object in an image
  • 2b. The class label associated with each of the bounding boxes
  • 2c. The probability/confidence score associated with each bounding box and class label

Today, you’ll see an example of this pattern in action.

How can we turn any deep learning image classifier into an object detector?

At this point, you’re likely wondering:

Hey Adrian, if I have a Convolutional Neural Network trained for image classification, how in the world am I going to use it for object detection?

Based on your explanation above, it seems like image classification and object detection are fundamentally different, requiring two different types of network architectures.

And essentially, that is correct — object detection does require a specialized network architecture.

Anyone who has read papers on Faster R-CNN, Single Shot Detectors (SSDs), YOLO, RetinaNet, etc. knows that object detection networks are more complex, more involved, and take multiple orders of magnitude and more effort to implement compared to traditional image classification.

That said, there is a hack we can leverage to turn our CNN image classifier into an object detector — and the secret sauce lies in traditional computer vision algorithms.

Back before deep learning-based object detectors, the state-of-the-art was to use HOG + Linear SVM to detect objects in an image.

We’ll be borrowing elements from HOG + Linear SVM to convert any deep neural network image classifier into an object detector.

The first key ingredient from HOG + Linear SVM is to use image pyramids.

An “image pyramid” is a multi-scale representation of an image:

Utilizing an image pyramid allows us to find objects in images at different scales (i.e., sizes) of an image (Figure 2).

At the bottom of the pyramid, we have the original image at its original size (in terms of width and height).

And at each subsequent layer, the image is resized (subsampled) and optionally smoothed (usually via Gaussian blurring).

The image is progressively subsampled until some stopping criterion is met, which is normally when a minimum size has been reached and no further subsampling needs to take place.

The second key ingredient we need is sliding windows:

As the name suggests, a sliding window is a fixed-size rectangle that slides from left-to-right and top-to-bottom within an image. (As Figure 3 demonstrates, our sliding window could be used to detect the face in the input image).

At each stop of the window we would:

1. Extract the ROI
2. Pass it through our image classifier (ex., Linear SVM, CNN, etc.)
3. Obtain the output predictions

Combined with image pyramids, sliding windows allow us to localize objects at different locations and multiple scales of the input image:

The final key ingredient we need is non-maxima suppression.

When performing object detection, our object detector will typically produce multiple, overlapping bounding boxes surrounding an object in an image.

This behavior is totally normal — it simply implies that as the sliding window approaches an image, our classifier component is returning larger and larger probabilities of a positive detection.

Of course, multiple bounding boxes pose a problem — there’s only one object there, and we somehow need to collapse/remove the extraneous bounding boxes.

The solution to the problem is to apply non-maxima suppression (NMS), which collapses weak, overlapping bounding boxes in favor of the more confident ones:

On the left, we have multiple detections, while on the right, we have the output of non-maxima suppression, which collapses the multiple bounding boxes into a single detection.

Combining traditional computer vision with deep learning to build an object detector

In order to take any Convolutional Neural Network trained for image classification and instead utilize it for object detection, we’re going to utilize the three key ingredients for traditional computer vision:

1. Image pyramids: Localize objects at different scales/sizes.
2. Sliding windows: Detect exactly where in the image a given object is.
3. Non-maxima suppression: Collapse weak, overlapping bounding boxes.

The general flow of our algorithm will be:

• Step #1: Input an image
• Step #2: Construct an image pyramid
• Step #3: For each scale of the image pyramid, run a sliding window
  • Step #3a: For each stop of the sliding window, extract the ROI
  • Step #3b: Take the ROI and pass it through our CNN originally trained for image classification
  • Step #3c: Examine the probability of the top class label of the CNN, and if meets a minimum confidence, record (1) the class label and (2) the location of the sliding window
• Step #4: Apply class-wise non-maxima suppression to the bounding boxes
• Step #5: Return results to calling function

That may seem like a complicated process, but as you’ll see in the remainder of this post, we can implement the entire object detection procedure in < 200 lines of code!

Configuring your development environment

To configure your system for this tutorial, I first recommend following either of these tutorials:

• How to install TensorFlow 2.0 on Ubuntu
• How to install TensorFlow 2.0 on macOS

Either tutorial will help you configure your system with all the necessary software for this blog post in a convenient Python virtual environment.

Please note that PyImageSearch does not recommend or support Windows for CV/DL projects.

Project structure

Once you extract the .zip from the “Downloads” section of this blog post, your directory will be organized as follows:

.
├── images
│   ├── hummingbird.jpg
│   ├── lawn_mower.jpg
│   └── stingray.jpg
├── pyimagesearch
│   ├── __init__.py
│   └── detection_helpers.py
└── detect_with_classifier.py

2 directories, 6 files

Implementing our image pyramid and sliding window utility functions

In order to turn our CNN image classifier into an object detector, we must first implement helper utilities to construct sliding windows and image pyramids.

Let’s implement this helper functions now — open up the detection_helpers.py file in the pyimagesearch module, and insert the following code:

# import the necessary packages
import imutils

def sliding_window(image, step, ws):
	# slide a window across the image
	for y in range(0, image.shape[0] - ws[1], step):
		for x in range(0, image.shape[1] - ws[0], step):
			# yield the current window
			yield (x, y, image[y:y + ws[1], x:x + ws[0]])

def image_pyramid(image, scale=1.5, minSize=(224, 224)):
	# yield the original image
	yield image

	# keep looping over the image pyramid
	while True:
		# compute the dimensions of the next image in the pyramid
		w = int(image.shape[1] / scale)
		image = imutils.resize(image, width=w)

		# if the resized image does not meet the supplied minimum
		# size, then stop constructing the pyramid
		if image.shape[0] < minSize[1] or image.shape[1] < minSize[0]:
			break

		# yield the next image in the pyramid
		yield image

For more details, please refer to my Image Pyramids with Python and OpenCV article, which also includes an alternative scikit-image image pyramid implementation that may be useful to you.

Using Keras and TensorFlow to turn a pre-trained image classifier into an object detector

# import the necessary packages
from tensorflow.keras.applications import ResNet50
from tensorflow.keras.applications.resnet import preprocess_input
from tensorflow.keras.preprocessing.image import img_to_array
from tensorflow.keras.applications import imagenet_utils
from imutils.object_detection import non_max_suppression
from pyimagesearch.detection_helpers import sliding_window
from pyimagesearch.detection_helpers import image_pyramid
import numpy as np
import argparse
import imutils
import time
import cv2

# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
	help="path to the input image")
ap.add_argument("-s", "--size", type=str, default="(200, 150)",
	help="ROI size (in pixels)")
ap.add_argument("-c", "--min-conf", type=float, default=0.9,
	help="minimum probability to filter weak detections")
ap.add_argument("-v", "--visualize", type=int, default=-1,
	help="whether or not to show extra visualizations for debugging")
args = vars(ap.parse_args())

# initialize variables used for the object detection procedure
WIDTH = 600
PYR_SCALE = 1.5
WIN_STEP = 16
ROI_SIZE = eval(args["size"])
INPUT_SIZE = (224, 224)

Understanding what each of the above constants controls is crucial to your understanding of how to turn an image classifier into an object detector with Keras, TensorFlow, and OpenCV. Be sure to mentally distinguish each of these before moving on.

Let’s load our ResNet classification CNN and input image:

# load our network weights from disk
print("[INFO] loading network...")
model = ResNet50(weights="imagenet", include_top=True)

# load the input image from disk, resize it such that it has the
# has the supplied width, and then grab its dimensions
orig = cv2.imread(args["image"])
orig = imutils.resize(orig, width=WIDTH)
(H, W) = orig.shape[:2]

Line 36 loads ResNet pre-trained on ImageNet. If you choose to use a different pre-trained classifier, you can substitute one here for your particular project. To learn how to train your own classifier, I suggest you read Deep Learning for Computer Vision with Python.

# initialize the image pyramid
pyramid = image_pyramid(orig, scale=PYR_SCALE, minSize=ROI_SIZE)

# initialize two lists, one to hold the ROIs generated from the image
# pyramid and sliding window, and another list used to store the
# (x, y)-coordinates of where the ROI was in the original image
rois = []
locs = []

# time how long it takes to loop over the image pyramid layers and
# sliding window locations
start = time.time()

# loop over the image pyramid
for image in pyramid:
	# determine the scale factor between the *original* image
	# dimensions and the *current* layer of the pyramid
	scale = W / float(image.shape[1])

	# for each layer of the image pyramid, loop over the sliding
	# window locations
	for (x, y, roiOrig) in sliding_window(image, WIN_STEP, ROI_SIZE):
		# scale the (x, y)-coordinates of the ROI with respect to the
		# *original* image dimensions
		x = int(x * scale)
		y = int(y * scale)
		w = int(ROI_SIZE[0] * scale)
		h = int(ROI_SIZE[1] * scale)

		# take the ROI and preprocess it so we can later classify
		# the region using Keras/TensorFlow
		roi = cv2.resize(roiOrig, INPUT_SIZE)
		roi = img_to_array(roi)
		roi = preprocess_input(roi)

		# update our list of ROIs and associated coordinates
		rois.append(roi)
		locs.append((x, y, x + w, y + h))

		# check to see if we are visualizing each of the sliding
		# windows in the image pyramid
		if args["visualize"] > 0:
			# clone the original image and then draw a bounding box
			# surrounding the current region
			clone = orig.copy()
			cv2.rectangle(clone, (x, y), (x + w, y + h),
				(0, 255, 0), 2)

			# show the visualization and current ROI
			cv2.imshow("Visualization", clone)
			cv2.imshow("ROI", roiOrig)
			cv2.waitKey(0)

# show how long it took to loop over the image pyramid layers and
# sliding window locations
end = time.time()
print("[INFO] looping over pyramid/windows took {:.5f} seconds".format(
	end - start))

# convert the ROIs to a NumPy array
rois = np.array(rois, dtype="float32")

# classify each of the proposal ROIs using ResNet and then show how
# long the classifications took
print("[INFO] classifying ROIs...")
start = time.time()
preds = model.predict(rois)
end = time.time()
print("[INFO] classifying ROIs took {:.5f} seconds".format(
	end - start))

# decode the predictions and initialize a dictionary which maps class
# labels (keys) to any ROIs associated with that label (values)
preds = imagenet_utils.decode_predictions(preds, top=1)
labels = {}

# loop over the predictions
for (i, p) in enumerate(preds):
	# grab the prediction information for the current ROI
	(imagenetID, label, prob) = p[0]

	# filter out weak detections by ensuring the predicted probability
	# is greater than the minimum probability
	if prob >= args["min_conf"]:
		# grab the bounding box associated with the prediction and
		# convert the coordinates
		box = locs[i]

		# grab the list of predictions for the label and add the
		# bounding box and probability to the list
		L = labels.get(label, [])
		L.append((box, prob))
		labels[label] = L

# loop over the labels for each of detected objects in the image
for label in labels.keys():
	# clone the original image so that we can draw on it
	print("[INFO] showing results for '{}'".format(label))
	clone = orig.copy()

	# loop over all bounding boxes for the current label
	for (box, prob) in labels[label]:
		# draw the bounding box on the image
		(startX, startY, endX, endY) = box
		cv2.rectangle(clone, (startX, startY), (endX, endY),
			(0, 255, 0), 2)

	# show the results *before* applying non-maxima suppression, then
	# clone the image again so we can display the results *after*
	# applying non-maxima suppression
	cv2.imshow("Before", clone)
	clone = orig.copy()

	# extract the bounding boxes and associated prediction
	# probabilities, then apply non-maxima suppression
	boxes = np.array([p[0] for p in labels[label]])
	proba = np.array([p[1] for p in labels[label]])
	boxes = non_max_suppression(boxes, proba)

	# loop over all bounding boxes that were kept after applying
	# non-maxima suppression
	for (startX, startY, endX, endY) in boxes:
		# draw the bounding box and label on the image
		cv2.rectangle(clone, (startX, startY), (endX, endY),
			(0, 255, 0), 2)
		y = startY - 10 if startY - 10 > 10 else startY + 10
		cv2.putText(clone, label, (startX, y),
			cv2.FONT_HERSHEY_SIMPLEX, 0.45, (0, 255, 0), 2)

	# show the output after apply non-maxima suppression
	cv2.imshow("After", clone)
	cv2.waitKey(0)

To apply NMS, we first extract the bounding boxes and associated prediction probabilities (proba) via Lines 159 and 160. We then pass those results into my imultils implementation of NMS (Line 161). For more details on non-maxima suppression, be sure to refer to my blog post.

After NMS has been applied, Lines 165-171 annotate bounding box rectangles and labels on the “after” image. Lines 174 and 175 display the results until a key is pressed, at which point all GUI windows close, and the script exits.

Great job! In the next section, we’ll analyze results of our method for using an image classifier for object detection purposes.

Image classifier to object detector results using Keras and TensorFlow

At this point, we are ready to see the results of our hard work.

Make sure you use the “Downloads” section of this tutorial to download the source code and example images from this blog post.

From there, open up a terminal, and execute the following command:

$ python detect_with_classifier.py --image images/stingray.jpg --size "(300, 150)"
[INFO] loading network...
[INFO] looping over pyramid/windows took 0.19142 seconds
[INFO] classifying ROIs...
[INFO] classifying ROIs took 9.67027 seconds
[INFO] showing results for 'stingray'

Here, you can see that I have inputted an example image containing a “stingray” which CNNs trained on ImageNet will be able to recognize (since ImageNet contains a “stingray” class).

Figure 7 (top) shows the original output from our object detection procedure.

Notice how there are multiple, overlapping bounding boxes surrounding the stingray.

Applying non-maxima suppression (Figure 7, bottom) collapses the bounding boxes into a single detection.

Let’s try another image, this one of a hummingbird (again, which networks trained on ImageNet will be able to recognize):

$ python detect_with_classifier.py --image images/hummingbird.jpg --size "(250, 250)"
[INFO] loading network...
[INFO] looping over pyramid/windows took 0.07845 seconds
[INFO] classifying ROIs...
[INFO] classifying ROIs took 4.07912 seconds
[INFO] showing results for 'hummingbird'

Figure 8 (top) shows the original output of our detection procedure, while the bottom shows the output after applying non-maxima suppression.

Again, our “image classifier turned object detector” procedure performed well here.

But let’s now try an example image where our object detection algorithm doesn’t perform optimally:

$ python detect_with_classifier.py --image images/lawn_mower.jpg --size "(200, 200)"
[INFO] loading network...
[INFO] looping over pyramid/windows took 0.13851 seconds
[INFO] classifying ROIs...
[INFO] classifying ROIs took 7.00178 seconds
[INFO] showing results for 'lawn_mower'
[INFO] showing results for 'half_track'

At first glance, it appears this method worked perfectly — we were able to localize the “lawn mower” in the input image.

But there was actually a second detection for a “half-track” (a military vehicle that has regular wheels on the front and tank-like tracks on the back):

$ python detect_with_classifier.py --image images/lawn_mower.jpg --size "(200, 200)" --min-conf 0.95
[INFO] loading network...
[INFO] looping over pyramid/windows took 0.13618 seconds
[INFO] classifying ROIs...
[INFO] classifying ROIs took 6.99953 seconds
[INFO] showing results for 'lawn_mower'

By increasing the minimum confidence to 95%, we have filtered out the less confident “half-track” prediction, leaving only the (correct) “lawn mower” object detection.

While our procedure for turning a pre-trained image classifier into an object detector isn’t perfect, it still can be used for certain situations, specifically when images are captured in controlled environments.

In the rest of this series, we’ll be learning how to improve upon our object detection results and build a more robust deep learning-based object detector.

Problems, limitations, and next steps

If you carefully inspect the results of our object detection procedure, you’ll notice a few key takeaways:

1. The actual object detector is slow. Constructing all the image pyramid and sliding window locations takes ~1/10th of a second, and that doesn’t even include the time it takes for the network to make predictions on all the ROIs (4-9 seconds on a 3 GHz CPU)!
2. Bounding box locations aren’t necessarily accurate. The largest issue with this object detection algorithm is that the accuracy of our detections is dependent on our selection of image pyramid scale, sliding window step, and ROI size. If any one of these values is off, then our detector is going to perform suboptimally.
3. The network is not end-to-end trainable. The reason deep learning-based object detectors such as Faster R-CNN, SSDs, YOLO, etc. perform so well is because they are end-to-end trainable, meaning that any error in bounding box predictions can be made more accurate through backpropagation and updating the weights of the network — since we’re using a pre-trained image classifier with fixed weights, we cannot backpropagate error terms through the network.

Throughout this four-part series, we’ll be examining how to resolve these issues and build an object detector similar to the R-CNN family of networks.

What’s next?

Inside today’s tutorial, we covered applying a pre-trained deep learning image classifier for the purposes of deep learning object detection.

These days, deep learning object detectors rely on highly complex CNN architectures that are both difficult to engineer and to train.

I’ve written a deep learning book that covers deep learning fundamentals and basics all the way up to advanced state-of-the-art techniques (including modern deep learning object detection).

If you’re inspired to create your own deep learning projects, I would recommend reading my book Deep Learning for Computer Vision with Python.

I crafted my book so that it perfectly balances theory with implementation, ensuring you properly master:

• Deep learning fundamentals and theory without unnecessary mathematical fluff. I present the basic equations and back them up with code walkthroughs that you can implement and easily understand. You don’t need a degree in advanced mathematics to understand this book.
• How to implement your own custom neural network architectures. Not only will you learn how to implement state-of-the-art architectures, including ResNet, SqueezeNet, etc., but you’ll also learn how to create your own custom CNNs.
• How to train CNNs on your own datasets. Most deep learning tutorials don’t teach you how to work with your own custom datasets. Mine do. You’ll be training CNNs on your own datasets in no time.
• Object detection (Faster R-CNNs, Single Shot Detectors, and RetinaNet) and instance segmentation (Mask R-CNN). Use these chapters to create your own custom object detectors and segmentation networks.

You’ll also find answers and proven code recipes to:

• Create and prepare your own custom image datasets for image classification, object detection, and segmentation
• Work through hands-on tutorials (with lots of code) that not only show you the algorithms behind deep learning for computer vision but their implementations as well
• Put my tips, suggestions, and best practices into action, ensuring you maximize the accuracy of your models

Beginners and experts alike tend to resonate with my no-nonsense teach style and high quality content. In fact, you may wish to read a selection of success stories from my archives if you’re on the fence about taking the next step in your computer vision, deep learning, and artificial intelligence education.

If you’re ready to begin, purchase your copy today. And if you aren’t convinced yet, I’d be happy to send you the full table of contents + sample chapters — simply click here. You can also browse my library of other book and course offerings.

Summary

In this tutorial, you learned how to take any pre-trained deep learning image classifier and turn into an object detector using Keras, TensorFlow, and OpenCV.

To accomplish this task, we combined deep learning with traditional computer vision algorithms:

• In order to detect objects at different scales (i.e., sizes), we utilized image pyramids, which take our input image and repeatedly downsample it.
• To detect objects at different locations, we used sliding windows, which slide a fixed size window from left-to-right and top-to-bottom across the input image — at each stop of the window, we extract the ROI and pass it through our image classifier.
• It’s natural for object detection algorithms to produce multiple, overlapping bounding boxes for objects in an image; in order to “collapse” these overlapping bounding boxes into a single detection, we applied non-maxima suppression.

The end results of our hacked together object detection routine were fairly reasonable, but there were two primary problems:

1. The network is not end-to-end trainable. We’re not actually “learning” to detect objects; we’re instead just taking ROIs and classifying them using a CNN trained for image classification.
2. The object detection results are incredibly slow. On my Intel Xeon W 3 Ghz processor, applying object detection to a single image took ~4-9.5 seconds, depending on the input image resolution. Such an object detector could not be applied in real time.

In order to fix both of these problems, next week, we’ll start exploring the algorithms necessary to build an object detector from the R-CNN, Fast R-CNN, and Faster R-CNN family.

This will be a great series of tutorials, so you won’t want to miss them!

To download the source code to this post (and be notified when the next tutorial in this series publishes), simply enter your email address in the form below!

Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you’ll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!

Download the code!

Website

The post Turning any CNN image classifier into an object detector with Keras, TensorFlow, and OpenCV appeared first on PyImageSearch.

Today, you will learn how to use OpenCV Selective Search for object detection.

Today’s tutorial is Part 2 in our 4-part series on deep learning and object detection:

• Part 1: Turning any deep learning image classifier into an object detector with Keras and TensorFlow
• Part 2: OpenCV Selective Search for Object Detection (today’s tutorial)
• Part 3: Region proposal for object detection with OpenCV, Keras, and TensorFlow (next week’s tutorial)
• Part 4: R-CNN object detection with Keras and TensorFlow (publishing in two weeks)

Selective Search, first introduced by Uijlings et al. in their 2012 paper, Selective Search for Object Recognition, is a critical piece of computer vision, deep learning, and object detection research.

In their work, Uijlings et al. demonstrated:

1. How images can be over-segmented to automatically identify locations in an image that could contain an object
2. That Selective Search is far more computationally efficient than exhaustively computing image pyramids and sliding windows (and without loss of accuracy)
3. And that Selective Search can be swapped in for any object detection framework that utilizes image pyramids and sliding windows

Automatic region proposal algorithms such as Selective Search paved the way for Girshick et al.’s seminal R-CNN paper, which gave rise to highly accurate deep learning-based object detectors.

Furthermore, research with Selective Search and object detection has allowed researchers to create state-of-the-art Region Proposal Network (RPN) components that are even more accurate and more efficient than Selective Search (see Girshick et al.’s follow-up 2015 paper on Faster R-CNNs).

But before we can get into RPNs, we first need to understand how Selective Search works, including how we can leverage Selective Search for object detection with OpenCV.

To learn how to use OpenCV’s Selective Search for object detection, just keep reading.

Looking for the source code to this post?

Jump Right To The Downloads Section

OpenCV Selective Search for Object Detection

In the first part of this tutorial, we’ll discuss the concept of region proposals via Selective Search and how they can efficiently replace the traditional method of using image pyramids and sliding windows to detect objects in an image.

From there, we’ll review the Selective Search algorithm in detail, including how it over-segments an image via:

1. Color similarity
2. Texture similarity
3. Size similarity
4. Shape similarity
5. A final meta-similarity, which is a linear combination of the above similarity measures

I’ll then show you how to implement Selective Search using OpenCV.

Region proposals versus sliding windows and image pyramids

In last week’s tutorial, you learned how to turn any image classifier into an object detector by applying image pyramids and sliding windows.

As a refresher, image pyramids create a multi-scale representation of an input image, allowing us to detect objects at multiple scales/sizes:

Sliding windows operate on each layer of the image pyramid, sliding from left-to-right and top-to-bottom, thereby allowing us to localize where in an image a given object is:

There are a number of problems with the image pyramid and sliding window approach, but the two primary ones are:

1. It’s painfully slow. Even with an optimized-for-loops approach and multiprocessing, looping over each image pyramid layer and inspecting every location in the image via sliding windows is computationally expensive.
2. They are sensitive to their parameter choices. Different values of your image pyramid scale and sliding window size can lead to dramatically different results in terms of positive detection rate, false-positive detections, and missing detections altogether.

Given these reasons, computer vision researchers have looked into creating automatic region proposal generators that replace sliding windows and image pyramids.

The general idea is that a region proposal algorithm should inspect the image and attempt to find regions of an image that likely contain an object (think of region proposal as a cousin to saliency detection).

The region proposal algorithm should:

1. Be faster and more efficient than sliding windows and image pyramids
2. Accurately detect the regions of an image that could contain an object
3. Pass these “candidate proposals” to a downstream classifier to actually label the regions, thus completing the object detection framework

The question is, what types of region proposal algorithms can we use for object detection?

What is Selective Search and how can Selective Search be used for object detection?

The Selective Search algorithm implemented in OpenCV was first introduced by Uijlings et al. in their 2012 paper, Selective Search for Object Recognition.

Selective Search works by over-segmenting an image using a superpixel algorithm (instead of SLIC, Uijlings et al. use the Felzenszwalb method from Felzenszwalb and Huttenlocher’s 2004 paper, Efficient graph-based image segmentation).

An example of running the Felzenszwalb superpixel algorithm can be seen below:

From there, Selective Search seeks to merge together the superpixels to find regions of an image that could contain an object.

Selective Search merges superpixels in a hierarchical fashion based on five key similarity measures:

1. Color similarity: Computing a 25-bin histogram for each channel of an image, concatenating them together, and obtaining a final descriptor that is 25×3=75-d. Color similarity of any two regions is measured by the histogram intersection distance.
2. Texture similarity: For texture, Selective Search extracts Gaussian derivatives at 8 orientations per channel (assuming a 3-channel image). These orientations are used to compute a 10-bin histogram per channel, generating a final texture descriptor that is 8x10x=240-d. To compute texture similarity between any two regions, histogram intersection is once again used.
3. Size similarity: The size similarity metric that Selective Search uses prefers that smaller regions be grouped earlier rather than later. Anyone who has used Hierarchical Agglomerative Clustering (HAC) algorithms before knows that HACs are prone to clusters reaching a critical mass and then combining everything that they touch. By enforcing smaller regions to merge earlier, we can help prevent a large number of clusters from swallowing up all smaller regions.
4. Shape similarity/compatibility: The idea behind shape similarity in Selective Search is that they should be compatible with each other. Two regions are considered “compatible” if they “fit” into each other (thereby filling gaps in our regional proposal generation). Furthermore, shapes that do not touch should not be merged.
5. A final meta-similarity measure: A final meta-similarity acts as a linear combination of the color similarity, texture similarity, size similarity, and shape similarity/compatibility.

The results of Selective Search applying these hierarchical similarity measures can be seen in the following figure:

On the bottom layer of the pyramid, we can see the original over-segmentation/superpixel generation from the Felzenszwalb method.

In the middle layer, we can see regions being joined together, eventually forming the final set of proposals (top).

If you’re interested in learning more about the underlying theory of Selective Search, I would suggest referring to the following resources:

• Efficient Graph-Based Image Segmentation (Felzenszwalb and Huttenlocher, 2004)
• Selective Search for Object Recognition (Uijlings et al., 2012)
• Selective Search for Object Detection (C++/Python) (Chandel, 2017)

Selective Search generates regions, not class labels

A common misconception I see with Selective Search is that readers mistakenly think that Selective Search replaces entire object detection frameworks such as HOG + Linear SVM, R-CNN, etc.

In fact, a couple of weeks ago, PyImageSearch reader Hayden emailed in with that exact same question:

Hi Adrian, I am using Selective Search to detect objects with OpenCV.

However, Selective Search is just returning bounding boxes — I can’t seem to figure out how to get labels associated with these bounding boxes.