examples/face-obj-detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DSC160 Data Science and the Arts - Twomey - Spring 2020 - [dsc160.roberttwomey.com](http://dsc160.roberttwomey.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Face and Object Detection\n",
    "\n",
    "This notebook demonstrates simple the Viola-Jones face detection on a painting. This is implemented in opencv.\n",
    "\n",
    "Viola and Jones, 'Rapid Object Detection using a Boosted Cascade of Simple\n",
    "Features' (2007) http://wearables.cc.gatech.edu/paper_of_week/viola01rapid.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Face and Eye Detection with Haar Cascade Classifiers in OpenCV\n",
    "\n",
    "- Here is a good video visualization of Haar Cascade Classifier in action, from Adam Harvey: https://vimeo.com/12774628\n",
    "- His project CVDazzle has to do with fashion/makeup design to avoid face classifiers: https://cvdazzle.com/\n",
    "- He gave a good interview on the Viola-Jones face detecter: https://web.archive.org/web/20171204220159/http://www.makematics.com/research/viola-jones/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have not yet installed opencv on your instance of datahub (or your local machine), uncomment the following command and run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install opencv-python-headless --user\n",
    "#!pip install opencv-contrib-python-headless --user"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import modules and do some setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skimage import io\n",
    "from scipy import ndimage\n",
    "from skimage.color import rgb2hsv\n",
    "import skimage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See if a haar face detector can find the face in a famous portrait"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "portrait = io.imread('https://uploads4.wikiart.org/images/leonardo-da-vinci/mona-lisa.jpg!Large.jpg')\n",
    "fig1 = plt.figure(figsize=(6, 12))\n",
    "ax = plt.imshow(portrait)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Haar cascade"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a machine learning based approach where a cascade function is trained from a lot of positive and negative images. It is then used to detect objects on test images.\n",
    "\n",
    "The algorithm has four stages:\n",
    "\n",
    "1. __Haar Feature Selection__ - First step is to collect the Haar Features.  A Haar feature considers adjacent rectangular regions at a specific location in a detection window, sums up the pixel intensities in each region and calculates the difference between these sums.\n",
    "\n",
    "2. __Creating  Integral Images__ - Integral image technique is used to speeden up calculating sum of pixels. Involves building precomputed matrices.\n",
    "\n",
    "3. __Adaboost Training__ - Adaboost selects the best features and trains the classifiers that use them. \n",
    "\n",
    "4. __Cascading Classifiers__ - The cascade classifier consists of a collection of stages, where each stage is an ensemble of weak learners. The weak learners are simple classifiers. The stages are designed to reject negative samples as fast as possible. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_frontalface_default.xml')\n",
    "eye_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_eye.xml')\n",
    "\n",
    "portrait_cv = cv2.cvtColor(portrait, cv2.COLOR_RGB2BGR)\n",
    "gray = cv2.cvtColor(portrait_cv, cv2.COLOR_BGR2GRAY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "detect with cascade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "faces = face_cascade.detectMultiScale(gray, 1.3, 5)\n",
    "for (x,y,w,h) in faces:\n",
    "    portrait_cv = cv2.rectangle(portrait_cv,(x,y),(x+w,y+h),(255,0,0),2)\n",
    "    roi_gray = gray[y:y+h, x:x+w]\n",
    "    roi_color = portrait_cv[y:y+h, x:x+w]\n",
    "    eyes = eye_cascade.detectMultiScale(roi_gray)\n",
    "    for (ex,ey,ew,eh) in eyes:\n",
    "        cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "show results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig1 = plt.figure(figsize=(6, 12))\n",
    "ax = plt.imshow(cv2.cvtColor(portrait_cv, cv2.COLOR_BGR2RGB))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also try to detect the smile using a pretrained smile cascade:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smile_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + 'haarcascade_smile.xml')\n",
    "\n",
    "faces = face_cascade.detectMultiScale(gray, 1.3, 5)\n",
    "for (x,y,w,h) in faces:\n",
    "    portrait_cv = cv2.rectangle(portrait_cv,(x,y),(x+w,y+h),(255,0,0),2)\n",
    "    roi_gray = gray[y:y+h, x:x+w]\n",
    "    roi_color = portrait_cv[y:y+h, x:x+w]\n",
    "    eyes = eye_cascade.detectMultiScale(roi_gray)\n",
    "    for (ex,ey,ew,eh) in eyes:\n",
    "        cv2.rectangle(roi_color,(ex,ey),(ex+ew,ey+eh),(0,255,0),2)\n",
    "    smiles = smile_cascade.detectMultiScale(roi_gray)\n",
    "    for (sx,sy,sw,sh) in smiles:\n",
    "        cv2.rectangle(roi_color,(sx,sy),(sx+sw,sy+sh),(0,0,255),2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "show results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig1 = plt.figure(figsize=(6, 12))\n",
    "ax = plt.imshow(cv2.cvtColor(portrait_cv, cv2.COLOR_BGR2RGB))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: both the eye and the mouth are \"detected\" as smiles/mouths.\n",
    "\n",
    "For what genres of painting would it be useful to detect faces a image features? (metadata)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "Most contemporary face and object detection is accomplished through convolutional neural networks. We will talk about this in week 4. \n",
    "\n",
    "Harr Cascade Classifiers still have some use on compute-constrained platforms, for instance embedded systems (raspberry pi), and for realtime applications.\n",
    "\n",
    "- Viola and Jones, 'Rapid Object Detection using a Boosted Cascade of Simple\n",
    "Features' (2007) http://wearables.cc.gatech.edu/paper_of_week/viola01rapid.pdf\n",
    "- Opencv-Python [Tutorial on Face Detection](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html#face-detection)\n",
    "- Extended video lecture on Haar Classifiers: https://www.youtube.com/watch?v=WfdYYNamHZ8"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}