Merge pull request #3 from LemurPwned/clip

0.6.0 - Guided video sampling

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,42 @@
+---
+name: 🐛 Bug report
+about: If something isn't working 🔧
+title: ''
+labels: bug
+assignees:
+---
+
+## 🐛 Bug Report
+
+<!-- A clear and concise description of what the bug is. -->
+
+## 🔬 How To Reproduce
+
+Steps to reproduce the behavior:
+
+1. ...
+
+### Code sample
+
+<!-- If applicable, attach a minimal code sample to reproduce the decried issue. -->
+
+### Environment
+
+* OS: [e.g. Linux / Windows / macOS]
+* Python version, get it with:
+
+```bash
+python --version
+```
+
+### Screenshots
+
+<!-- If applicable, add screenshots to help explain your problem. -->
+
+## 📈 Expected behavior
+
+<!-- A clear and concise description of what you expected to happen. -->
+
+## 📎 Additional context
+
+<!-- Add any other context about the problem here. -->

.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,3 @@
+# Configuration: https://help.github.com/en/github/building-a-strong-community/configuring-issue-templates-for-your-repository
+
+blank_issues_enabled: false

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,23 @@
+---
+name: 🚀 Feature request
+about: Suggest an idea for this project 🏖
+title: ''
+labels: enhancement
+assignees:
+---
+
+## 🚀 Feature Request
+
+<!-- A clear and concise description of the feature proposal. -->
+
+## 🔈 Motivation
+
+<!-- Please describe the motivation for this proposal. -->
+
+## 🛰 Alternatives
+
+<!-- A clear and concise description of any alternative solutions or features you've considered. -->
+
+## 📎 Additional context
+
+<!-- Add any other context or screenshots about the feature request here. -->

.github/ISSUE_TEMPLATE/question.md

@@ -0,0 +1,25 @@
+---
+name: ❓ Question
+about: Ask a question about this project 🎓
+title: ''
+labels: question
+assignees:
+---
+
+## Checklist
+
+<!-- Mark with an `x` all the checkboxes that apply (like `[x]`) -->
+
+- [ ] I've searched the project's [`issues`](https://github.com/LemurPwned/video-sampler/issues).
+
+## ❓ Question
+
+<!-- What is your question -->
+
+How can I [...]?
+
+Is it possible to [...]?
+
+## 📎 Additional context
+
+<!-- Add any other context or screenshots about the feature request here. -->

.github/workflows/build.yml

@@ -0,0 +1,40 @@
+name: Release to PyPI
+
+
+on:
+  pull_request:
+    types: [closed]
+    branches: [master]
+  workflow_dispatch:
+    inputs:
+      dry-run: 
+        required: true
+        default: true
+        type: boolean
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.9"]
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v2.2.2
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install flit
+        flit install --deps production    
+    - name: Configure test pypi
+      run: |
+        flit publish --repository testpypi
+    - name: Build 
+      run: |
+        flit build 
+        flit publish

.github/workflows/docker.yml

@@ -0,0 +1,22 @@
+name: Docker Image CI
+
+on:
+  push:
+    branches: [ "main" ]
+  workflow_dispatch:
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v3
+    - name: Login to GH container registry
+      uses: docker/login-action@v1
+      with:
+        registry: ghcr.io
+        username: ${{github.actor}}
+        password: ${{secrets.GITHUB_TOKEN}}
+    - name: Build and push
+      run: |
+        docker build . --tag ghcr.io/lemurpwned/video-sampler:latest
+        docker push ghcr.io/lemurpwned/video-sampler:latest
+          

.github/workflows/test.yml

@@ -0,0 +1,36 @@
+# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
+# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
+
+name: Test sampling code
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+  workflow_dispatch:
+
+jobs:
+  build:
+
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: true
+      matrix:
+        python-version: ["3.10", "3.11"]
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install flake8 pytest
+        python -m pip install -e .[dev,test,clip]
+        
+    - name: Test with pytest
+      run: |
+        pytest

.gitignore

@@ -2,7 +2,8 @@
 # Created by https://www.gitignore.io/api/osx,python,pycharm,windows,visualstudio,visualstudiocode
 # Edit at https://www.gitignore.io/?templates=osx,python,pycharm,windows,visualstudio,visualstudiocode
 notebooks/
-
+evaluation.py
+gifs.py
 ### OSX ###
 # General
 .DS_Store

CHANGELOG.md

@@ -0,0 +1,22 @@
+# Changelog
+
+## Description
+
+Changelog for the `video-sampler`.
+
+### 0.6.0
+
+- adding CLIP-based gating for sampling
+- added docker build image
+- fixed some bugs in requirements and setup scripts
+- adding gif creation from sampled frames
+- added a notebook in `notebooks` to show the distribution shift of the sampled frames
+- added some basic tests (really basic)
+- added a short explanation and usage of guided sampling in README.md.
+- added grid hash sampling
+
+### 0.5.0
+
+- fixed a major bug in the hashing buffer
+- added entropy and gzip methods for the sliding top-k buffer
+- improved build process

Dockerfile

@@ -0,0 +1,13 @@
+FROM python:3.11
+
+# let's copy all the necessary files
+# we install our only 2 dependencies :) and vim for nice workflow
+RUN apt-get update && \
+    apt-get install -y vim
+RUN python3 -m pip install --upgrade pip
+WORKDIR /scratch
+RUN git clone https://github.com/LemurPwned/video-sampler.git && \
+    cd video-sampler && \
+    python3 -m pip install .
+WORKDIR /app
+RUN video_sampler --help

README.md

@@ -13,7 +13,7 @@
 
 
 Video sampler allows you to efficiently sample video frames.
-Currently it uses keyframe decoding, frame interval gating and perceptual hashing to reduce duplicated samples.
+Currently, it uses keyframe decoding, frame interval gating and perceptual hashing to reduce duplicated samples.
 
 **Use case:** for sampling videos for later annotations used in machine learning.
 
@@ -45,13 +45,13 @@ python3 -m video-sampler hash FatCat.mp4 ./dataset-frames/ --hash-size 3 --buffe
 
 ### Advanced usage
 
-There are currently 3 sampling methods:
+There are 3 sampling methods available:
 
 - `hash` - uses perceptual hashing to reduce duplicated samples
 - `entropy` - uses entropy to reduce duplicated samples (work in progress)
 - `gzip` - uses gzip compressed size to reduce duplicated samples (work in progress)
 
-To launch either you can run
+To launch any of them you can run and substitute `method-name` with one of the above:
 
 ```bash
 video_sampler buffer `method-name` ...other options
@@ -65,6 +65,59 @@ video_sampler buffer entropy --buffer-size 20 ...
 
 where `buffer-size` for `entropy` and `gzip` mean the top-k sliding buffer size. Sliding buffer also uses hashing to reduce duplicated samples.
 
+## Gating
+
+Aside from basic sampling rules, you can also apply gating rules to the sampled frames, further reducing the number of frames.
+Right now, there is only one gating rule available, which is based on CLIP model.
+
+Here's a quick example of how to use it:
+
+```bash
+python3 -m video_sampler clip ./videos ./scratch/clip --pos-samples "a cat" --neg-samples "empty background, a lemur"  --hash-size 4
+```
+
+### CLIP-based gating comparison
+
+Here's a brief comparison of the frames sampled with and without CLIP-based gating with the following config:
+
+```python
+  gate_def = dict(
+      type="clip",
+      pos_samples=["a cat"],
+      neg_samples=[
+          "an empty background",
+          "text on screen",
+          "a forest with no animals",
+      ],
+      model_name="ViT-B-32",
+      batch_size=32,
+      pos_margin=0.2,
+      neg_margin=0.3,
+  )
+```
+
+Evidently, CLIP-based gating is able to filter out frames that do not contain a cat and in consequence, reduce the number of frames with plain background. It also thinks that a lemur is a cat, which is not entirely wrong as fluffy creatures go.
+
+|                      Pass gate (no gating)                      |                            CLIP gate                            |                            Grid                             |
+| :-------------------------------------------------------------: | :-------------------------------------------------------------: | :---------------------------------------------------------: |
+|   <img width="256" src="./assets/FatCat.mp4_hash_4_pass.gif">   |   <img width="256" src="./assets/FatCat.mp4_hash_4_clip.gif">   | <img width="256" src="./assets/FatCat.mp4_grid_4_pass.gif"> |
+|  <img width="256" src="./assets/SmolCat.mp4_hash_4_pass.gif">   |  <img width="256" src="./assets/SmolCat.mp4_hash_4_clip.gif">   | <img width="256" src="./assets/FatCat.mp4_grid_4_pass.gif"> |
+| <img width="256" src="./assets/HighLemurs.mp4_hash_4_pass.gif"> | <img width="256" src="./assets/HighLemurs.mp4_hash_4_clip.gif"> | <img width="256" src="./assets/FatCat.mp4_grid_4_pass.gif"> |
+
+The effects of gating in numbers, for this particular set of examples (see `produced` vs `gated` columns). `produced` represents the number of frames sampled without gating, here after the perceptual hashing, while `gated` represents the number of frames sampled after gating.
+
+| video          | buffer | gate | decoded | produced | gated |
+| -------------- | ------ | ---- | ------- | -------- | ----- |
+| FatCat.mp4     | grid   | pass | 179     | 31       | 31    |
+| SmolCat.mp4    | grid   | pass | 118     | 24       | 24    |
+| HighLemurs.mp4 | grid   | pass | 161     | 35       | 35    |
+| FatCat.mp4     | hash   | pass | 179     | 101      | 101   |
+| SmolCat.mp4    | hash   | pass | 118     | 61       | 61    |
+| HighLemurs.mp4 | hash   | pass | 161     | 126      | 126   |
+| FatCat.mp4     | hash   | clip | 179     | 101      | 73    |
+| SmolCat.mp4    | hash   | clip | 118     | 61       | 31    |
+| HighLemurs.mp4 | hash   | clip | 161     | 126      | 66    |
+
 ## Benchmarks
 
 Configuration for this benchmark:

pyproject.toml

@@ -2,7 +2,7 @@
 name = "video_sampler"
 description = "Video Sampler -- sample frames from a video file"
 url = "https://github.com/LemurPwned/video-sampler"
-version = "0.5.0"
+version = "0.6.0"
 authors = [
     { name = "LemurPwned", email = "lemurpwned@gmail.com" }
 ]
@@ -40,11 +40,18 @@ Source = "https://github.com/LemurPwned/video-sampler"
 video_sampler = "video_sampler.__main__:main_loop"
 
 [project.optional-dependencies]
+clip = [
+  "open_clip_torch >= 2.23.0",
+  "torch >= 2.1.0"
+]
 dev = [
     "pre-commit",
-    "bump2version"
+    "bump2version",
+    "tabulate",
+]
+test = [
+  "pytest"
 ]
-
 [build-system]
 build-backend = "flit_core.buildapi"
 requires = ["flit_core >=3.2,<4"]