+2
日增长
5,684
stars
477
forks
59
watchers
103
prs
135
issues
11
releases
README
Image Deduplicator (imagededup)
imagededup is a python package that simplifies the task of finding exact and near duplicates in an image collection.
This package provides functionality to make use of hashing algorithms that are particularly good at finding exact
duplicates as well as convolutional neural networks which are also adept at finding near duplicates. An evaluation
framework is also provided to judge the quality of deduplication for a given dataset.
Following details the functionality provided by the package:
- Finding duplicates in a directory using one of the following algorithms:
- Convolutional Neural Network (CNN) - Select from several prepackaged models or provide your own custom model.
- Perceptual hashing (PHash)
- Difference hashing (DHash)
- Wavelet hashing (WHash)
- Average hashing (AHash)
- Generation of encodings for images using one of the above stated algorithms.
- Framework to evaluate effectiveness of deduplication given a ground truth mapping.
- Plotting duplicates found for a given image file.
Detailed documentation for the package can be found at: https://idealo.github.io/imagededup/
imagededup is compatible with Python 3.9+ and runs on Linux, MacOS X and Windows.
It is distributed under the Apache 2.0 license.
📖 Contents
- Installation
- Quick Start
- Benchmarks
- Contribute
- Citation
- Maintainers
- License
⚙️ Installation
There are two ways to install imagededup:
- Install imagededup from PyPI (recommended):
pip install imagededup
- Install imagededup from the GitHub source:
bash
git clone https://github.com/idealo/imagededup.git
cd imagededup
pip install .
🚀 Quick Start
In order to find duplicates in an image directory using perceptual hashing, following workflow can be used:
- Import perceptual hashing method
python
from imagededup.methods import PHash
phasher = PHash()
- Generate encodings for all images in an image directory
python
encodings = phasher.encode_images(image_dir='path/to/image/directory')
- Find duplicates using the generated encodings
python
duplicates = phasher.find_duplicates(encoding_map=encodings)
- Plot duplicates obtained for a given file (eg: 'ukbench00120.jpg') using the duplicates dictionary
python
from imagededup.utils import plot_duplicates
plot_duplicates(image_dir='path/to/image/directory',
duplicate_map=duplicates,
filename='ukbench00120.jpg')
The output looks as below:
The complete code for the workflow is:
python
from imagededup.methods import PHash
phasher = PHash()
Generate encodings for all images in an image directory
encodings = phasher.encode_images(image_dir='path/to/image/directory')
Find duplicates using the generated encodings
duplicates = phasher.find_duplicates(encoding_map=encodings)
plot duplicates obtained for a given file using the duplicates dictionary
from imagededup.utils import plot_duplicates
plot_duplicates(image_dir='path/to/image/directory',
Top issues
#191Deduping large amount of images (1mil+)9 comments · 2023-04-14#64Tensorflow Requirement6 comments · 2019-10-26#225Fail compiling on iOS.5 comments · 2024-12-20#199No module named 'imagededup.evaluation'4 comments · 2023-07-26#189Add a device parameter to perform encoding using different torch devices such as mps4 comments · 2023-03-07#78Illegal instruction4 comments · 2019-12-19#232GPU OOMs on large directory scans3 comments · 2025-02-22#213CUDA OOM3 comments · 2024-01-23
Discussions
#245Problem with .heic0 comments · 2026-05-09#240Simple UI for imagededup0 comments · 2026-02-11#231Deduping two separate directories0 comments · 2025-02-21#147possible feature request0 comments · 2021-04-26#138Support for python 3.62 comments · 2021-01-25
Latest release
Documentation and build pipeline improvements v0.3.3.post2 2025-08-15What's Changed Avoid download on import by @bogdan-galileo in https://github.com/idealo/imagededup/pull/201 Address documentation improvements and other minor changes by @tanujjain in https://github.com/idealo/imagededup/pull/238 Build pipeline now also tests built wheels on mac and manylinux after building them Update project Readme to reflect the need for python 3.9 as the minimum version Breaking changes None New Contributors @bogdan-galileo made their first contribution in https://github.com/idealo/imagededup/pull/201 Full Changelog: https://github.com/idealo/imageded