Duplicates Detector is a cross-platform GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for efficient and convenient operation.

Advanced Duplicate File Finder for Python

Tool for removing duplicate documents from Elasticsearch

Advanced similarity and duplicate source code proof of concept for our research efforts.

QGIS Processing plugin to add an algorithm for upserting features from a source vector layer to an existing target vector layer.

Find, remove and avoid duplicates with dugu: The Duplicates Guru

A command-line tool which automate the deletion of duplicate files based on their hash or perceptual-hash.

Uses SSIM and MSE to get rid of duplicates and near duplicates

A python program to locate duplicate files - and do it fast

Remove duplicates from parallel corpora

This program finds duplicate files in a folder and its subfolders. Duplicates are moved to a separate folder. A few other modes of operation are also planned/available.

A project aiming to leverage text embeddings and Milvus, a high-performance vector search engine, to detect duplicate job postings.

This cleans up duplicate SMS entries in a backup created by SMS Backup & Restore Android app.

Reusable Django admin action for merging duplicated entries.

Python scripts for finding audio duplicates in a dataset using chroma features and openCV functions. The approach uses raw audio information (transformed to the chroma features) without any metadata.

When Importing multiple CSV files Bitwarden creates Duplicate Entries. So this Python script will remove duplicate entries and keep ONE.

Compares the files in a folder with md5 checksums and deletes duplicate files or moves them to the desired folder.

A fast random number file generator, that generates a text document of randomized integers (uniquely random also supported) within the specified constraints. This project was created to aid in benchmarking my own Double-Edged Sorting algorithm.

🐻 The decluttering deduplicator

a tool to predict highly similar duplicates (HSDs) in eukaryotes