What metadata and hash databases are used to identify known CSAM files?

1. How hashing actually works — cryptographic vs perceptual fingerprints

The fundamental technical method is hashing: transforming an image or video into a short signature that can be compared quickly; cryptographic hashes (e.g., MD5) change if any bit of a file changes and are useful for deduplication, while perceptual hashes (PhotoDNA, PDQ and similar algorithms) are designed to treat visually similar but edited files as matches so known CSAM can be detected despite resizing, reencoding or minor edits ^{[1] [2] [6]}.

2. The major hash systems and vendor offerings in use today

PhotoDNA is widely cited as a foundational perceptual-hash system used by many platforms to detect images that match previously identified CSAM, while Meta’s PDQ and TMK+PDQF have been openly shared for image/video perceptual hashing; proprietary commercial offerings such as Safer (Thorn) claim very large verified hash repositories and specialized video hashing like SSVH for scene‑sensitive video matching ^{[1] [2] [5] [7]}.

3. The databases: NCMEC, CAID, Project VIC and commercial lists

National clearinghouses and law‑enforcement collections feed the hashes platforms use: NCMEC maintains assessed CSAM databases and its CyberTipline acts as the reporting clearinghouse for industry reports; CAID (UK Child Abuse Image Database) and Project VIC are law‑enforcement-focused image repositories whose hashes and metadata are used in investigations; commercial providers (Thorn/Safer and other vendors) maintain large verified hash lists that clients match against, often numbering in the millions to tens of millions of hashes ^{[6] [8] [9] [7] [10]}.

4. Metadata and contextual signals that augment hash matching

Beyond image hashes, platforms and tools use contextual metadata—file timestamps, EXIF/geotags, filenames, IP addresses and conversation context—to triage, prioritize and corroborate detected material; AI classifiers also analyze visual and textual cues to flag previously unknown CSAM for human review before hashing and inclusion in databases ^{[3] [11] [2]}.

5. Practical realities: who can access what, and the chain to law enforcement

Access is not uniform: NCMEC and some law‑enforcement hash repositories are considered sensitive and are typically available only to major participating platforms or investigators, while commercial hash lists are licensed to customers; when a match occurs, procedures commonly require removal/quarantine of content and formal reporting to authorities such as NCMEC’s CyberTipline, which then forwards to law enforcement ^{[6] [1] [11]}.

6. Limits, tradeoffs and competing agendas

Hash matching excels for “known” CSAM but cannot detect newly created or uncatalogued material without classifiers and human review, and perceptual hashing balances sensitivity against false positives—an issue that has made some platforms cautious about automated CSAM classifiers given real consequences for misidentification ^{[2] [3]}. Advocacy, commercial and platform actors have vested interests: nonprofits like Thorn promote broad deployment and have commercialized services (Safer) that tout large hash counts, platforms emphasize scalability and legal compliance (Tech Coalition members), and civil‑liberties voices highlight access, privacy and centralized control of sensitive hash repositories ^{[7] [1] [6]}.

7. The bottom line for identification workflows

In practice, systems combine perceptual and cryptographic hashing (PhotoDNA, PDQ, MD5 and vendor algorithms), searchable law‑enforcement hash databases (NCMEC, CAID, Project VIC) and commercial hash lists (Thorn/Safer et al.), supplemented by metadata cues and human verification; these components together generate the vast majority of automated CSAM detections and the subsequent reports to authorities, while leaving open the hard problem of reliably finding genuinely new abuse material without unacceptable error rates ^{[1] [4] [5] [2]}.