Count occurrences of the same media item

[dennyabrain]

For every media item that we receive on the tipline, we need to show to the users how many occurences of this exact file exist on the server. Given our infra, the scope of this task is to

• Identify what is the best technique to do this
• Propose changes in infrastructure or data to solve this
• Create Feluda workers
• Add the hash operator value to a PostgreSQL Database in the worker #91
• Evaluate the CRC worker and Hash Worker #102

[aatmanvaidya]

1. Identify what is the best technique to do this

All the popular methods use hashing

• https://stackoverflow.com/questions/1072569/see-if-two-files-have-the-same-content-in-python
• https://stackoverflow.com/questions/748675/finding-duplicate-files-and-removing-them
• https://stackoverflow.com/questions/6374717/should-all-implementations-of-sha512-give-the-same-hash

filecmp

• https://docs.python.org/3/library/filecmp.html

Audio similarity

• https://stackoverflow.com/questions/38971969/how-to-compare-audio-on-similarity-in-python
• Librosa - https://github.com/d4r3topk/comparing-audio-files-python/blob/master/mfcc.py
• https://librosa.org/doc/main/generated/librosa.segment.cross_similarity.html#librosa.segment.cross_similarity
• https://dev.to/hiisi13/find-an-audio-within-another-audio-in-10-lines-of-python-1866

Video Simiarity

• https://pypi.org/project/vpdq/0.0.1/
• https://stackoverflow.com/questions/28935851/how-to-compare-a-video-with-a-reference-video-using-opencv-and-python
• https://docs.opencv.org/2.4.10/doc/tutorials/highgui/video-input-psnr-ssim/video-input-psnr-ssim.html?highlight=video
• https://stackoverflow.com/questions/42902100/comparing-two-exact-videos-using-opencv-python
• https://en.wikipedia.org/wiki/Structural_similarity_index_measure
• https://stackoverflow.com/a/33773833/13864403

[aatmanvaidya]

In the short term hashing is the way to check if files are same or not. It is also the fastest way to do so

Are SHA-256 and SHA-512 collision resistant?

• https://crypto.stackexchange.com/a/89559

The probability of collision is very very low

We call an event is-not-gonna-happen if it has probability <1/2^100

You can use any 512-bit cryptographic hash function like SHA-512, SHA3-512, and BLAKE2b 
without fear of collision. 
You may look at BLAKE2b quite fast compared to alternatives and its parallel version BLAKE3.

[dennyabrain]

In the short term hashing is the way to check if same or not. It is also the fastest way to do so

Great. Then lets move onto checking if they apply for our use case. Do share the various way you test out media items received on whatsapp. To keep as a log of things that worked and which did not.

[aatmanvaidya]

In the short term hashing is the way to check if same or not. It is also the fastest way to do so

Great. Then lets move onto checking if they apply for our use case. Do share the various way you test out media items received on whatsapp. To keep as a log of things that worked and which did not.

Yes I have started working on this, can you also see the updated comment with the stackoverflow link that talks about how sha256 and sha512 are collision resistant

We should consider using Blacke3 over sha512. It is much much faster

• https://github.com/BLAKE3-team/BLAKE3

[aatmanvaidya]

Time taken by blake2b to find the hash of audio and video files of different lengths and sizes

Audio

Media Type - Length	Time Taken
audio - 30s	0.018s
audio - 60s	0.027s
audio - 120s	0.056s
audio - 300s	0.122s
audio - 600s	0.234s
audio - 1200s	0.425s
audio - 1800s	0.631s

Video

Media Type - Length	Time Taken
video - 30s	0.0081s
video - 60s	0.013s
video - 300s	0.022s
video - 600s	0.074s
video - 1200s	0.087s
video - 1800s	0.148s
video - 3600s	0.33s