pairwise-similarities

Code collection for comparative analysis of text similarity algorithms for detecting near duplicates in JavaDoc software documentation.

Folder structure

similarity: similarity algorithms.
helpers: functions for dataset generation and parsing, caching and similarity algorithm exhaustion.
datasets: source project files and generated datasets.
cache: cached similarities for the test dataset.
pipeline: segmentation and normalization functions.
metrics: functions for algorithm evaluation.
stats: evaluation results.
visualization: functions for result visualization.

Similarity algorithms

Cosine distance (via term frequency vectors)
Longest common subsequence (token-wise)
Levenshtein distance
Locality sensitive hashing
Siamese neural network (via wor2vec embeddings and LSTM)
Word mover's distance (via word2vec)

Usage

Run one of the functions from the metrics folder, providing it with necessary datasets.