Leksika

A web-based system to detect similarity between thesis titles using a hybrid approach: TF-IDF Cosine Similarity combined with Jaccard Similarity. Developed as part of research at the Informatics Engineering program, Unimal.

The problem

As the number of submitted theses grows, manually checking for title duplication becomes impractical. The system quantifies similarity objectively, flagging titles that are too close to existing records.

Text preprocessing pipeline

Before any similarity calculation, each title goes through five stages:

1. Case folding — lowercase everything
2. Cleansing — strip punctuation, numbers, special characters
3. Tokenisation — split into individual tokens
4. Stopword removal — using the Sastrawi Indonesian stopword list
5. Stemming — reduce words to their root form (Sastrawi)

This matters for Indonesian: “pembelajaran” and “belajar” and “pelajar” share a root. Without stemming, they score as completely different terms.

Hybrid scoring

TF-IDF Cosine handles semantic weight — common words across the corpus are down-weighted automatically. Jaccard covers token overlap at the surface level. The final score combines both, which reduces false negatives that either method alone would miss.