The TakeLab Semantic Text Similarity System

Version: 1.0
Release date: April 18, 2012

1 Description

The TakeLab Semantic Text Similarity System was entered as task6-takelab-simple in the SemEval-2012: Semantic Evaluation Exercises to perform the Semantic Textual Similarity task:

Given two sentences, s1 and s2, the system provides a similarity score as a floating point number on the scale of 0 (no relation) to 5 (semantic equivalence).
The SemEval-2012 organizers measured the performance of our system by comparing its output with the aggregated similarity scores obtained from human subjects on the same data set using the Pearson correlation coefficient.

Please see this page for more detailed information.

If you use the software or the datasets, please cite the following paper:

Šarić, F., Glavaš G., Karan M., Šnajder J., Dalbelo Bašić B.: TakeLab: Systems for Measuring Semantic Text Similarity. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012). pp. 441-448. ACL, Montreal, Canada (7-8 June 2012).

The BibTeX format is:
  author    = {\v{S}ari\'{c}, Frane  and  Glava\v{s}, Goran  and  Karan, Mladen  
               and  \v{S}najder, Jan  and  Dalbelo Ba\v{s}i\'{c}, Bojana},
  title     = {TakeLab: Systems for Measuring Semantic Text Similarity},
  booktitle = {Proceedings of the Sixth International Workshop on Semantic Evaluation
               {(SemEval 2012)}},
  month     = {7-8 June},
  year      = {2012},
  address   = {Montr\'{e}al, Canada},
  publisher = {Association for Computational Linguistics},
  pages     = {441--448},
  url       = {}

2 Instructions

2.1 Installation prerequisites

This system requires the following packages: Python 2.7, NLTK, NumPy, and LIBSVM.
If you are using Ubuntu Linux or a similar distribution, you can install all of the prerequisites by typing:
sudo apt-get install python-nltk python-numpy libsvm-tools

Once you install the NLTK package, you need to issue the following commands to download the POS tagger model and the WordNet corpus:
python -c "import nltk;"
When prompted, please select the maxent_treebank_pos_tagger model and the wordnet corpus.

2.2 Download

The source code and the data files can be downloaded from this location:

2.3 Usage

Generating the features

python train/STS.input.MSRvid.txt train/ > msrvid-train.txt
python test/STS.input.MSRvid.txt > msrvid-test.txt
will generate the requisite features from the train/STS.input.MSRvid.txt and test/STS.input.MSRvid.txt files respectively. Note: In order to avoid an unreasonably large file download, the provided implementation contains the file with the word frequencies (word-frequencies.txt) and the LSA word vectors (nyt_words.txt, nyt_word_vectors.txt, wikipedia_words.txt and wikipedia_word_vectors.txt) which were filtered to contain only the words appearing in the official train and test sets. Using the larger unfiltered datasets on the provided train and test sets will not affect the output of this software in any way.

Obtaining the model parameters

Executing the provided shell script
./ msrvid-train.txt
will find the optimal model parameters for the Support Vector Regression using LIBSVM. The above command will finish with the following output:
Best correlation: 0.873686
Type: svm-train -s 3 -t 2 -c 200 -g .02 -p .5 msrvid-train.txt model.txt
Thus the optimal LIBSVM parameters for msrvid-train.txt are: -s 3 -t 2 -c 200 -g .02 -p .5.
Note: The best correlation and the set of optimal parameters might vary slightly depending on the installed version of NLTK, the WordNet corpus, and LIBSVM.


By copy-pasting the command suggested by, i.e. the text following "Type:"
svm-train -s 3 -t 2 -c 200 -g .02 -p .5 msrvid-train.txt model.txt
one trains the model for MSR-Video corpus.
The training will finish with the following output:
Warning: using -h 0 may be faster
optimization finished, #iter = 2777
nu = 0.431544
obj = -33239.258433, rho = -2.938525
nSV = 337, nBSV = 311


To evaluate the msrvid model against the msrvid test set, one can issue
svm-predict msrvid-test.txt model.txt msrvid-output.txt
which should output the following:
Mean squared error = 7.03655 (regression)
Squared correlation coefficient = -nan (regression)
The obtained sentence similarity scores need postprocessing by the script
python test/STS.input.MSRvid.txt msrvid-output.txt
which handles some trivial corner cases of our scoring function. To obtain the Pearson correlation, one can run the script which was provided by the SemEval organizers alongside the training data:
perl msrvid-output.txt test/
which outputs
Pearson: 0.88516
indicating that the sentence similarity scores produced by our system have the Pearson correlation of 0.88516 when compared to the similarity scores reported by the human subjects (contained in the file test/ Please note that the system that we provide on this page differs slightly from the system that was submitted to SemEval. A couple of features described in the paper are not used in this implementation (in particular sentence length and the number of differing lemmas) and the implementation of some features might be slightly different. Please email us if you notice any major differences.

2.4 Additional data

We used the Google Books Ngrams dataset to obtain word frequencies. The word-frequencies.txt file provided herein is filtered to only the words appearing in the train and test sets. However, providing a larger, unfiltered file does not change the output of the system in any way. Several features use the word LSA vectors obtained from the Wikipedia and New York Times Annotated Corpus. Since the full matrices containing all the LSA vectors are very large, we provide only the shortened versions containing the vectors corresponding to train and test set words. If you are interested in full matrices or the code that generates them, please email

3 License

This code is available under a derivative of a BSD-license that requires proper attribution. Essentially, you can use, modify and sell this software or the derived software in academic and non-academic, commercial and non-commercial, open-source and closed-source settings, but you have to give proper credit.