ruby - Computing Document Similarity Matrices in Sphinx? -

ruby - Computing Document Similarity Matrices in Sphinx? -

does sphinx provide way precompute document similarity matrices? have looked @ sphinx/solr/lucene; seems lucene able indirectly using term vectors computing document similarity term vectors.

currently using tf-idf-similarity gem these calculations, incredibly slow dataset grows; on^(n-1!).

currently trying find faster alternative this. lucene seems potential solution, doesn't have much back upwards within ruby community if sphinx has way of doing ideal.

just clarify; not trying live search similarity matching, appears mutual utilize case both lucene , sphinx, trying precompute similarity matrix create similarity between documents dataset. subsequently used in info visualizations different types of user analysis.

also prior experience doing i'm curious benchmarks. how looks in terms of time process , how much computing powerfulness and/or parallelization using reference number of documents , doc size average.

currently taking 40 minutes me process 4000 documents , 2 hours process 6400 records. providing 2 different sizes , times here give indication of growth expansion can see how slow become big datasets.

ruby lucene similarity tf-idf cosine-similarity