Home / Java / Search

Elasticsearch plugin for nearest neighbor search. Store vectors and run similarity search using exact and approximate algorithms.

README
Issues

Elastiknn

Elasticsearch Plugin for Nearest Neighbor Search

Documentation

Documentation at elastiknn.klibisz.com

Community

Join the Gitter channel at gitter.im/elastiknn/community.

If you're using Elastiknn in your research or work, I'd love to hear feedback. Post an issue, post in the Gitter channel, or contact me directly (contact info in my Github bio).

Builds and Releases

Item	Status
Github CI Build
Github Release Build
Plugin Release
Plugin Snapshot
Python Client, Release
Scala 2.12 Client, Release
Scala 2.12 Client, Snapshot

License

Elastiknn is licensed under GPLv3. See License.txt. If you have any questions, contact me.

12 Open More issues

Closed

How to use image search

When I used image search, I returned all documents from elasticsearch.How does the elastiknn plugin generate image feature vectors?The language I use is python

opened May 18, 2020 by zhenzi0322 21

Closed

"Couldn't advance to binary doc values for doc" exception for 7.9.2

Using the new plug-in for ES 7.9.2 I get this exception when trying to search:

{'error': {'root_cause': [{'type': 'runtime_exception', 'reason': "Couldn't advance to binary doc values for doc with id [404521]"}], 'type': 'search_phase_execution_exception', 'reason': 'all shards failed', 'phase': 'query', 'grouped': True, 'failed_shards': [{'shard': 0, 'index': 'discovery_events_v2', 'node': 'ucbOKdpaRMil3sMIVh17eA', 'reason': {'type': 'runtime_exception', 'reason': "Couldn't advance to binary doc values for doc with id [404521]"}}], 'caused_by': {'type': 'runtime_exception', 'reason': "Couldn't advance to binary doc values for doc with id [404521]", 'caused_by': {'type': 'runtime_exception', 'reason': "Couldn't advance to binary doc values for doc with id [404521]"}}}, 'status': 500}

My search (the value of the vector is omitted) is:

    es_query = {
        "_source": ['id'],
        "size": 20,
        "query": {
            "elastiknn_nearest_neighbors": {
                "field": "esknn_embedding",
                "vec": {
                    "values": [...],
                },
                "model": "lsh",
                "similarity": "angular",
                "candidates": 50
            }
        }
    }

The mapping is:

    "esknn_embedding" : {
      "type" : "elastiknn_dense_float_vector",
      "elastiknn" : {
        "model" : "lsh",
        "similarity" : "angular",
        "dims" : 512,
        "L" : 99,
        "k" : 1
      }
    },

opened Oct 25, 2020 by ejackson-eb 18

Closed

Field problem

Currently, it seems that the elastiknn plugin only supports adding an index. If I want to add an additional index image_path to save the path of the image, how can I add this one with the elastiknn plugin?

opened May 22, 2020 by zhenzi0322 11

Closed

'failed to create query' request error when using NearestNeighborsQuery.L2LSH in ElastiKnnClient

I am trying to use the ElastiKnnClient and I implemented as such while following the TestClient.test_exact_jaccard() method.

eknn = ElastiKnnClient()
n, dim = 1000, 512
index = "py-test"
field = "vec"
mapping = Mapping.DenseFloat(dims=dim)
 
eknn.es.indices.delete(index, ignore=[400, 404])
eknn.es.indices.refresh()
eknn.es.indices.create(index)
eknn.es.indices.refresh()
m = eknn.put_mapping(index, field, mapping)
 
vecs = [Vec.DenseFloat(np.random.rand(512).tolist()) for _ in range(n)]
ids = [f"vec-{i}" for i in range(n)]
(n_, errors) = eknn.index(index, field, vecs, ids, refresh=True)
assert n_ == n
assert len(errors) == 0
 
qvec = vecs[0]
query = NearestNeighborsQuery.L2LSH(field, qvec, Similarity.L2)
res = eknn.nearest_neighbors(index, query, 10, False)
hits = res['hits']['hits']

RequestError
---> 21 res = eknn.nearest_neighbors(index, query, 10, False) RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to create query: { }')

The above code works fine if NearestNeighborsQuery.Exact is used instead. Also tried using the REST API directly but got the same error. Is there something I am doing wrongly?

documentation

opened Jun 10, 2020 by shawnchen63 10

Closed

Query nested field not working

I have the following mapping for my index:

{
  "properties": {
    "id": {
      "type": "long"
    },
    "features": {
      "properties": {
        "color": {
          "type": "elastiknn_dense_float_vector",
          "elastiknn": {
            "dims": 108,
            "model": "lsh",
            "similarity": "l2",
            "bands": 99,
            "rows": 1,
            "width": 3
          }
        }
      }
    }
  }
}

I used the Bulk API to insert my data (around 3 million features)

"index": {}}
{"id": 1, "features": {"color": {"values": [0.20392156862745098, 0.21176470588235294, 0.25098039215686274, 0.26666666666666666, 0.2784313725490196, 0.2235294117647059, 0.047058823529411764, 0.0392156862745098, 0.03529411764705882, 0.023529411764705882, 0.34509803921568627, 0.2235294117647059, 0.21568627450980393, 0.2235294117647059, 0.3137254901960784, 0.00392156862745098, 0.32941176470588235, 0.25098039215686274, 0.22745098039215686, 0.26666666666666666, 0.00784313725490196, 0.011764705882352941, 0.34901960784313724, 0.2627450980392157, 0.21176470588235294, 0.21176470588235294, 0.21176470588235294, 0.21176470588235294, 0.21176470588235294, 0.21568627450980393, 0.23529411764705882, 0.21568627450980393, 0.21568627450980393, 0.21568627450980393, 0.21568627450980393, 0.17647058823529413, 0.6, 0.4745098039215686, 0.25098039215686274, 0.24705882352941178, 0.19607843137254902, 0.27450980392156865, 0.25882352941176473, 0.19215686274509805, 0.13725490196078433, 0.2901960784313726, 0.1843137254901961, 0.19215686274509805, 0.3137254901960784, 0.23529411764705882, 0.1568627450980392, 0.3176470588235294, 0.19215686274509805, 0.15294117647058825, 0.34901960784313724, 0.19607843137254902, 0.33725490196078434, 0.5098039215686274, 0.24705882352941178, 0.13333333333333333, 0.5725490196078431, 0.6, 0.5215686274509804, 0.4627450980392157, 0.5725490196078431, 0.5607843137254902, 0.047058823529411764, 0.09803921568627451, 0.10196078431372549, 0.09803921568627451, 0.09019607843137255, 0.03529411764705882, 0.09803921568627451, 0.2901960784313726, 0.26666666666666666, 0.23921568627450981, 0.23921568627450981, 0.08627450980392157, 0.22745098039215686, 0.788235294117647, 0.9176470588235294, 0.8352941176470589, 0.42745098039215684, 0.10196078431372549, 0.28627450980392155, 0.7725490196078432, 0.5450980392156862, 0.44313725490196076, 0.28627450980392155, 0.10196078431372549, 0.1568627450980392, 0.2980392156862745, 0.26666666666666666, 0.24705882352941178, 0.20784313725490197, 0.058823529411764705, 0.054901960784313725, 0.19607843137254902, 0.17254901960784313, 0.1607843137254902, 0.1843137254901961, 0.06274509803921569, 0.08235294117647059, 0.12156862745098039, 0.11372549019607843, 0.12156862745098039, 0.13333333333333333, 0.10980392156862745]}}}
{"index": {}}
{"id": 2, "features": {"color": {"values": [0.0392156862745098, 0.03529411764705882, 0.027450980392156862, 0.03137254901960784, 0.0392156862745098, 0.0392156862745098, 0.0392156862745098, 0.027450980392156862, 0.0196078431372549, 0.027450980392156862, 0.0392156862745098, 0.0392156862745098, 0.0392156862745098, 0.0196078431372549, 0.023529411764705882, 0.023529411764705882, 0.03137254901960784, 0.043137254901960784, 0.03137254901960784, 0.023529411764705882, 0.027450980392156862, 0.023529411764705882, 0.027450980392156862, 0.043137254901960784, 0.03137254901960784, 0.023529411764705882, 0.023529411764705882, 0.023529411764705882, 0.03529411764705882, 0.043137254901960784, 0.027450980392156862, 0.023529411764705882, 0.023529411764705882, 0.023529411764705882, 0.03529411764705882, 0.043137254901960784, 0.4392156862745098, 0.4392156862745098, 0.5215686274509804, 0.49019607843137253, 0.36470588235294116, 0.39215686274509803, 0.34509803921568627, 0.5019607843137255, 0.48627450980392156, 0.4980392156862745, 0.3058823529411765, 0.30980392156862746, 0.3254901960784314, 0.3137254901960784, 0.34509803921568627, 0.2784313725490196, 0.26666666666666666, 0.30196078431372547, 0.38823529411764707, 0.33725490196078434, 0.21568627450980393, 0.396078431372549, 0.3058823529411765, 0.30196078431372547, 0.5176470588235295, 0.3843137254901961, 0.26666666666666666, 0.49019607843137253, 0.44313725490196076, 0.3137254901960784, 0.3254901960784314, 0.3686274509803922, 0.34901960784313724, 0.3215686274509804, 0.3411764705882353, 0.2823529411764706, 0.7529411764705882, 0.6901960784313725, 0.4196078431372549, 0.5843137254901961, 0.8274509803921568, 0.788235294117647, 0.8509803921568627, 0.5294117647058824, 0.5411764705882353, 0.5607843137254902, 0.8941176470588236, 0.8745098039215686, 0.8588235294117647, 0.7725490196078432, 0.7843137254901961, 0.8352941176470589, 0.8627450980392157, 0.8941176470588236, 0.6980392156862745, 0.7098039215686275, 0.8980392156862745, 0.6509803921568628, 0.792156862745098, 0.8980392156862745, 0.41568627450980394, 0.5725490196078431, 0.7372549019607844, 0.47058823529411764, 0.6039215686274509, 0.7607843137254902, 0.6666666666666666, 0.5529411764705883, 0.5725490196078431, 0.5882352941176471, 0.6078431372549019, 0.48627450980392156]}}}

I tried the following query with no success:

% curl -X GET http://localhost:9200/posts/_search -H "Content-type: application/json" -d @-
{
    "query": {
        "elastiknn_nearest_neighbors": {
            "field": "features.color", 
            "vec": {
                "true_indices": [0.00784313725490196, 0.30196078431372547, 0.34901960784313724, 0.3411764705882353, 0.23137254901960785, 0.0392156862745098, 0.01568627450980392, 0.00392156862745098, 0.00392156862745098, 0.011764705882352941, 0.011764705882352941, 0.027450980392156862, 0.03137254901960784, 0.2549019607843137, 0.2549019607843137, 0.34901960784313724, 0.28627450980392155, 0.3058823529411765, 0.011764705882352941, 0.3411764705882353, 0.25098039215686274, 0.24313725490196078, 0.28627450980392155, 0.011764705882352941, 0.011764705882352941, 0.0196078431372549, 0.011764705882352941, 0.011764705882352941, 0.01568627450980392, 0.0196078431372549, 0.20392156862745098, 0.33725490196078434, 0.2823529411764706, 0.00784313725490196, 0.23921568627450981, 0.24313725490196078, 0.19215686274509805, 0.058823529411764705, 0.11764705882352941, 0.1411764705882353, 0.1843137254901961, 0.03529411764705882, 0.14901960784313725, 0.10588235294117647, 0.20392156862745098, 0.37254901960784315, 0.26666666666666666, 0.10980392156862745, 0.18823529411764706, 0.10980392156862745, 0.2784313725490196, 0.27058823529411763, 0.12549019607843137, 0.0392156862745098, 0.27058823529411763, 0.21176470588235294, 0.36470588235294116, 0.30196078431372547, 0.17647058823529413, 0.24313725490196078, 0.30196078431372547, 0.27058823529411763, 0.20392156862745098, 0.17647058823529413, 0.16470588235294117, 0.18823529411764706, 0.0784313725490196, 0.0196078431372549, 0.023529411764705882, 0.13333333333333333, 0.054901960784313725, 0.03137254901960784, 0.43137254901960786, 0.5450980392156862, 0.43529411764705883, 0.5294117647058824, 0.36470588235294116, 0.803921568627451, 0.39215686274509803, 0.403921568627451, 0.2901960784313726, 0.6627450980392157, 0.3803921568627451, 0.7568627450980392, 0.3137254901960784, 0.2823529411764706, 0.16862745098039217, 0.3607843137254902, 0.2196078431372549, 0.5019607843137255, 0.3058823529411765, 0.3137254901960784, 0.12941176470588237, 0.16862745098039217, 0.1568627450980392, 0.5803921568627451, 0.25882352941176473, 0.6705882352941176, 0.6862745098039216, 0.6705882352941176, 0.7372549019607844, 0.7450980392156863, 0.6627450980392157, 0.796078431372549, 0.9058823529411765, 0.7137254901960784, 0.8627450980392157, 0.9137254901960784],
                "total_indices": 108
            },
            "model": "lsh",
            "similarity": "l2", 
            "candidates": 50
        }
    }
}
{"error":{"root_cause":[{"type":"","reason":"Attempt to decode value on failed cursor: DownField(index),DownField(vec)"}],"type":"","reason":"Attempt to decode value on failed cursor: DownField(index),DownField(vec)"},"status":500}

It's my first time using elasticsearch, so I have no idea how to query things...

bug

opened Apr 20, 2020 by ReneHollander 9

Closed

Add a filter clause to each of the nearest neighbor queries

This was brought up as part of some discussion on the Lucene mailing list. It seems reasonable that someone would want to filter a set of documents based on other fields (e.g. product type, location, price, etc.), and then compute vector similarity on the filtered subset.

Should first check if this kind of construct already exists in Elasticsearch in a generic form. Function score queries sort-of have this construct, where you provide q "query" and a function, and it only evaluates your function on the queries that match.

enhancement

opened Jun 26, 2020 by alexklibisz 9

Open

Update to Scala 2.13

opened Jan 24, 2021 by alexklibisz 0

Open

Billion-scale benchmark

The goal here will be to use a dataset with 1B vectors and find a cluster configuration that gets 0.8 recall with 100 queries/second. I'm guesstimating these figures, but it seems like a not-crazy goal.

opened Dec 24, 2020 by alexklibisz 0

Open

Completion suggester fails when elastiknn is set to true

Using the completion suggester feature fails when elastiknn is set to true in the index setting.

Steps to replicate issue:

Create index with elastiknn set to true and completion

PUT music
{
  "settings": {
    "elastiknn": true
  }, 
  "mappings": {
    "properties": {
      "suggest": {
        "type": "completion",
        "analyzer": "standard"
      },
      "title": {
        "type": "keyword"
      }
    }
  }
}

Add a document

PUT music/_doc/1?refresh
{
  "title": "Smells like teen spirit",
  "suggest" : {
    "input": [ "Nevermind", "Nirvana" ],
    "weight" : 34
  }
}

Query suggestion

POST music/_search?pretty
{
  "suggest": {
    "song-suggest": {
      "prefix": "nir",        
      "completion": {         
          "field": "suggest"  
      }
    }
  }
}

The query fails with response:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "suggest is not a SuggestField"
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "music",
        "node": "_UooGXTrSV66WkdBnJmb8g",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "suggest is not a SuggestField"
        }
      }
    ],
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "suggest is not a SuggestField",
      "caused_by": {
        "type": "illegal_argument_exception",
        "reason": "suggest is not a SuggestField"
      }
    }
  },
  "status": 400
}

The issue goes away if elastiknn is set to false

bug

opened Dec 8, 2020 by emla2805 1

Open

Feature Request: Support Multiple Vectors per Field

I don't know how difficult this would be, since I haven't investigated the elastiknn field implementation. Most fields in Elasticsearch can accept an array of values that you can match and sort against, and we could use the same thing for the dense vector field. Here is our use case:

We have long form descriptions of objects we want to index at the sentence-vector level. Each object description therefore has multiple vectors associated with it. When I perform a knn query, I want to get back a document if any of those sentence-vectors is close.

We considered having a separate document for every sentence-vector along with the object id, but this creates a lot of challenges if we want to filter against other object properties.

Thank you for the truly excellent software!

opened Nov 11, 2020 by akoumjian 4

Open

Optimize top-k counting for approximate queries

Currently the biggest bottleneck at query time is the countHits method in MatchHashesAndScoreQuery. This counts the number of times each doc in the segment matches one of the query vector's hashes. https://github.com/alexklibisz/elastiknn/blob/master/elastiknn-lucene/src/main/java/org/apache/lucene/search/MatchHashesAndScoreQuery.java#L54-L73

AFAIK, Lucene is generally optimized for a small number of terms (e.g. the words in a search query). Elastiknn, on the other hand, can require retrieving doc IDs for tens or hundreds of terms (the hashes of a query vector).

It seems the main thing worth exploring is using a different PostingsFormat, or potentially implementing a custom one. Maybe there's a way to optimize the storage/access pattern for checking a larger number of terms? Maybe there's a way to simply wrap an existing postings format and offer some helpers that cache the expensive method calls?

Some specific numbers:

When running the MatchHashesAndScoreQueryPerformanceSuite, with 100k indexed vectors and 5k search vectors, the VisualVM sampler reports spending ~92% of its time in the countHits method:

When running the ContinuousBenchmark with the SIFT dataset (1M indexed vectors, 1k search vectors), the VisualVM sampler reports spending ~36% of the search thread time in countHits:

help wanted performance priority-high

opened Sep 20, 2020 by alexklibisz 32

Open

More efficient counter for query hits

The custom MatchHashesAndScoreQuery currently uses a counter that requires allocating an array of shorts with one entry for every document in the segment to track the number of matches for each doc, and then iterating over that array twice to get the top k docs. This is actually substantially faster than using any sort of hashmap I've found, including primitive hash maps. I've tried hppc, hppcrt, and fastlib, and all of them are least 2x as slow (e.g. a segment with 1.1M docs gets 40 q/s with arrays, 20 q/s with hashmaps). I figure this kind of array setup won't scale forever, but I don't want to change it until there's some comparably fast alternative.

performance priority-low

opened Sep 16, 2020 by alexklibisz 1

Information

Category: Java / Search
Watchers: 5
Star: 129
Fork: 13
Last update: Feb 21, 2021

Resource links

https://github.com/alexklibisz/elastiknn

Share this repo

Related Repos

7.2k

The Missing Web UI for Elasticsearch: Import, browse and edit data with rich filters and q...

dejavu: The missing Web UI for Elasticsearch Dejavu: Intro Features a. Easily Connect to Indices b. Visual Filters c. Modern UI Elements d. Import JSON or CSV Data e. Build search UIs Comparison Roadmap Build Local

101

Java Client for RediSearch

JRediSearch A Java Client Library for RediSearch Overview This project contains a Java library abstracting the API of the RediSearch Redis module, that implements a powerful in-memory search engine inside

7.2k

The Missing Web UI for Elasticsearch: Import, browse and edit data with rich filters and q...

Elasticsearch Java Rest Client.

JEST Jest is a Java HTTP Rest client for ElasticSearch. ElasticSearch is an Open Source (Apache 2), Distributed, RESTful, Search Engine built on top of Apache Lucene. ElasticSearch already has a Java API which is also used

7.2k

The Missing Web UI for Elasticsearch: Import, browse and edit data with rich filters and q...

100

Search and Search Aggregation module for the YaCy Grid

YaCy Grid Component: Search The YaCy Grid is the second-generation implementation of YaCy, a peer-to-peer search engine. A YaCy Grid installation consists of a set of micro-services which communicate with each other using the MCP

Simple full text indexing and searching library for Java

indexer4j Simple full text indexing and searching library for Java Install Gradle repositories { jcenter() } dependencies { compile 'com.haeungun.indexer4j:indexer4j:<:the-latest-version&

564

A Lucene toolkit for replicable information retrieval research

Anserini Anserini is an open-source information retrieval toolkit built on Lucene that aims to bridge the gap between academic information retrieval research and the practice of building real-world search applications. T