Home / Java / Search

Elasticsearch plugin for nearest neighbor search. Store vectors and run similarity search using exact and approximate algorithms.

Elasticsearch plugin for nearest neighbor search. Store vectors and run similarity search using exact and approximate algorithms.
Information
Category: Java / Search
Watchers: 5
Star: 263
Fork: 39
Last update: Jun 26, 2022
Resource links
https://github.com/alexklibisz/elastiknn

Elastiknn

Elasticsearch Plugin for Nearest Neighbor Search

Documentation

Documentation at elastiknn.klibisz.com

Community

Join the Gitter channel at gitter.im/elastiknn/community.

If you're using Elastiknn in your research or work, I'd love to hear feedback. Post an issue, post in the Gitter channel, or contact me directly (contact info in my Github bio).

Builds and Releases

Item Status
Github CI Build Github CI Status
Github Release Build Github Release Status
Plugin Release Plugin Release Status
Plugin Snapshot Plugin Snapshot Status
Python Client, Release Python Client Release Status
Scala 2.12 Client, Release Scala Client Release Status
Scala 2.12 Client, Snapshot Scala Client Snapshot Status

License

Elastiknn is licensed under GPLv3. See License.txt. If you have any questions, contact me.

20 Open More issues
Closed

How to use image search

When I used image search, I returned all documents from elasticsearch.How does the elastiknn plugin generate image feature vectors?The language I use is python

opened May 18, 2020 by zhenzi0322 21
Closed

Elastiknn plugin size (possibly change JSON library to reduce dependency sizes)

Starting from version 7.12.0.0, the plugin size went beyond 20MB (the previous version was 19.1MB).

Most Elastic cloud subscription levels have a hard limit of 20MB for custom plugins (see here).

It would be great if the plugin could stay within this limit.

I had a quick look at the project and I think some heavy dependencies can be removed with little changes. An example would be guava, currently used for base64 encoding and simple caching logic. Replacing this with Java 8 code would remove ~3MB.

What do you think? If this sounds reasonable I can work on this.

opened Oct 27, 2021 by fabriziofortino 19
Closed

"Couldn't advance to binary doc values for doc" exception for 7.9.2

Using the new plug-in for ES 7.9.2 I get this exception when trying to search:

{'error': {'root_cause': [{'type': 'runtime_exception', 'reason': "Couldn't advance to binary doc values for doc with id [404521]"}], 'type': 'search_phase_execution_exception', 'reason': 'all shards failed', 'phase': 'query', 'grouped': True, 'failed_shards': [{'shard': 0, 'index': 'discovery_events_v2', 'node': 'ucbOKdpaRMil3sMIVh17eA', 'reason': {'type': 'runtime_exception', 'reason': "Couldn't advance to binary doc values for doc with id [404521]"}}], 'caused_by': {'type': 'runtime_exception', 'reason': "Couldn't advance to binary doc values for doc with id [404521]", 'caused_by': {'type': 'runtime_exception', 'reason': "Couldn't advance to binary doc values for doc with id [404521]"}}}, 'status': 500}

My search (the value of the vector is omitted) is:

    es_query = {
        "_source": ['id'],
        "size": 20,
        "query": {
            "elastiknn_nearest_neighbors": {
                "field": "esknn_embedding",
                "vec": {
                    "values": [...],
                },
                "model": "lsh",
                "similarity": "angular",
                "candidates": 50
            }
        }
    }

The mapping is:

    "esknn_embedding" : {
      "type" : "elastiknn_dense_float_vector",
      "elastiknn" : {
        "model" : "lsh",
        "similarity" : "angular",
        "dims" : 512,
        "L" : 99,
        "k" : 1
      }
    },
opened Oct 25, 2020 by ejackson-eb 18
Closed

Scala Client with multiple nodes

How do I create a distributed es elastic4sknn client

Q&A 
opened Mar 12, 2021 by shizeying 11
Closed

Field problem

Currently, it seems that the elastiknn plugin only supports adding an index. If I want to add an additional index image_path to save the path of the image, how can I add this one with the elastiknn plugin?

opened May 22, 2020 by zhenzi0322 11
Closed

Upgrade scipy dependency to 1.7.0

Hi there, wonderful project and I became addicted to it. I use it very well with sklearn pandas to build the transformed dataframes which I then convert into the vector fields. However sklearn pandas requires scipy-1.7.0 but elasticknn depends on scipy==1.4.1. It seems to be working just find but would be nice to bump up the dependency if possible. Cheers!

opened Jul 2, 2021 by priamai 10
Open

Future of Elastiknn

My current plan is to begin winding down my contributions and support of Elastiknn. I envision the project continuing in "maintenance mode", and I want to give any users a specific heads-up about this. In many ways, this is how I've approached the project for the past year or so. I'm just spelling it out so that expectations are clear.

The project faces several headwinds, not all bad.

Good headwinds:

  1. Elasticsearch and Opensearch are both investing in ANN implementations. After over a year of development, Lucene and Elasticsearch are finally starting to expose an API for ANN search. There's also a growing ecosystem of ANN search solutions, both open source and proprietary. It's great to see the ecosystem growing.
  2. I have other technical interests I'd like to pursue. Elastiknn has been a great platform for learning and experimenting, but I actually haven't implemented any sort of ANN search professionally since 2017, back when the idea for Elastiknn was born. I'm glad I followed through on implementing this, but, all things considered, my time at this point is better spent on other problems.

Other headwinds:

  1. The project has seen almost no outside contribution. There have been many issues and emails in which I respond, "I don't have time to work on this right this moment, but there's a developer-guide.md and I'm happy to review PRs." I can count on one hand the number of times someone has followed up on this. To be very clear, I have had some help doing maintenance, like upgrading ES versions, for which I'm very appreciative.
  2. It's unclear if anyone is even using this in any consequential way. I see some downloads, but only one person has submitted to the list of users in the readme. For better or worse, I get a lot of satisfaction from knowing my effort translates to solved problems. So the lack of feedback makes it hard to find motivation for this effort compared to some other efforts.

There are a few final tasks I'd like to complete to satisfy my own curiosities:

  1. Upgrade to Elasticsearch 8.x (#348). I'll continue reviewing upgrade PRs if other folks make them, but this will be the last major upgrade which I personally do.
  2. Complete and merge a benchmark implementation based on some of the big-ann-benchmarks datasets (#278). Ideally this would also include an apples/apples comparison to Lucene's HNSW implementation. I'm curious how the numbers play out.
  3. Get rid of the Unsafe vector serialization (#263).

Concretely, once the above are done, here's how I see the future of Elastiknn playing out in "maintenance mode":

  1. I'll continue to review and merge version bump PRs. Based on Elastic's historical cadence, I think getting onto 8.x will lead to relative steady-state for about a year or so.
  2. If someone reports a bug, It'll come through my email and I might comment on how it might be fixed. If it's particularly interesting, and scratches an intellectual itch, I might look into it myself.
  3. I won't pursue big feature additions like #197, #279, #298, #323. If someone submits a PR, I'll give it a look. I'll keep a high-standard for the project and won't merge low-quality PRs. It would take some impressive ambition to solve these kinds of problems well.
opened Mar 13, 2022 by alexklibisz 5
Open

Upgrade to Elasticsearch 8.x

This might also be a good opportunity to solve #254.

opened Mar 13, 2022 by alexklibisz 0
Open

Elasticsearch 8.0.0

  • Upgrade Elasticsearch from 7.17.3 to Elasticsearch 8.8.0
  • Upgrade Lucene from 8.11.1 to Lucene 9.0.0
  • Upgrade elastic4s from 7.17.2 to 8.0.0
  • Make the index.elastiknn setting a no-op and stop using a custom codec. As of 9.0.0, Lucene issue 9843 is resolved, which means the default doc values codec is optimized for best speed. This means Elastiknn can stop overriding the codec. We keep the old custom codecs around for backwards-compatibility. This also resolves #254.

Outstanding tasks:

  • Update dependencies to make the JavaClientSuite tests work again. Elastic deprecated one of the clients used in that suite.
  • Finish running benchmarks and make sure we have parity with current master branch.

Benchmarks on current master branch: | dataset | similarity | algorithm | mapping | query | k | shards | replicas | parallelQueries | esNodes | esCoresPerNode | esMemoryGb | warmupQueries | minWarmupRounds | maxWarmupRounds | recall | queriesPerSecond | durationMillis | | ---------------- | ---------- | --------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | --- | ------ | -------- | --------------- | ------- | -------------- | ---------- | ------------- | --------------- | --------------- | ---------- | ---------------- | -------------- | | AnnbFashionMnist | "l2" | Exact | {"elastiknn":{"dims":784},"type":"elastiknn_dense_float_vector"} | {"field":"vec","model":"exact","similarity":"l2","vec":{}} | 100 | 1 | 0 | 1 | 1 | 1 | 4 | 200 | 10 | 10 | 0.9982029 | 13.089005 | 764083 | | AnnbFashionMnist | "l2" | LSH | {"elastiknn":{"L":75,"dims":784,"k":4,"model":"lsh","similarity":"l2","w":7},"type":"elastiknn_dense_float_vector"} | {"candidates":2000,"field":"vec","model":"lsh","probes":3,"similarity":"l2","vec":{}} | 100 | 1 | 0 | 1 | 1 | 1 | 4 | 200 | 10 | 10 | 0.856039 | 125.0 | 80766 | | AnnbFashionMnist | "l2" | LSH | {"elastiknn":{"L":75,"dims":784,"k":4,"model":"lsh","similarity":"l2","w":7},"type":"elastiknn_dense_float_vector"} | {"candidates":1000,"field":"vec","model":"lsh","probes":0,"similarity":"l2","vec":{}} | 100 | 1 | 0 | 1 | 1 | 1 | 4 | 200 | 10 | 10 | 0.53446233 | 96.15385 | 104732 | | AnnbSift | "l2" | LSH | {"elastiknn":{"L":100,"dims":128,"k":4,"model":"lsh","similarity":"l2","w":2},"type":"elastiknn_dense_float_vector"} | {"candidates":5000,"field":"vec","model":"lsh","probes":0,"similarity":"l2","vec":{}} | 100 | 1 | 0 | 1 | 1 | 1 | 4 | 200 | 10 | 10 | 0.8232296 | 41.152264 | 243890 |

Benchmarks on es8.0.0 branch: | dataset | similarity | algorithm | mapping | query | k | shards | replicas | parallelQueries | esNodes | esCoresPerNode | esMemoryGb | warmupQueries | minWarmupRounds | maxWarmupRounds | recall | queriesPerSecond | durationMillis | | ---------------- | ---------- | --------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | --- | ------ | -------- | --------------- | ------- | -------------- | ---------- | ------------- | --------------- | --------------- | ---------- | ---------------- | -------------- | | AnnbFashionMnist | "l2" | Exact | {"elastiknn":{"dims":784},"type":"elastiknn_dense_float_vector"} | {"field":"vec","model":"exact","similarity":"l2","vec":{}} | 100 | 1 | 0 | 1 | 1 | 1 | 4 | 200 | 10 | 10 | 0.9982029 | 13.157895 | 760231 | | AnnbFashionMnist | "l2" | LSH | {"elastiknn":{"L":75,"dims":784,"k":4,"model":"lsh","similarity":"l2","w":7},"type":"elastiknn_dense_float_vector"} | {"candidates":2000,"field":"vec","model":"lsh","probes":3,"similarity":"l2","vec":{}} | 100 | 1 | 0 | 1 | 1 | 1 | 4 | 200 | 10 | 10 | 0.856039 | 98.039215 | 102295 | | AnnbFashionMnist | "l2" | LSH | {"elastiknn":{"L":75,"dims":784,"k":4,"model":"lsh","similarity":"l2","w":7},"type":"elastiknn_dense_float_vector"} | {"candidates":1000,"field":"vec","model":"lsh","probes":0,"similarity":"l2","vec":{}} | 100 | 1 | 0 | 1 | 1 | 1 | 4 | 200 | 10 | 10 | 0.53446233 | 119.04762 | 84204 | | AnnbSift | "l2" | LSH | {"elastiknn":{"L":100,"dims":128,"k":4,"model":"lsh","similarity":"l2","w":2},"type":"elastiknn_dense_float_vector"} | {"candidates":5000,"field":"vec","model":"lsh","probes":0,"similarity":"l2","vec":{}} | 100 | 1 | 0 | 1 | 1 | 1 | 4 | 200 | 10 | 10 | 0.8232381 | 37.878788 | 264759 |

opened Mar 13, 2022 by alexklibisz 0
Open

Support for elastiknn_dense_float_vector in script_score

I'm using elastiknn_dense_float_vector as a vector for a field to achieve ANN search. But I want to use the elastiknn_dense_float_vector in script_score to modify the score which is not working as I'm getting "Fielddata is not supported on field [knn_vectors] of type [elastiknn_dense_float_vector]" as an error.

Any possible way to convert elastiknn_dense_float_vector to dense_vector. If not how can we use elastiknn_dense_float_vector in script_score query.

This is the query I'm using right now and Im using ElaskiKNN 7.9.2 version

{
    "query": {
        "function_score": {
            "query": {
                "bool": {
                    "should": [
                        {
                            "multi_match": {
                                "query": "statement",
                                "fields": [ ]
                            }
                        },
                        {
                            "elastiknn_nearest_neighbors": {
                                "field": "page_title_vectors_knn",
                                "vec": {
                                    "values": []
                                },
                                "model": "lsh",
                                "similarity": "l2",
                                "candidates": 10
                            }
                        }
                    ]
                }
            },
            "functions": [
                {
                    "script_score": {
                        "script": {
                            "source": "double cosVal = Math.acos(cosineSimilarity(params.query_vector, doc["page_title_vectors_knn"])); return 1-(cosVal/3.14);",
                            "params": {
                                "query_vector": []
                            }
                        }
                    }
                }
            ],
            "boost_mode": "multiply",
            "score_mode": "multiply",
            "boost": 1.0
        }
    }
}
enhancement 
opened Nov 12, 2021 by venkatana-kore 3
Open

Can't create vector fields by es index templates

Describe the bug When I create an es index template for creating vector fields and creating the docs, it returns failed to parse.

Expected behavior The docs should be created successfully with the vector fields.

Environment (please complete the following information):

  • Elastiknn version: 7.14.1.2

To Reproduce Steps to reproduce the behavior:

  1. Create an index template
PUT _index_template/test_template
{
  "index_patterns": [
    "test"
  ],
  "template": {
    "settings": {
      "number_of_shards": 9,
      "number_of_replicas": 1,
      "elastiknn": true
    },
    "mappings": {
      "dynamic_templates": [
        {
          "vector_as_cosine_lsh": {
            "match_mapping_type": "double",
            "path_match": "vectors.*",
            "mapping": {
              "type": "elastiknn_dense_float_vector",
              "elastiknn": {
                "dims": 2,
                "model": "lsh",
                "similarity": "cosine",
                "L": 99,
                "k": 1
              }
            }
          }
        }
      ]
    }
  }
}
  1. Create a doc
POST test/_update/123
{
  "doc": {
    "vectors.bbb": [
      0.1,
      0.1
    ]
  },
  "upsert": {}
}
  1. First time see
{
  "_index" : "test",
  "_type" : "_doc",
  "_id" : "123",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

but no data in doc 4. If create again, see

{
  "error" : {
    "root_cause" : [
      {
        "type" : "",
        "reason" : ": C[A]"
      }
    ],
    "type" : "mapper_parsing_exception",
    "reason" : "failed to parse",
    "caused_by" : {
      "type" : "",
      "reason" : ": C[A]"
    }
  },
  "status" : 400
}
bug 
opened Nov 8, 2021 by livelxw 2
Open

Exception types used by Elastiknn do not always map to the appropriate HTTP status codes

Describe the bug

Elastiknn exceptions do not extend ElasticsearchException, so they seem to always be turned into a 500, when a 4xx would often be more appropriate.

Expected behavior

(There are probably some other places where this is the case)

Environment (please complete the following information):

  • Elastiknn version: All
  • OS: All

To Reproduce

  • Attempt to run a query that is incompatible with a given mapping.
  • Attempt to index a vector of the wrong dimensionality.

Additional context

  • I stumbled onto this issue while fixing #310 .
  • Elasticsearch has an ElasticsearchException, which seems to be the intended way to specify the severity of an exception and thus its eventual status code.
bug good first issue 
opened Sep 26, 2021 by alexklibisz 0

Contents

    lmcinnes A Python nearest neighbor descent for approximate nearest neighbors
    LinkedInAttic A scalable nearest neighbor search library in Apache Spark
    netrasys Fast Approximate Nearest Neighbor (ANN) searches with a PostgreSQL database.
    microsoft A distributed approximate nearest neighborhood search (ANN) library which provides a high quality vector index build, search and distributed online serving toolkits for large scale vector search scenario.
    flann-lib Fast Library for Approximate Nearest Neighbors
    jobergum Performance evaluation of nearest neighbor search using Vespa, Elasticsearch and Open Distro for Elasticsearch K-NN
    ZJULearning fast library for ANN search and KNN graph construction
    granne Graph-based Approximate Nearest Neighbor Search
    hora-search 🚀Hora is an approximate nearest neighbor search algorithm library written in Rust 🦀 .
    jxhe Pytorch implementation of paper "Efficient Nearest Neighbor Language Models" (EMNLP 2021)
    kakao TOROS N2 - lightweight approximate Nearest Neighbor library which runs faster even with large datasets
    facebookresearch Faiss is a library for efficient similarity search and clustering of dense vectors
    cgtuebingen GGNN: Graph-based GPU Nearest Neighbor Search
    felipexw Just a simple implementation of K-Nearest Neighbour algorithm.
    AI-secure Official Repo for "Efficient task-specific data valuation for nearest neighbor algorithms"
    twistedcubic Neural LSH [ICLR 2020] - Using supervised learning to produce better space partitions for fast nearest neighbor search.
    vearch A distributed system for efficient similarity search of embedding vectors
    ankane Nearest neighbor search for Rails and Postgres
    babylonhealth Multilingual word vectors in 78 languages
    ekzhu All-pair set similarity search on millions of sets in Python and on a laptop
    jelmerk Java library for approximate nearest neighbors search using Hierarchical Navigable Small World graphs
    Share this repo

    Related Repos


    Search
    17
    Forked repo of seunjeon for Elasticsearch 7 or newer.
    likejazz seunjeon for Elasticsearch 7 or newer analysis-seunjeon(은전한닢) is the best Korean analyzer, especially useful for Elasticsearch. Unfortunately original
     
    Search
    3
    Mini Search Engine with Python - assume there are 10 documents
    hyeseonko mini-search-engine Mini Search Engine with Python - assume there are 10 documents 1. Super Simple Search Engine Assume that there are 10 documents and
     
    Search
    144
    easy use for elastich search
    xpc1024 Born To Simplify Development What is Easy-Es? Easy-Es is a powerfully enhanced toolkit of RestHighLevelClient for simplify development. This toolkit p
     
    Search
    3
    Criando um jogo Tower Defense com minha Engine
    zallih ♜ TowerDefense •Completo• Linguagem utilizada: JAVA Tower Defense Tower Defense é um subgênero de estratégia. Geralmente, o objetivo desses jogos é de
     
    Search
    3
    sql convert to elasticsearch dsl
    eolinker 简介 convert SQL to Elasticsearch DSL in java. 将SQL转成Elasticsearch的DSL的工具，语言类型：Java。 内含javabean转sql的工具类（DSLSqlHelper） 实现DSLSelectHandler接口可将SQL转成其他类SQL查
     
    Search
    1.1k
    Elasticsearch File System Crawler (FS Crawler)
    dadoonet Elasticsearch File System Crawler (FS Crawler)
     
    Search
    3
    Search Engine - Based on Inverted Index Algorithms
    AshkanShakiba Joojle Search Engine, Based on Inverted Index Algorithms How to use Select Add Documents option and enter documents path Then use Process Documents op
     
    Search
    14
    Enterprise Backend as a service for data retrieval
    Y77CH EBQuery Table of Contents Introduction - Enterprise Backend as a Service Requirements Getting started Using EBQuery Features Introduction - Enterprise
     
    Search
    34
    all prs are automatically merged - star the mod to become a collaborator
    sample-text-here The Community Mod A Mod by The Community for the Community! How to use Star the mod for free collab. If starring it doesn't work try this. How to add
     
    Search
    30
    At-a-glance overview diagrams of Apache Lucene's default PostingsFormat (inverted index bi...
    mocobeta At-a-glance overview diagrams of Apache Lucene's default PostingsFormat (inverted index binary format).
     
    Search
    5.2k
    Open source distributed and RESTful search engine.
    opensearch-project OpenSearch is an open source search and analytics engine derived from Elasticsearch 7.10.2, and is currently in an alpha state. While we're still in the process of getting set up, we are excited to build the best distributed search engine on the planet with you.
     
    Search
    116
    Apache Lucene open-source search software
    apache Apache Lucene Apache Lucene is a high-performance, full featured text search engine library written in Java. Online Documentation This README file onl
     
    Search
    672
    HanLP Analyzer for Elasticsearch
    KennFalcon HanLP Analyzer for Elasticsearch
     
    Search
    48
    Doks is a CLI tool that aggregates documents coming from multiple sources (github, google ...
    wlezzar Doks is a CLI tool that aggregates documents coming from multiple sources (github, google drive, etc.) into a search engine (Lucene is used by default for minimum local setup but elasticsearch can be used instead).
     
    Search
    263
    Elasticsearch plugin for nearest neighbor search. Store vectors and run similarity search ...
    alexklibisz Elasticsearch plugin for nearest neighbor search. Store vectors and run similarity search using exact and approximate algorithms.