Home / JavaScript / Storage

javascript embeddable / in-memory database

LokiJS Overview LokiJS is a document oriented database written in javascript, published under MIT License. Its purpose is to store javascript objects as documents in a nosql fashion and retrieve them with a simila
Information
Category: JavaScript / Storage
Watchers: 163
Star: 6.4k
Fork: 490
Last update: Mar 5, 2023
Resource links
http://lokijs.org
https://github.com/techfort/LokiJS

LokiJS

Join the chat at https://gitter.im/techfort/LokiJS alt CI-badge npm version alt packagequality

Overview

LokiJS is a document oriented database written in javascript, published under MIT License. Its purpose is to store javascript objects as documents in a nosql fashion and retrieve them with a similar mechanism. Runs in node (including cordova/phonegap and node-webkit), nativescript and the browser. LokiJS is ideal for the following scenarios:

  1. client-side in-memory db is ideal (e.g., a session store)
  2. performance critical applications
  3. cordova/phonegap mobile apps where you can leverage the power of javascript and avoid interacting with native databases
  4. data sets loaded into a browser page and synchronised at the end of the work session
  5. node-webkit desktop apps
  6. nativescript mobile apps that mix the power and ubiquity of javascript with native performance and ui

LokiJS supports indexing and views and achieves high-performance through maintaining unique and binary indexes (indices) for data.

Demo

The following demos are available:

  • Sandbox / Playground
  • a node-webkit small demo in the folder demos/desktop_app. You can launch it by running /path/to/nw demos/desktop_app/

Wiki

Example usage can be found on the wiki

Main Features

  1. Fast performance NoSQL in-memory database, collections with unique index (1.1M ops/s) and binary-index (500k ops/s)
  2. Runs in multiple environments (browser, node, nativescript)
  3. Dynamic Views for fast access of data subsets
  4. Built-in persistence adapters, and the ability to support user-defined ones
  5. Changes API
  6. Joins

Current state

LokiJS is at version 1.3 [Eostre].

As LokiJS is written in JavaScript it can be run on any environment supporting JavaScript such as browsers, node.js/node-webkit, nativescript mobile framework and hybrid mobile apps (such as phonegap/cordova).

Made by @techfort, with the precious help of Dave Easterday.

Leave a tip or give us a star if you find LokiJS useful!

Installation

For browser environments you simply need the lokijs.js file contained in src/

You can use bower to install lokijs with bower install lokijs

For node and nativescript environments you can install through npm install lokijs.

Roadmap

  • exactIndex
  • key-value datastore
  • MRU cache
  • MongoDB API compatibility
  • server standalone (tcp and http servers and clients)
  • replication and horizontal scaling

Contact

For help / enquiries contact [email protected]

Commercial Support

For commercial support contact [email protected]

License

Copyright (c) 2015 TechFort [email protected]

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

22 Open More issues
Closed

Joins

At the moment Loki supports in-memory joins e.g.

var author = authors.insert({name: 'Stephen King'});
var book = books.insert({title: 'Salems Lot', author: author})

But this will not persist correctly as the author will be serialized as an embedded object, not the original author object. There should be a more solid mechanism to operate joins.

opened Mar 19, 2015 by techfort 67
Closed

Bulk Inserts and Collection Constraints

I don't think I'm using indexes correclty.

When merging 10,000+ records from a remote database into a local database, I have to use the id (not the $loki id) to make sure the document is not already there. (if its there I want to merge instead of update). To do this effeciently, I'm trying to use Loki indices:

let loki = new Loki('testdb');
myCollection = loki.addCollection('myCollection', {indices:['id']})

Then on each insert I do

localRecord = myCollection.findOne({id: remoteRecord.id})
if( localRecord ) {
  // Merge code
} else {
  myCollection.insert( remoteRecord );  
}

When I disable the localRecord check and just insert, things are smoking fast! Only slowed by data fetching/paging.

  • http://screencast.com/t/H3wsetchhQ

When I enable the findOne check, things slow down a bit:

  • http://screencast.com/t/gBJQ49yBJwW

When I hack an index to the side that is a big hash of ids mapped to $loki I get really fast again.

  • http://screencast.com/t/V5R0g7hPt

I think there must be something I don't understand about indexes.

Please advise :)

enhancement 
opened Mar 24, 2015 by jrhicks 43
Closed

Documentation for HTTP wrapper

I couldn't see the documentation for the HTTP wrapper although it is mentioned in the features list. Would you mind pointing me in the right direction please?

wontfix 
opened Nov 8, 2015 by dukedougal 38
Closed

db save circular structure issue

Currently I have a problem in the area of loading/saving database. After object update and try to save db the following error occurs:

stream.js:94
      throw er; // Unhandled stream error in pipe.
            ^
TypeError: Converting circular structure to JSON
    at Object.stringify (native)
    at LokiEventEmitter.Loki.serialize (/home/teef/leonardo/WebApplications/Leonardo-CommHub/node_modules/lokijs/src/lokijs.js:385:19)
    at LokiEventEmitter.Loki.saveDatabase (/home/teef/leonardo/WebApplications/Leonardo-CommHub/node_modules/lokijs/src/lokijs.js:744:47)

I am using one collection as well as one dynamic view.

Seems like there are two dynamic views which should not happen.

DynamicViews: 
   [ { collection: [Circular],
       name: 'aciveDevices',
       persistent: false,
       resultset: [Object],
       resultdata: [],
       resultsdirty: true,
       cachedresultset: null,
       filterPipeline: [],
       sortFunction: null,
       sortCriteria: null,
       sortDirty: false,
       events: [Object] },
     { collection: [Circular],
       name: 'aciveDevices',
       persistent: false,
       resultset: [Object],
       resultdata: [],
       resultsdirty: false,
       cachedresultset: null,
       filterPipeline: [Object],
       sortFunction: null,
       sortCriteria: null,
       sortDirty: false,
       events: [Object] } ],

Database: https://gist.github.com/teef/c3f8f2139a1e13188e09

opened Jan 13, 2015 by teef 35
Closed

Querying empty collection with binary index throws error

If a collection has a binary index set, and the collection is empty, querying it will result in an error. The error happens in the calculateRange function on the following lines:

var minVal = rcd[index[min]][prop]; var maxVal = rcd[index[max]][prop];

If the collection is empty, then rcd[index[min/max]] will be undefined. We may want to set default min and max values here, but also, maybe shortcut the find() function to return an empty result set if the data is empty to save some work.

opened Dec 13, 2014 by assafweinberg 32
Closed

lokiFSAdaptor

Not an issue as such so sorry but I was wondering if there was an example anywhere of using loki with window.requestFileSystem/window.webkitRequestFileSystem...? My aim is for my DB to persists in a Cordova app by saving it to the FileSystem... I'm developing on Chrome so I could save it to PERSISTENT storage on the browser as well. I've looked all over and it seems as though I'll need to write my own adapter but I was wondering if anyone had already done so so I could cast my eye over their solution?

opened May 12, 2015 by annoyingmouse 31
Open

Error: this collection cannot be loaded lazily

Hey all, running in to a problem, would love some guidance.

I'm using the incremental indexeddb adapter created by @radex . I've had success with Vanilla Loki - as well as with the normal IndexedDB adapter, but really like the look of the Incremental Adapter.

I have about 20 collections, of varying sizes. All of them have the following settings:

  1. a proto define to inflate on load
  2. lazyLoad: true
  3. clone: true
  4. clone_method: shallow-assign
  5. The collection name is in the adapter options list for "lazyCollections".

I've tried this setup with all of the lazy settings turned off - both in individual collections and the adapter definition.

I'm at a bit of a loss on how to proceed. I've combed through all of the code that leads to that line, but can't figure out how to get my data loaded from idb for some of my collections. Notably, not all collections have this issue.

Thanks!

opened Jan 6, 2023 by tomkennedy22 2
Open

The unit test of `test` is not correctly implemented

Do you mean .toEqual(6.2)?

https://github.com/techfort/LokiJS/blob/07e36d156764a4d3d9172621c7b4dd3f723aeb8e/spec/generic/test.spec.js#L1024-L1030

opened Dec 7, 2022 by Losses 0
Open

The unit test of `sotringIndexing` is not correctly implemented

The variable len is defined here: https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/sortingIndexing.spec.js#L307

But the value is not replaced here: https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/sortingIndexing.spec.js#L316-L320

Here: https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/sortingIndexing.spec.js#L323-L327

The code is testing the same result for three times, which is incorrect.

opened Dec 5, 2022 by Losses 0
Open

`compoundEval` is not correctly implemented

Prosperities value is passed to compoundEval here, the type signature of the value is ([keyof Document, isDesc] | keyof Document)[].

https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L3364-L3369

But compoundEval only support [keyof Document, isDesc][]:

https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L385

https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L394

But do not support (keyof Document)[].

opened Dec 5, 2022 by Losses 0
Open

Unit test of `persistence` is not correctly implemented

https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/persistence.spec.js#L118-L144

Since the type of idx is number, so all methods wound not be tested.

opened Dec 4, 2022 by Losses 0
Open

`simpleSort` with `useJavascriptSorting` don't support `dotNotation`

Here's the implementation:

https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/src/lokijs.js#L3295-L3301

But the unit test is trying to use this feature here:

https://github.com/techfort/LokiJS/blob/25b9a33d57509717d43b4da06d92064f2b7a6c95/spec/generic/ops.spec.js#L251-L253

opened Dec 3, 2022 by Losses 0

Contents

    Restream Embeddable, in-memory, document-oriented database with a high-level Query builder interface.
    DynamoRIO Memory Debugger for Windows, Linux, Mac, and Android
    Akaion A Windows virtual memory editing library with support for pattern scanning.
    emluque Embeddable in memory key/value store for strings in golang
    wolkykim Embeddable Event-based Asynchronous Message/HTTP Server library for C/C++
    milostosic MTuner is a C/C++ memory profiler and memory leak finder for Windows, PlayStation 4 and 3, Android and other platforms
    deroproject Graviton Database: ZFS for key-value stores.
    canonical Embeddable, replicated and fault tolerant SQL engine.
    ccxvii An embeddable Javascript interpreter in C.
    FriendsOfREDAXO YCOM Impersonate. Login as selected YCOM user 🧙‍♂️in frontend.
    hashicorp Golang in-memory database built on immutable radix trees
    hse-project Heterogeneous-memory storage engine (HSE)
    small-tech A transparent, in-memory, streaming write-on-update JavaScript database for Small Web applications that persists to a JavaScript transaction log.
    svaarala Duktape - embeddable Javascript engine with a focus on portability and compact footprint
    lithdew Go bindings to QuickJS: a fast, small, and embeddable ES2020 JavaScript interpreter.
    tidwall BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support
    AqibMukhtar A lightweight JavaScript package for generating and validating OTP (One Time Password) maintaining high performance using in-memory database.
    pbnjay A go function to report total system memory
    ufrisk The Memory Process File System
    koute memory-profiler — A memory profiler for Linux
    OrenLeung AWS Tags As A Database (AWS TaaDb)🚀 is a Python 🐍 library using AWS Tags as a Key-Value database. This database is completely free* 💸
    Share this repo

    Related Repos


    Storage
    7.3k
    A Minimalistic Wrapper for IndexedDB
    dexie Dexie.js Dexie.js is a wrapper library for indexedDB - the standard database in the browser. https://dexie.org Why? Dexie solves three main issues wit
     
    Storage
    119
    An in-memory SQL database written in Javascript to demonstrate how SQL works.
    weinberg An in-memory SQL database written in Javascript to demonstrate how SQL works.
     
    Storage
    583
    A DynamoDB library to ease the use of modeling complex hierarchical relationships and impl...
    tywalch A DynamoDB library to ease the use of modeling complex hierarchical relationships and implementing a Single Table Design while keeping your query code readable.
     
    Storage
    81
    JS / CSS / files loader + key/value storage
    nodeca bag.js - JS / CSS loader + KV storage bag.js is loader for .js / .css and other files, that uses IndexedDB/ WebSQL / localStorage for caching. Conside
     
    Storage
    232
    A small, fast, in-browser database engine written in JavaScript.
    StanfordHCI Datavore Datavore is a small in-browser database engine written in JavaScript. Datavore enables you to perform fast aggregation queries within web-bas
     
    Storage
    55
    Datasette Desktop - A macOS desktop application that wraps Datasette.
    simonw Datasette Desktop - A macOS desktop application that wraps Datasette.
     
    Storage
    28
    A library to use Web Storage API with Observables
    ng-web-apis A library to use Web Storage API with Observables
     
    Storage
    29
    Neovim (nvim) Postgres SQL query execution
    mzarnitsa Neovim/nvim plugin to execute PostgreSQL queries. Written in Lua This is a simple plugin to execute SQL query from inside nvim. The plugin executes qu
     
    Storage
    26
    A tiny Javsacript library for sideloading static assets on pages and caching them in the b...
    knadh A tiny Javsacript library for sideloading static assets on pages and caching them in the browser's IndexedDB for longer-term storage.
     
    Storage
    1.3k
    pg-mem is an experimental in-memory emulation of a postgres database.
    oguimbal An in memory postgres DB instance for your unit tests
     
    Storage
    3.7k
    It implements a backend for sql.js (sqlite3 compiled for the web) that treats IndexedDB li...
    jlongster It implements a backend for sql.js (sqlite3 compiled for the web) that treats IndexedDB like a disk and stores data in blocks there. That means your sqlite3 database is persisted. And not in the terrible way of reading and writing the whole image at once -- it reads and writes your db in small chunks.
     
    Storage
    36
    Smallest and fastest JSON database for Node, Electron and browser.😋
    thearkxd A simple JSON database module
     
    Storage
    3k
    sql.js is a light wrapper around SQLite compiled with EMScripten for use in the browser (c...
    phiresky sql.js is a light wrapper around SQLite compiled with EMScripten for use in the browser (client-side).
     
    Storage
    19
    DiggyDB - Amazon Route53 as a blazingly fast and reliable database 🚀
    njbmartin DiggyDB goes one step further than simple a key/value TXT record by allowing you to use (or indeed abuse) DNS TXT records by storing JSON data, almost as though it was a MongoDB or AWS DynamoDB!
     
    Storage
    2.1k
    Cloud Database Manager
    dbeaver CloudBeaver is a web server which provides rich web interface. Server itself is a Java application, web part is written on TypeScript and React.