LevelDB and Node
An Introduction

by Rod Vagg / tw:@rvagg / gh:rvagg / bl:http://r.va.gg

LevelDB in a Nutshell

  • Open-source, embedded key/value store by Google
  • Sorted by keys
  • Values are compressed with Snappy
  • Basic operations: Get(), Put(), Del()
  • Atomic Batch()
  • Bi-directional iterators

Basic architecture

LSM-tree

  • Writes go straight into a log / memtable
  • Log is flushed to 2MB string sorted table (SST) files
  • Reads merge the log and the table files
  • Cache speeds up common reads

Table file hierarchy

The "Level" in LevelDB

Log: Max size of 4MB then flushed into a set of Level 0 SST files
Level 0: Max of 4 SST files then one file compacted into Level 1
Level 1: Max total size of 10MB then one file compacted into Level 2
Level 2: Max total size of 10 x Level 1 then one file compacted into Level 3
Level 3+: Max total size of 10 x previous level then one file compacted into next level

Roughly: 0 ↠ 4 SST, 1 ↠ 10M, 2 ↠ 100M, 3 ↠ 1G, 4 ↠ 10G, 5 ↠ 100G, 6 ↠ 1T+

Get to the Node bit!

LevelDOWN http://ghub.io/leveldown

Pure C++ interface between Node and LevelDB

LevelUP http://ghub.io/levelup

A Node-style interface to LevelDB, extending LevelDOWN: Sugar, Streams, JSON & other encodings, extensible

AbstractLevelDOWN http://ghub.io/abstract-leveldown

A set of classes for implementing the LevelDOWN API

Level http://ghub.io/level

Basic operations

var db = level('/path/to/database')

db.put('key', 'value', function (err) { /* ... */ })
db.get('key', function (err, value) { /* ... */ })
db.del('key', function (err) { /* ... */ })

db.close(function (err) { /* closed */ })
// multiple atomic writes with batch()
var operations = [
    { type: 'put', key: 'Franciscus', value: 'Jorge Bergoglio' }
  , { type: 'del', key: 'Benedictus XVI' }
]

db.batch(operations, function (err) { /* ... */ })

A simple example

var level = require('level')
var db = level('/tmp/dprk.db')

db.put('name', 'Kim Jong-un', function (err) {
  db.batch([
      { type: 'put', key: 'spouse', value: 'Ri Sol-ju' }
    , { type: 'put', key: 'dob', value: '8 January 1983' }
    , { type: 'put', key: 'occupation', value: 'Clown' }
  ], function (err) {
    db.createReadStream()
      .on('data', console.log)
      .on('end', function () { db.close() })
  })
})

Streams!

function copy (srcdb, destdb, callback) {
  srcdb.createReadStream()
    .pipe(destdb.createWriteStream())
    .on('error', callback)
    .on('end', callback)
}

Streams!

a.k.a. Range Queries

var rs = db.createReadStream()
rs.on('error', function (err) { /* handle err */ })
rs.on('data' , function (data) { /* data.key & data.value */ })
rs.on('end', function () { /* stream finished */ })

// Options! Oh my!
db.createReadStream({
    start     : 'somewheretostart'
  , end       : 'endkey'
  , reverse   : true
})

// assuming namespacing with '~' separator
db.createReadStream({ start: 'range~', end: 'range~\xff' })

Encoding

var db = level('/path/to/db', { valueEncoding: 'json' })
var data = {
    name       : 'Kim Jong-un'
  , spouse     : 'Ri Sol-ju'
  , dob        : '8 January 1983'
  , occupation : 'Clown'
}

db.put('dprk', data, function (err) {
  db.get('dprk', function (err, value) {
    console.log('dprk:', value)
    db.close()
  })
})

UTF8 (default), JSON, Buffer encoding types

Modularity FTW!

Some pretty benchmark graphing

To be taken with a few large grains of salt

The End!

Rod Vagg / tw:@rvagg / gh:rvagg / bl:http://r.va.gg

Resources

LevelUPghub.io/levelup
LevelDOWNghub.io/leveldown
Extensions & toolsghub.io/levelup/wiki/Modules
Articlesghub.io/levelup/wiki/Resources

Help & community

IRC ↠ ##leveldb on Freenode
Google groupnode-levelup
Issue trackerghub.io/levelup/issues