r.va.gg

LevelUP v0.9 - Some Major Changes

LevelDB

LevelUP is still quite young and bound to go through some major shifts. It's best to not be too tied to immature APIs early in a project's lifetime.

That said, we're very interested in stability so we try to keep breaking changes to a minimum. However, we're about to publish version 0.9 and there's one change that's not exactly a "breaking" change in the normal sense, but it is something that I need to explain because it will impact on almost everyone currently using LevelUP.

Severing the dependency on LevelDOWN

LevelUP depends on LevelDOWN to do its LevelDB thing. LevelDOWN was once part of LevelUP until we split it off to a discrete project that focuses entirely on acting as a direct C++ bridge between LevelDB and Node. We get to focus on making LevelUP an awesome LevelDB-ish interface without being tied directly to LevelDB implementation details (e.g. Iterators vs Streams).

In fact, a new project was spawned to define the LevelDOWN interface that LevelUP requires. AbstractLevelDOWN is a set of strict tests for the functionality that LevelUP uses and it also implements a basic abstract shell that can be extended to create additional back-ends for LevelUP.

So far, there are 3 projects worth mentioning that extend AbstractLevelDOWN:

  • level.js operates on top of IndexedDB (which is in turn implemented on top of LevelDB in Chrome!).

  • leveldown-gap is another browser implementation that uses localStorage and is designed to be able to work in PhoneGap applications.

  • MemDOWN is a pure in-memory implementation that doesn't touch the disk. It's obviously not good for persistent data but sometimes that's not what you need.

Plus some other efforts to adapt other embedded and non-embedded data stores to the LevelDOWN interface. Additionally, there are other versions of LevelDB that can be used, including the fork that Basho maintains for use in Riak. (I have a branch of LevelDOWN that uses this version of LevelDB that I'll release as soon as I can explain and demonstrate the performance differences to vanilla LevelDB for Node users).

In short, LevelUP doesn't need LevelDOWN in the way it once did and LevelUP is turning into a more generic interface to sorted key/value storage systems, albeit with a distinct LevelDB-flavour.

Since version 0.8 we've supported a 'db' option when you create a LevelUP instance. This option can be used to provide an alternative LevelDOWN-compatible back-end. Unfortunately, LevelDOWN being defined as a strict dependency of LevelUP means that each time you install it you have to compile LevelDOWN, even if you don't want it. So, we've removed it as a dependency but it's still wired up so that that the only thing you need to do is actually install LevelDOWN alongside LevelUP and it'll take care of the rest.

$ npm install levelup leveldown

From version 0.9 onwards, you'll need to do this, or you'll see an (informative) error.

Introducing "Level"

To make life easier, we're publishing an additional package in npm that will make this easier by bundling both LevelUP and LevelDOWN as dependencies and exposing LevelUP directly. The Level package is a very simple wrapper that exists purely as a convenience. It'll track the same versioning as LevelUP so it's a straight substitution.

$ npm install level

You can simply change your "dependencies" from "levelup" to "level", plus you can use it just like LevelUP:

var levelup = require('level')
var db = levelup('./my.db')
db.put('yay!', 'it works!')

Switching things up

Now we have a properly pluggable back-end, expect to see a growing array of choice and innovation. The most exciting space at the moment is browser-land. Consider level.js:

var levelup = require('levelup')
  , leveljs = require('level-js')

window.db = levelup('foo', { db: leveljs })

db.put('name', 'LevelUP string', function (err) {
  db.get('name', function (err, value) {
    console.log('name=' + value)
  })
})

Yep, that's browser code. Simply npm install levelup level-js and run the module through Browserify and you get the full LevelUP API in your browser!


Stay tuned! This is just one step in the quest for a truly modular database system that lets you build a database that suits your applications and not the other way around.

Node.ninjas Presentation - LevelDB and Node Sitting in a Tree

I'm giving a presentation at Node.ninjas tonight in Sydney. I've put together a talk about LevelDB and Node that covers:

  1. What LevelDB is and the basics of how it works
  2. A quick introduction to the core LevelDB libraries in Node: LevelUP and LevelDOWN
  3. Some preaching about the awesomeness of modularity around a small, extensible core; including a whirlwind tour of the current, flourishing, LevelDB+Node ecosystem

It's this last point that excites me the most. There's some very smart people building some very clever pieces to the Node Database puzzle. What's more, people are actually building functional databases in Node now, I've just collected a list from npm of what looks like functional databases that use LevelDB:

  • Rumours
  • LevelGraph
  • PushDB
  • NeutrinoDB
  • PlumbDB
  • Syncstore

And a few more that look like a work in progress. Plus, I'm sure there's more people out there we've never even heard of who are cooking up some amazing things using the LevelDB+Node combination!

The slides to my talk are here.

LevelDB and Node: Getting Up and Running

This is the second article in a three-part series on LevelDB and how it can be used in Node.

Our first article covered the basics of LevelDB and its internals. If you haven't already read it you are encouraged to do so as we will be building upon this knowledge as we introduce the Node interface in this article.

LevelDB

There are two primary libraries for using LevelDB in Node, LevelDOWN and LevelUP.

LevelDOWN is a pure C++ interface between Node.js and LevelDB. Its API provides limited sugar and is mostly a straight-forward mapping of LevelDB's operations into JavaScript. All I/O operations in LevelDOWN are asynchronous and take advantage of LevelDB's thread-safe nature to parallelise reads and writes.

LevelUP is the library that the majority of people will use to interface with LevelDB in Node. It wraps LevelDOWN to provide a more Node.js-style interface. Its API provides more sugar than LevelDOWN, with features such as optional arguments.

LevelUP exposes iterators as Node.js-style object streams. A LevelUP ReadStream can be used to read sequential entries, forward or reverse, to and from any key.

LevelUP handles JSON and other encoding types for you. For example, when operating on a LevelUP instance with JSON value-encoding, you simply pass in your objects for writes and they are serialised for you. Likewise, when you read them, they are deserialised and passed back in their original form.

Continue reading this article on DailyJS.com

LevelDB and Node: What is LevelDB Anyway?

This is the first article in a three-part series on LevelDB and how it can be used in Node.

This article will cover the LevelDB basics and internals to provide a foundation for the next two articles. The second and third articles will cover the core LevelDB Node libraries: LevelUP, LevelDOWN and the rest of the LevelDB ecosystem that's appearing in Node-land.

LevelDB

What is LevelDB?

LevelDB is an open-source, dependency-free, embedded key/value data store. It was developed in 2011 by Jeff Dean and Sanjay Ghemawat, researchers from Google. It's written in C++ although it has third-party bindings for most common programming languages. Including JavaScript / Node.js of course.

LevelDB is based on ideas in Google's BigTable but does not share code with BigTable, this allows it to be licensed for open source release. Dean and Ghemawat developed LevelDB as a replacement for SQLite as the backing-store for Chrome's IndexedDB implementation.

It has since seen very wide adoption across the industry and serves as the back-end to a number of new databases and is now the recommended storage back-end for Riak.

Continue reading this article on DailyJS.com

Node.js Dublin Presentation - LevelDB

I visited lovely Dublin last month to attend PeerConf. While there I got to meet a great bunch of Irish programmers at Node.js Dublin, a semi-regular Node.js meet-up that happens in the Engine Yard office in Dublin.

I was invited to give a presentation on LevelDB and the work that I've been doing on it in Node.js. I was followed by Dominic Tarr who's doing some amazing work on top of LevelDB.

You can view my slides here but a written version is currently being spread over 3 parts on DailyJS. More about that soon!