A Real Database Rethink

@rvagg

"NodeBase"
(or) "Level*"

Databases: a short history

1960s: From tapes and batch to disks, shared access and interactivity

...not a "database"

Late 1960s: Navigational databases: links

Early 1970s: The relational model: content

Late 1970s: SQL

Early 1980s: A database on my desktop (dBASE and its ilk)

Late 1980s: Object Orient ALL THE THINGS!

2000s: Speed and scale: NoSQL
"NewSQL": never let a beautiful abstraction go to waste

The tyranny of a beautiful abstraction

An abstraction that fits many problems very well will be made to fit all related problems

Programmers take Maslow's hammer to the next level

So what is a Database?

A tool for interacting with structured data, externalised from the core of our application

  • Persistence
  • Performance
  • Simplify access to complex data

And sometimes...

  • Shared access
  • Scalability

The Node approach

  • Small core, vibrant user-land
  • Extreme modularity
  • Reimplement everything in JavaScript!

Applied to databases?

  • Small core: LevelUP
  • Everything as a module
  • Pulling in many aspects of database practice & theory

Targeted solutions

Tools
lev levelweb
Packages
tacodb couchup LevelGraph firedup level-assoc
level-static level-store level-session level-fs LevelTTLCache
Extensions
level-live-stream map-reduce level-queryengine Level-Multiply
multilevel level-replicate level-master Level TTL
Extensibility
sublevel level-hooks level-mutex
Core
LevelUP
Storage
LevelDOWN LevelDOWN (Hyper) LevelDOWN (Basho) MemDOWN level.js leveldown-gap LMDB mysqlDOWN

LevelDB

  • Open-source, embedded key/value store by Google
  • Sorted by keys
  • Values are compressed with Snappy
  • Basic operations: Get(), Put(), Del()
  • Atomic Batch()
  • Bi-directional iterators

LevelUP

Inspired by LevelDB

Backed by a key/value store for arbitrary data, sorted by key

  • Core operations: put(), get(), del()
  • Atomic batch writes
  • ReadStream: the secret sauce
  • WriteStream: for convenience

@chesles @raynos @dominictarr @maxogden @ralphtheninja @kesla @juliangruber
@hij1nx @no9 @mcollina @pgte @substack @rvagg

ReadStream

An essential primitive for building complex features

The core query mechanism to access sorted data

Arbitrary start and end

db.createReadStream({ start: 'Water', end: 'Water\xff' })
  .on('data', function (entry) {
    console.log(entry.key)
  })

// → Waterford
// → Watergrasshill
// → Waterville

Key structure

Key-based sorting and querying requires good key design

Keys as hierarchical descriptors of content:

'countries~Ireland'
'countries~Israel'
...
'towns~Ireland~Waterford'
'towns~Ireland~Watergrasshill'
...
'streets~Ireland~Waterford~Dunmore Road'
'streets~Ireland~Waterford~Derrynane Close'
...

Building-blocks

From: a 1-dimensional storage array

To: a multi-dimensional data tool for customised solutions

The Level* ecosystem: A menu of beautiful, but small abstractions

end / deireadh