Riak, Ruby & Ripple

Tim McGilchrist

(@lambda_foo)

Overview

What is Riak?
Riak Concepts & Features

Buckets and Keys
Links and Metadata
Clustering and Distribution
Replication
Map Reduce

Using Riak from Ruby
The Good / The Bad
Some Conclusions

What is Riak?

Riak is an open source, highly scalable, fault-tolerant and distributed database.

Features

Key/Value store
Highly available, the AP in CAP
Clustered by default
Map Reduce processing
P2P architecture
Secondary Indexes
Full Text Search
Simple Ops

Background

Based on Amazon's Dynamo Key/Value paper and Dr Eric Brewer's CAP Theorem.

CAP theorem, states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees:

Consistency
Availability
Partition Tolerance

You can only pick 2!

Buckets and Keys

Buckets + keys are the main way to organise user data.

Bucket is a collection of keys, with associated configuration data.

Keys can be either a string or binary, user defined or riak can generate one.
eg 37458abc752f8413e, fancy_string or 0xDEADBEEF

Links and Metadata

Lightweight pointers, like HTML tags
Composed of a bucket, key & tag
Traverse through "link-walking"
User defined metadata.

    Link: </riak/bucket/key>; riaktag="whatever?"

    X-Riak-Meta-Color: whatever?

curl -X PUT http://localhost:8091/riak/cages/1 \
  -H "Content-Type: application/json" \
  -H "X-Riak-Meta-Color: Pink" \
  -H "Link: ; riaktag=\"contains\"" \
  -d '{"room" : 101}'

Clustering

Riak's default mode of operation is in a cluster.

CAP Controls: N, R and W

The extent to which data is replicated, when, and with what merge strategy and failure model, is configurable at runtime.

Riak computes a 160-bit binary hash of the bucket/key pair, and maps this value to a position on an ordered "ring" of all such values.

eg 32 partitions / 4 nodes = 8 vnodes for each node

More Clustering

Writes are split across multiple nodes and only returns success if write is confirmed by N nodes.

Reads are also split across multiple nodes with the same consensus protocol.

Inconsistent read results can be repaired without user intervention (Read Repair).

Ring state is shared around the cluster by means of a gossip protocol.

Replication

Replication is fundamental and automatic in Riak.
All data stored in Riak will be replicated to a number of nodes in the cluster according to the n_val property set on the bucket.
Tune n_val wisely; high values provide more redundancy and lower values provide more speed.

Map Reduce

Riak's Map Reduce allows richer queries over the data stored in Riak.

Can be written in Javascript or Erlang.

Spreads the processing of the query to where the data is.

Map Reduce Cont.

Talking the talk

Basho provides supported clients for:

Java
Ruby
Erlang
PHP
Javascript

Many more community contributed ones.

Riak and Ruby

Basho provides great Ruby support via the Ripple library,
https://github.com/seancribbs/ripple

It includes two namespaces:

Riak basic wrapper around typical operations.
Ripple provides a rich modeling layer for Riak, based on ActiveModel abstraction similar to ActiveRecord, DataMapper and MongoMapper.

Supports Rails 3 and greater.

Setting up Riak

Available for all good Unix systems. Instructions available on wiki.basho.com

phoenix$ curl -O http://downloads.basho.com/riak/riak-1.1.1/riak-1.1.1-osx-x86_64.tar.gz
phoenix$ tar xzvf riak-1.1.1-osx-x86_64.tar.gz
phoenix$ cd riak-1.1.1
phoenix:riak-1.1.1$ ./bin/riak start
phoenix:riak-1.1.1$ ./bin/riak ping
pong

Saving data to Riak

require 'riak'

# Create a client connection
client = Riak::Client.new

# Retrieve a bucket
bucket = client.bucket('books')

# Create a new object
new_book = Riak::RObject.new(bucket, 'erlang_otp_in_action')
new_book.content_type = 'application/json'
new_book.data = { title: 'Erlang and OTP in Action',
                  author: 'Logan Martin',
                  summary: 'Erlang is a functional language'}

# Save!
new_book.store

Using Ripple ActiveModel

require 'ripple'

class Book
  include Ripple::Document

  property :title, String, :presence => true
  property :author, String, :presence => true
  property :summary, String, :presence => true
end

book = Book.new
book.title = 'The Well-Grounded Rubyist'
book.author = 'David Black'
book.summary = 'Takes you from interested novice to proficient practitioner.'

book.save

Where's my data?

phoenix$ curl -v http://127.0.0.1:8098/riak/books/erlang_otp_in_action
* About to connect() to 127.0.0.1 port 8098 (#0)
*   Trying 127.0.0.1...
* connected
* Connected to 127.0.0.1 (127.0.0.1) port 8098 (#0)
> GET /riak/books/erlang_otp_in_action HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin11.3.0) libcurl/7.24.0 OpenSSL/1.0.0g zlib/1.2.6 libidn/1.22
> Host: 127.0.0.1:8098
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Riak-Vclock: a85hYGBgzGDKBVIcypz/fvpHeitmMCUy5bEyOOw6f4IvCwA=
< Vary: Accept-Encoding
< Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
< Link: ; rel="up"
< Last-Modified: Sat, 10 Mar 2012 11:42:56 GMT
< ETag: "76dfItNC7ZsSupMV0VIGZ4"
< Date: Sun, 11 Mar 2012 10:15:55 GMT
< Content-Type: application/json
< Content-Length: 213
<
* Connection #0 to host 127.0.0.1 left intact
{"title":"Erlang and OTP in Action","author":"Logan Martin","summary":"Erlang is
an adaptable and fault tolerant functional programming language originally
designed for the unique demands of the telecom industry"}
* Closing connection #0

More Data

phoenix$ curl -v http://127.0.0.1:8098/buckets/books/keys?keys=true
...
< HTTP/1.1 200 OK
< Vary: Accept-Encoding
< Server: MochiWeb/1.1 WebMachine/1.9.0 (someone had painted it blue)
< Date: Sun, 11 Mar 2012 10:42:36 GMT
< Content-Type: application/json
< Content-Length: 155
<
* Connection #0 to host 127.0.0.1 left intact
{"keys":["machine_learning_in_action","erlang_otp_in_action","programming_erlang","well_grounded_rubyist","practical_common_lisp","javascript_good_parts"]}
* Closing connection #0

phoenix$ curl -v -X DELETE http://127.0.0.1:8098/riak/books/erlang_otp_in_action
...
< HTTP/1.1 204 No Content
< Vary: Accept-Encoding
...

Map Reduce

require 'riak'

# do connection stuff here.

results = Riak::MapReduce.new(client).add("books").
  map("function(v) {
     var results = [];
     var book = JSON.parse(v.values[0].data);
     if (book.author == 'Joe Armstrong') {
       results.push(book);
     }
     return results;
  }", :keep => true).run

puts results

{"title"=>"Programming Erlang: Software for a Concurrent World",
 "author"=>"Joe Armstrong",
 "summary"=>"Erlang solves one of the most pressing problems facing developers today."}
{"title"=>"Things Joe Says",
 "author"=>"Joe Armstrong",
 "summary"=>"Collection of quotes from Joe's talks, like Erlang is the most OO language."}

Good Riak

Masterless, all nodes are equal
Read / Write performance scales with nodes
Replication and Sharding are transparent
Administration is simple
Store whatever you want, json/bson, text, images, binaries ...

Bad Riak

Data modeling requires more forward planning
Requires > 2 nodes to see benefits
Data safety is the default
Query interface, map reduce gets old
Map Reduce query tools

What else?

Full Text Search
Commit hooks, invoke code before or after save
Secondary indexes
Luwak, store large binaries
riak_core, github.com/basho/riak_core
riak_mongo, github.com/pavlobaron/riak_mongo

Demo

Change the node name in vm.args
Change HTTP and PB ports in app.config
Startup node
Add node to cluster riak-admin join riak@cluster-node

What's happening?

Resources

http://wiki.basho.com/Riak.html
Dynamo: Amazon's Highly Available Key-value Store
http://en.wikipedia.org/wiki/CAP_theorem
https://github.com/seancribbs/ripple
http://wiki.basho.com/Client-Libraries.html
Seven Databases in Seven Weeks by Eric Redmond and Jim R. Wilson
The Riak Handbook by Mathias Meyer