Newcomers to CouchDB offerings often fall into two categories: people that use it purely as a key-value store, and people that are stuck wondering how to query non-primary-keyed data.
One answer built in to CouchDB is “map-reduce”.
Let’s dive in on a simple example. We’ll model a recipe book of bartending drinks. This example provides two interesting points. First, modeling recipes in relational databases provides for strong integrity, however, it is not very intuitive modeling. I once tried to explain to a Rails newcomer how to create join models between a recipe and ingredients, and they found it fairly confusing. Secondly, drink recipes provide a clear use case for a secondary index search by ingredient.
First, we’ll store some simple single document drink recipes that look like this:
{
"type": "DrinkRecipe",
"name": "Angler's Cocktail",
"ingredients": [
{ "name": "Gin", "amount": "2 oz" },
{ "name": "Angostura Bitters", "amount": "3 dashes" },
{ "name": "Orange Bitters", "amount": "3 dashes" },
{ "name": "Grenadine", "amount": "splash" },
{ "name": "Red Maraschino Cherry", "amount": "1"}
],
"note": "Serve in an old fashioned glass over cracked ice"
}
Briefly, if you’re not familiar with CouchDB, take note of the type property. That’s a CouchDB convention of marking the document to distinguish them from each other. Unlike collections in MongoDB, or tables in SQL stores, your docs for an app typically exist in a single database. The rest of the properties are pretty self explanatory, but take note that it models the recipe as you might write it on a bit of scrap paper (minus the braces and keys).
Now let’s talk about two simple views. First, it’s pretty likely you’ll want to be able to list all your recipes by name. Here’s the one possible view for that:
function(doc) {
if (doc.type === 'DrinkRecipe') {
emit(doc.name.toLowerCase(), doc.name);
}
}
We’ll store that in the ‘drinks’ design doc as ‘byName’. We’re emitting the document name in lowercase as the key (the first argument to emit), and the name as the value so we can preserve the case. Let’s grab at that view with curl:
$ curl http://127.0.0.1:5984/drinks/_design/drinks/_view/byName
{"total_rows":4,"offset":0,"rows":[
{"id":"3b510371b46c2f20cd7d72a527007e2a","key":"angler's cocktail","value":"Angler's Cocktail"},
{"id":"3b510371b46c2f20cd7d72a52700af45","key":"manhattan","value":"Manhattan"},
{"id":"3b510371b46c2f20cd7d72a527008dd5","key":"martini","value":"Martini"},
{"id":"3b510371b46c2f20cd7d72a527009c27","key":"old fashioned","value":"Old Fashioned"}
]}
The most important thing to note here is that the keys come back in sorted order, in this case alphabetical.
Now, say we wanted drinks starting with ‘m’. We can leverage the start and endkey params as such:
$ curl 'http://127.0.0.1:5984/drinks/_design/drinks/_view/byName?startkey="m"&endkey="n"&inclusive_end=false'
{"total_rows":4,"offset":1,"rows":[
{"id":"3b510371b46c2f20cd7d72a52700af45","key":"manhattan","value":"Manhattan"},
{"id":"3b510371b46c2f20cd7d72a527008dd5","key":"martini","value":"Martini"}
]}
Note that total_rows is 4, that’s because the view has 4 rows in total, but, our startkey “m” and endkey “n” only return two rows. Kind of strange but just be aware of it. Also, we’ve got back just our rows that start with “m” but not “n” because we disabled the inclusive_end.
OK, let’s introduce a second view to enable searching by ingredient:
// map
function(doc) {
if (doc.type === "DrinkRecipe") {
doc.ingredients.forEach(function(i) {
emit([i.name.toLowerCase(), doc.name.toLowerCase()], null);
});
}
}
// reduce
_count // uses the built in _count implementation
We’ve included the ingredient name as the first element of the key array (recall we had a string as key on the previous view), and the drink name as the second. This gives us already sorted results, as well as some search and aggregation ability as we’ll shortly see.
Let’s query it, we’ll get a row for every ingredient in every drink. We’ll pass it reduce=false to tell it to only run the map phase:
curl 'http://127.0.0.1:5984/drinks/_design/drinks/_view/byIngredient?reduce=false'
{"total_rows":17,"offset":0,"rows":[
{"id":"3b510371b46c2f20cd7d72a527007e2a","key":["angostura bitters","angler's cocktail"],"value":null},
{"id":"3b510371b46c2f20cd7d72a52700af45","key":["angostura bitters","manhattan"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527009c27","key":["angostura bitters","old fashioned"],"value":null},
{"id":"3b510371b46c2f20cd7d72a52700af45","key":["bourbon","manhattan"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527009c27","key":["bourbon","old fashioned"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527008dd5","key":["dry vermouth","martini"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527007e2a","key":["gin","angler's cocktail"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527008dd5","key":["gin","martini"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527007e2a","key":["grenadine","angler's cocktail"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527007e2a","key":["orange bitters","angler's cocktail"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527008dd5","key":["orange bitters","martini"],"value":null},
{"id":"3b510371b46c2f20cd7d72a52700af45","key":["orange peel","manhattan"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527009c27","key":["orange wedge","old fashioned"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527007e2a","key":["red maraschino cherry","angler's cocktail"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527009c27","key":["sugar","old fashioned"],"value":null},
{"id":"3b510371b46c2f20cd7d72a52700af45","key":["sweet vermouth","manhattan"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527009c27","key":["water","old fashioned"],"value":null}
]}
Cool, let’s try some queries with different options.
Find any drink recipe that uses gin:
curl 'http://127.0.0.1:5984/drinks/_design/drinks/_view/byIngredient?reduce=false&startkey=\["gin"\]&endkey=\["gin",\[\]\]'
{"total_rows":17,"offset":6,"rows":[
{"id":"3b510371b46c2f20cd7d72a527007e2a","key":["gin","angler's cocktail"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527008dd5","key":["gin","martini"],"value":null}
]}
The startkey is [“gin”] and the endkey is [“gin”,[]]. The endkey contains a bit of a conventional trick, it is the start key with [] appended on to it. Since we have a 2 element array key, this is a bit like a wildcard search on the last element, and matches on the first. Because CouchDB will keysort [] after a string like ‘zzzzzzz’ we are sure to get all our gin drinks back.
Now let’s get a little trickier and use the reduce phase as well as a group_level option.
curl 'http://127.0.0.1:5984/drinks/_design/drinks/_view/byIngredient?group_level=1'
{"rows":[
{"key":["angostura bitters"],"value":3},
{"key":["bourbon"],"value":2},
{"key":["dry vermouth"],"value":1},
{"key":["gin"],"value":2},
{"key":["grenadine"],"value":1},
{"key":["orange bitters"],"value":2},
{"key":["orange peel"],"value":1},
{"key":["orange wedge"],"value":1},
{"key":["red maraschino cherry"],"value":1},
{"key":["sugar"],"value":1},
{"key":["sweet vermouth"],"value":1},
{"key":["water"],"value":1}
]}
By default, a view with a reduce specified runs, so we just remove that param from our query. Also we’ve added group_level as 1. This means that we will be adding up the number of rows using the first key element only for our reduce. The value above is the sum of rows for each key, so there are 3 drinks that use “angostura bitters” in them. Had we used group_level 2, in this case everything would come back with value “1” because all our keys in full are unique.
Now, a natural reaction to searching by ingredient leads to a question: what if you want to search by two ingredients? Well, at this point, there are at least two obvious solutions. First, you could perform 2 queries of the view, 1 for each ingredient, and then do an in app intersection. A second would be to use couchdb-lucene to enable full text search on your ingredients. Cloudant also bakes this straight into their hosted BigCouch offering, you can read their docs here about it if you want to get up and running quickly.
At the outset I said we’d make two views. Let’s add a bonus one. Imagine you’ve got some bizarro recipe that calls for an ingredient of the same name twice (this happens much more in baking than mixology). An observant eye of our byIngredients view would spot that, we’d output two rows for the same ingredient, thereby possibly offsetting our results (think pagination) when duplicates are involved.
We could add another (or tweak our original code()byIngredient) view to include the doc._id. Let’s call it byDedupedIngredient:
// map
function(doc) {
if (doc.type === "DrinkRecipe") {
doc.ingredients.forEach(function(i) {
emit([i.name.toLowerCase(), doc.name.toLowerCase(), doc._id], null);
});
}
}
// reduce
_count
Now let’s add a rather contrived “Extra Gin Martini” to our book:
{
"type": "DrinkRecipe",
"name": "Extra Gin Martini",
"ingredients": [
{
"name": "Gin",
"amount": "2 oz"
},
{
"name": "Gin",
"amount": "1 oz"
},
{
"name": "Dry Vermouth",
"amount": "1/4 oz"
},
{
"name": "Orange Bitters",
"amount": "A dash"
}
],
"note": "Shake with ice and serve in a martini glass"
}
Let’s query it twice, first without a reduce, and secondly with a reduce phase
curl 'http://127.0.0.1:5984/drinks/_design/drinks/_view/byDedupedIngredient?reduce=false&startkey=\["gin"\]&endkey=\["gin",\[\]\]'
{"total_rows":21,"offset":7,"rows":[
{"id":"3b510371b46c2f20cd7d72a527007e2a","key":["gin","angler's cocktail","3b510371b46c2f20cd7d72a527007e2a"],"value":null},
{"id":"3b510371b46c2f20cd7d72a52700c0fe","key":["gin","extra gin martini","3b510371b46c2f20cd7d72a52700c0fe"],"value":null},
{"id":"3b510371b46c2f20cd7d72a52700c0fe","key":["gin","extra gin martini","3b510371b46c2f20cd7d72a52700c0fe"],"value":null},
{"id":"3b510371b46c2f20cd7d72a527008dd5","key":["gin","martini","3b510371b46c2f20cd7d72a527008dd5"],"value":null}
]}
curl 'http://127.0.0.1:5984/drinks/_design/drinks/_view/byDedupedIngredient?reduce=true&group_level=3&startkey=\["gin"\]&endkey=\["gin",\[\]\]'
{"rows":[
{"key":["gin","angler's cocktail","3b510371b46c2f20cd7d72a527007e2a"],"value":1},
{"key":["gin","extra gin martini","3b510371b46c2f20cd7d72a52700c0fe"],"value":2},
{"key":["gin","martini","3b510371b46c2f20cd7d72a527008dd5"],"value":1}
]}
Can you spot the difference? Look closely for the “extra gin martini”. In the first query, it shows up twice. In the second, because we are running a reduce and group on the exact key at a group level of 3, it dedupes and returns with a value of 2.
Hopefully this has been of some help to folks interested in CouchDB, but not familiar with it, especially its map-reduce side. I should hatch a post in the near future to show you how to make more complicated collated views for returning richer, associated data in a single query.
—Nov 14, 2013
Unless you live under a rock, you’ve heard of the key-value store Redis. It’s possible though, you’ve never used it. Maybe you don’t have giant, loaded site needs like Imgur, so you haven’t added it to your stack. That’s cool. In fact, I’m a proponent of not adding an additional cog to a production stack unless it either greatly reduces load on an existing part or makes the code easier to understand. That said, Redis has a reputation for helping out when your site is under giant load. But there’s something I think is great about Redis that makes it useful for small tasks: its simplicity.
Redis has a few different structure types, take Hashes and Lists. For example, using the redis-rb client, you can set values in a Redis hash:
redis.hset "user:1", :name, "Brendon"
redis.hset "user:1", :gender, "male"
redis.hget "user:1", "name" #=> "Brendon"
redis.hgetall "user:1" # => {"gender"=>"male", "name"=>"Brendon"}
redis.hvals "user:1" # => ["male", "Brendon"]
This feels like a simple, hash like primative to work with. Though it is syntatically different than working with a plain old ruby Hash, the feel is similar. I think this is a big win; it’s easy to understand and feels familiar, plus, it’s network distributed.
Check out some List basics:
redis.rpush "comments", "Awesome"
redis.rpush "comments", "Super"
redis.llen "comments" # => 2
redis.lpop "comments" # => "Awesome"
Again, syntatically a little different than using an Array, but still familiar.
Redis itself is very easy to install. You can install it from homebrew, or apt-get or just compile from source. It’s small, installs quickly, and unless you fill it with gigabytes of data, it’s likely you will forget it’s there. I install it by default on most my systems, even little underpowered ones. The reason being, I love Redis for my little one off scripting tasks because of a few features:
- It’s liteweight
- Data has optional expiration so it will clean up after itself for free
- Its commands for get/set, lists, and pub/sub are great for basic IPC.
Here’s some examples. I’m excluding exception handling and Redis watches/multi/transaction concepts to focus on the core idea.
Redis is great at deduping. Say you want to write a script that pushes campfire notifications to gently remind the team to keep commit messages under 50 characters, but not resend reminders in the case of forced pushes, rebases, merges, etc:
# Generate your own digest to avoid git hash changing
id = Digest::MD5.hexdigest("#{commit.author}:#{commit.message}")
# unseen will only be true if the id is not already in Redis
unseen = redis.setnx(id, Time.now.to_i)
# expire after 90 days to save on memory
redis.expire(id, 7776000) if unseen
# run notification if unseen
Or imagine you have a system that is applying regex to RSS feeds, and you want to be alerted at most once a day about matches:
id = Digest::MD5.hexdigest(regex_pat.to_s)
# Exclusively set the key, returning false if it already exists
unseen = redis.setnx(id, Time.now.to_i)
redis.expire(id, 86_400) if unseen
I do this in small scripts all the time. Now, I could write to sqlite, use the ruby PStore, or maybe a tempfile as lock, etc, but this just feels so dang simple, and I don’t worry about file locking, sql queries, weird commit code, etc.
Maybe you have scripts you want to “shut up” for a few hours. You could write a file and stat it, or you could just set a value to read:
# Don't bug for 5 minutes
redis.setex("shutup", 300, "true")
Often I have little tasks or cronjob on my home server that I want notifications about. Currently I use Pushover for all those notifications. Someday I might change my mind and switch to email or SMS. So what I’ve done is treat Redis like a good old Unix named pipe. But now I don’t have to worry about creating the pipe, or permissions, and I seem to remember Redis commands much easier (pop quiz…how do you call mkfifo in Ruby without shelling out?).
If you want to make a FIFO for notifications, write something like this:
# Producer script, cron job, etc
redis.rpush("notifications", "You've got mail!")
# Notification daemon
loop do
# blpop is a blocking read left pop
key, message = redis.blpop("notifications")
# send message to SMS, email, Pushover, etc
end
Sometimes, your notification daemon might die and not restart properly. Maybe you want to “catch up” on messages when you restart it, then the above list use is good. If you don’t want to get flooded by those missed notifications, simply switch to pub/sub:
# Producer
redis.publish("notifications", "A wild message arrives!")
# Notification daemon
Redis.new.subscribe("notifications") do |on|
on.message do |channel, msg|
send_notification(msg)
end
end
Now for one last tip. Redis provides redis-cli, a command line client for interacting with Redis. If you fire it up with no commands, you’ll get an interactive prompt. However, you can also invoke it with a command to run. This is great for shell scripts:
redis-cli publish notifications 'John sent you a private message on irc'
If you haven’t tried Redis yet and program small tasks, you can get started today!
—Jun 07, 2013
Shield is a simple authentication gem I tend to reach for first when developing small Sinatra or Cuba apps. The code is short, reliable, and easy to understand. It’s not a kitchen sink solution, it does one thing well: lets users log in with a password.
As an experiment, I wondered what it might look like to layer a simple two-factor authentication scheme on top of Shield. If you are unfamiliar with two-factor authentication, it typically follows that a user will first log in to a site with their password as usual, and if successful, must pass a second step of entering a PIN from a SecureID type token, or out-of-band delivery message such as SMS. Popular sites implementing two-factor auth include PayPal, Twitter, and Google.
Let’s break down some of the components of what a simple system might look like.
First, we’ll create a basic User model with Ohm, a persistance library backed by Redis.
class User < Ohm::Model
include Shield::Model
attribute :email
attribute :crypted_password
index :email
def self.fetch(email)
User.find(email: email).first
end
end
This is pretty straightforward. We’ve established a user with an email, crypted password, and the fetch method Shield expects per its implementation. Moving on, we’ll tweak the Shield helpers a little and setup some of our own for handling the second factor.
helpers do
include Shield::Helpers
alias_method :initially_authenticated, :authenticated
def authenticated(model)
if user = initially_authenticated(model)
user.id.to_s == session["#{model}_secondary_auth"].to_s && user
end
end
def challenge_authentication(model)
ChallengeAuthentication.new(initially_authenticated(model))
end
def send_challenge(model)
challenge_authentication(model).push
end
def challenge_accepted(model, challenge)
if challenge_authentication(model).check!(challenge)
user = initially_authenticated(model)
session["#{model}_secondary_auth"] = user.id.to_s
end
end
end
There’s a few interesting things to note here. First, we alias Shield’s authenticated method out to initially_authenticate. We’ll use this to check if a user passes the initial password authentication step. Next, we define our new authenticated method, which will rely on password authentication, and a second check against the session to see if the user has passed the secondary authentication step. Sprinkle in some methods around checking our challenge authentication (more details on that in a minute) and our helpers are good to go.
Now let’s move on to some simple Sinatra app and routing setup. First, we’ll handle 401 Unauthorized errors by redirecting to the /login path:
error 401 do
redirect '/login'
end
Next, add in the /login routes Shield typically expects, however, instead of redirecting on success to the main page, we’ll redirect the browser on to a verification step:
get '/login' do
erb :login
end
post '/login' do
if login(User, params[:login], params[:password])
remember(initially_authenticated(User)) if params[:remember_me]
send_challenge(User)
redirect '/login_verification'
else
redirect '/login'
end
end
Note that when a user passes the first login stage, we’ll send them the challenge for the verification step.
Now we’ll get into the meat of the routing and diverge from the vanilla Shield login flow a bit. Setup the verification handling:
get '/login_verification' do
error(401) unless initially_authenticated(User)
erb :login_verification
end
post '/login_verification' do
error(401) unless initially_authenticated(User)
if challenge_accepted(User, params[:challenge])
redirect '/'
else
redirect '/login_verification'
end
end
For either action to succeed, first the user must be initially authenticated. If so, we’ll verify the challenge presented, in this case a randomly assigned 6 digit PIN, matches up. Then we’ll let the user proceed on to the app.
The core of the challenge authentication is handled in a ChallengeAuthentication class:
class ChallengeAuthentication
EXPIRE_TIME = 300
attr_reader :user
def initialize(user)
@user = user
end
def redis
Ohm.redis
end
def push
challenge = generate_challenge
redis.setex key, EXPIRE_TIME, challenge
deliver_challenge(challenge)
end
def check(challenge)
return false if user.nil? || challenge.to_s.empty?
# Should be a secure compare to prevent timing attacks
redis.get(key) == challenge
end
def check!(challenge)
!! ( check(challenge) && redis.del(key) )
end
def key
[user.class.name, user.id, 'challenge'].join(':')
end
# Returns a 6 digit challenge phrase
def generate_challenge
(SecureRandom.random_number * 1_000_000).to_i
end
def deliver_challenge(challenge)
# send an out of band challenge like SMS or Pushover here
end
end
The flow of the challenge check is fairly simple. First, we can push a new challenge by setting a random 6 digit pin for the user, and deliver that out of band. This can be easily accomplished via SMS with a provider such as Twilio, or my favorite push notification service Pushover. Checkout my Rushover gem as a simple client for sending to Pushover. We’ll toss that PIN into Redis with a 5 minute expiration; this provides a simple limited window for which the PIN is valid. Likewise, if you want to implement this on top of a SQL ORM, you could add challenge and expiration timestamps on to your User model.
For checking that the user has provided a valid challenge PIN, we can compare against the value in Redis if it exists. Upon match, we’ll delete the PIN from redis to invalidate it and confirm the challenge is accepted.
You can find the complete code for this example as a gist. Hopefully it’s easy enough to follow, and now you can provide an extra level of security for your social cat-video apps!
Until next time…
—Jun 05, 2013
As a slight extension to the previous post, I wanted to make a quick point about authorization failures.
Given you’ve raised SomeAuthorizationFailure exception in a controller action, you might have a general rescue handling it:
rescue_from 'SomeAuthorizationFailure' do
render :text => "Bad user!", :status => 403
end
The key here is the 403 status, Forbidden. This is a pretty natural, and technically correct status to feed the client.
Cool, let’s wrap that up, it’s done! Hold on, not so fast.
If you use Github (if?), you may have noticed something that struck you as curious the first time it happened. Say you’re hanging in the dev campfire room, and somebody pastes a link to a line of code for you to checkout (like /foocorp/awesomeproject/config/application.rb#L7). You clicked on the link but forgot you’ve logged out. Boom….403 Forbidden.
Wait..no that’s not a 403, it’s a 404. What the heck?
The answer is pretty simple. On your little todo app you run for friends and family, it’s probably not a huge deal for somebody to hit /todo_lists/42/item/5 and get a 403. Wow, somebody now knows you have a todo list 42 and item 5. Probably not a big deal.
But on a site like Github, let’s change that application.rb link to say, /foocorp/awesomeproject/config/initializers/devise.rb. 403? Oh look, that project is using Devise!
The moral of the story: best to give a 404 status on authorization failures if you don’t want to cater to mining and leaking of sensitive info.
—Apr 24, 2013