Tutorial: Node.js and MongoDB JSON REST API server with Mongoskin and Express.js

Update3: Expess 4 version of this tutorial is available at Express.js 4, Node.js and MongoDB REST API Tutorial, and github.com/azat-co/rest-api-express (master branch). This tutorial will work with Express 3.x.

Update2: “Mongoskin removed ‘db.collection.id’ and added some actionById methods” from this pull request with this code changes. To use the code in this post, just install an older version of Mongoskin (0.5.0?). The code in the GitHub will work with Mongoskin 1.3.20.

Update2: “Mongoskin removed ‘db.collection.id’ and added some actionById methods” from this pull request with this code changes. To use the code in this post, just install an older version of Mongoskin (0.5.0?)

Update: use the enhanced code from this repository github.com/azat-co/rest-api-express (express3 branch).

Note: This text is a part of Express.js Guide: The Comprehensive Book on Express.js.

This tutorial will walk you through writing test using the Mocha and Super Agent libraries and then use them in a test-driven development manner to build a Node.js free JSON REST API server utilizing Express.js framework and Mongoskin library for MongoDB. In this REST API server, we’ll perform create, read, update and delete (CRUD) operations and harness Express.js middleware concept with app.param() and app.use() methods.

Test Coverage

Before anything else let’s write functional tests that make HTTP requests to our soon-to-be-created REST API server. If you know how to use Mocha or just want to jump straight to the Express.js app implementation feel free to do so. You can use CURL terminal commands for testing too.

Assuming we already have Node.js, NPM and MongoDB installed, let’s create a new folder (or if you wrote the tests use that folder):

mkdir rest-api
cd rest-api

We’ll use Mocha, Expect.js and Super Agent libraries. To install them run these command from the project folder:

$ npm install mocha@1.18.2 --save-dev
$ npm install expect.js@0.3.1 --save-dev 
$ npm install superagent@0.17.0 --save-dev

Now let’s create express.test.js file in the same folder which will have six suites:

• Creating a new object
• Retrieving an object by its ID
• Retrieving the whole collection
• Updating an object by its ID
• Checking an updated object by its ID
• Removing an object by its ID

HTTP requests are just a breeze with Super Agent’s chained functions which we’ll put inside of each test suite. Here is the full source code for the express.test.js file:

var superagent = require('superagent')
var expect = require('expect.js')

describe('express rest api server', function(){
  var id

  it('post object', function(done){
    superagent.post('http://localhost:3000/collections/test')
      .send({ name: 'John'
        , email: 'john@rpjs.co'
      })
      .end(function(e,res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(res.body.length).to.eql(1)
        expect(res.body[0]._id.length).to.eql(24)
        id = res.body[0]._id
        done()
      })    
  })

  it('retrieves an object', function(done){
    superagent.get('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body._id.length).to.eql(24)        
        expect(res.body._id).to.eql(id)        
        done()
      })
  })

  it('retrieves a collection', function(done){
    superagent.get('http://localhost:3000/collections/test')
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(res.body.length).to.be.above(0)
        expect(res.body.map(function (item){return item._id})).to.contain(id)        
        done()
      })
  })

  it('updates an object', function(done){
    superagent.put('http://localhost:3000/collections/test/'+id)
      .send({name: 'Peter'
        , email: 'peter@yahoo.com'})
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body.msg).to.eql('success')        
        done()
      })
  })

  it('checks an updated object', function(done){
    superagent.get('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body._id.length).to.eql(24)        
        expect(res.body._id).to.eql(id)        
        expect(res.body.name).to.eql('Peter')        
        done()
      })
  })    
  it('removes an object', function(done){
    superagent.del('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body.msg).to.eql('success')    
        done()
      })
  })      
})

To run the tests we can use the $ mocha express.test.js command.

Dependencies

In this tutorial we’ll utilize Mongoskin, a MongoDB library which is a better alternative to the plain good old native MongoDB driver for Node.js. In additition Mongoskin is more light-weight than Mongoose and schema-less. For more insight please check out Mongoskin comparison blurb.

Express.js is a wrapper for the core Node.js HTTP module objects. The Express.js framework is build on top of Connect middleware and provided tons of convenience. Some people compare the framework to Ruby’s Sinatra in terms of how it’s non-opinionated and configurable.

If you’ve create a rest-api folder in the previous section Test Coverage, simply run these commands to install modules for the application:

npm install express@3.5.1 --save
npm install mongoskin@1.3.20 --save

Implementation

First things first, so let’s define our dependencies:

var express = require('express')
  , mongoskin = require('mongoskin')

After the version 3.x, Express streamlines the instantiation of its app instance, in a way that this line will give us a server object:

var app = express()

To extract params from the body of the requests we’ll use bodyParser() middleware which looks more like a configuration statement:

app.use(express.bodyParser())

Middleware (in this and other forms) is a powerful and convenient pattern in Express.js and Connect to organize and re-use code.

As with the bodyParser() method that saves us from the hurdles of parsing a body object of HTTP request, Mongoskin makes possible to connect to the MongoDB database in one effortless line of code:

var db = mongoskin.db('localhost:27017/test', {safe:true});

Note: If you wish to connect to a remote database, e.g., MongoHQ instance, substitute the string with your username, password, host and port values. Here is the format of the URI string: mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]

The app.param() method is another Express.js middleware. It basically says “do something every time there is this value in the URL pattern of the request handler”. In our case we select a particular collection when request pattern contains a sting collectionName prefixed with a colon (you’ll see it later in the routes):

app.param('collectionName', function(req, res, next, collectionName){
  req.collection = db.collection(collectionName)
  return next()
})

Merely to be user-friendly let’s put a root route with a message:

app.get('/', function(req, res) {
  res.send('please select a collection, e.g., /collections/messages')
})

Now the real work begins, here is how we retrieve a list of items sorted by _id and which has a limit of 10:

app.get('/collections/:collectionName', function(req, res) {
  req.collection.find({},{limit:10, sort: [['_id',-1]]}).toArray(function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

Have you noticed a :collectionName string in the URL pattern parameter? This and the previous app.param() middleware is what gives us the req.collection object which points to a specified collection in our database.

The object creating endpoint is slightly easier to grasp since we just pass the whole payload to the MongoDB (method a.k.a. free JSON REST API):

app.post('/collections/:collectionName', function(req, res) {
  req.collection.insert(req.body, {}, function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

Single object retrieval functions are faster than find(), but they use different interface (they return object directly instead of a cursor), so please be aware of that. In addition, we’re extracting the ID from :id part of the path with req.params.id Express.js magic:

app.get('/collections/:collectionName/:id', function(req, res) {
  req.collection.findOne({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send(result)
  })
})

PUT request handler gets more interesting because update() doesn’t return the augmented object, instead it returns us a count of affected objects.

Also {$set:req.body} is a special MongoDB operator (operators tend to start with a dollar sign) that sets values.

The second {safe:true, multi:false} parameter is an object with options that tell MongoDB to wait for the execution before running the callback function and to process only one (first) item.

app.put('/collections/:collectionName/:id', function(req, res) {
  req.collection.update({_id: req.collection.id(req.params.id)}, {$set:req.body}, {safe:true, multi:false}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

Finally, the DELETE method which also output a custom JSON message:

app.del('/collections/:collectionName/:id', function(req, res) {
  req.collection.remove({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

Note: The delete is an operator in JavaScript, so Express.js uses app.del instead.

The last line that actually starts the server on port 3000 in this case:

app.listen(3000)

Just in case something is not working quite well here is the full code of express.js file:

var express = require('express')
  , mongoskin = require('mongoskin')

var app = express()
app.use(express.bodyParser())

var db = mongoskin.db('localhost:27017/test', {safe:true});

app.param('collectionName', function(req, res, next, collectionName){
  req.collection = db.collection(collectionName)
  return next()
})
app.get('/', function(req, res) {
  res.send('please select a collection, e.g., /collections/messages')
})

app.get('/collections/:collectionName', function(req, res) {
  req.collection.find({},{limit:10, sort: [['_id',-1]]}).toArray(function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

app.post('/collections/:collectionName', function(req, res) {
  req.collection.insert(req.body, {}, function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

app.get('/collections/:collectionName/:id', function(req, res) {
  req.collection.findOne({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send(result)
  })
})
app.put('/collections/:collectionName/:id', function(req, res) {
  req.collection.update({_id: req.collection.id(req.params.id)}, {$set:req.body}, {safe:true, multi:false}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})
app.del('/collections/:collectionName/:id', function(req, res) {
  req.collection.remove({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

app.listen(3000)

Exit your editor and run this in your terminal:

$ node express.js

And in a different window (without closing the first one):

$ mocha express.test.js

If you really don’t like Mocha and/or BDD, CURL is always there for you. :-)

For example, CURL data to make a POST request:

$ curl -d "" http://localhost:3000

GET requests also work in the browser, for example http://localhost:3000/test.

In this tutorial our tests are longer than the app code itself so abandoning test-driven development might be tempting, but believe me the good habits of TDD will save you hours and hours during any serious development when the complexity of the applications you work one is big.

Conclusion

The Express.js and Mongoskin libraries are great when you need to build a simple REST API server in a few line of code. Later, if you need to expand the libraries they also provide a way to configure and organize your code.

NoSQL databases like MongoDB are good at free-REST APIs where we don’t have to define schemas and can throw any data and it’ll be saved.

The full code of both test and app files: https://gist.github.com/azat-co/6075685.

If you like to learn more about Express.js and other JavaScript libraries take a look at the series Intro to Express.js tutorials.

Note: In this example I’m using semi-colon less style. Semi-colons in JavaScript are absolutely optional except in two cases: in the for loop and before expression/statement that starts with parenthesis (e.g., Immediately-Invoked Function Expression).