Tutorial: Node.js and MongoDB JSON REST API server with Mongoskin and Express.js

Tutorial: building a JSON REST API server with Node.js and MongoDB using Mongoskin and Express.js utilizing Mocha and Super Agent for BDD/TDD.

Update3: Expess 4 version of this tutorial is available at Express.js 4, Node.js and MongoDB REST API Tutorial, and github.com/azat-co/rest-api-express (master branch). This tutorial will work with Express 3.x.

Update2: “Mongoskin removed ‘db.collection.id’ and added some actionById methods” from this pull request with this code changes. To use the code in this post, just install an older version of Mongoskin (0.5.0?). The code in the GitHub will work with Mongoskin 1.3.20.

Update2: “Mongoskin removed ‘db.collection.id’ and added some actionById methods” from this pull request with this code changes. To use the code in this post, just install an older version of Mongoskin (0.5.0?)

Update: use the enhanced code from this repository github.com/azat-co/rest-api-express (express3 branch).

Note: This text is a part of Express.js Guide: The Comprehensive Book on Express.js.

This tutorial will walk you through writing test using the Mocha and Super Agent libraries and then use them in a test-driven development manner to build a Node.js free JSON REST API server utilizing Express.js framework and Mongoskin library for MongoDB. In this REST API server, we’ll perform create, read, update and delete (CRUD) operations and harness Express.js middleware concept with app.param() and app.use() methods.

Test Coverage

Before anything else let’s write functional tests that make HTTP requests to our soon-to-be-created REST API server. If you know how to use Mocha or just want to jump straight to the Express.js app implementation feel free to do so. You can use CURL terminal commands for testing too.

Assuming we already have Node.js, NPM and MongoDB installed, let’s create a new folder (or if you wrote the tests use that folder):

mkdir rest-api
cd rest-api

We’ll use Mocha, Expect.js and Super Agent libraries. To install them run these command from the project folder:

$ npm install mocha@1.18.2 --save-dev
$ npm install expect.js@0.3.1 --save-dev 
$ npm install superagent@0.17.0 --save-dev

Now let’s create express.test.js file in the same folder which will have six suites:

  • Creating a new object
  • Retrieving an object by its ID
  • Retrieving the whole collection
  • Updating an object by its ID
  • Checking an updated object by its ID
  • Removing an object by its ID

HTTP requests are just a breeze with Super Agent’s chained functions which we’ll put inside of each test suite. Here is the full source code for the express.test.js file:

var superagent = require('superagent')
var expect = require('expect.js')

describe('express rest api server', function(){
  var id

  it('post object', function(done){
    superagent.post('http://localhost:3000/collections/test')
      .send({ name: 'John'
        , email: 'john@rpjs.co'
      })
      .end(function(e,res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(res.body.length).to.eql(1)
        expect(res.body[0]._id.length).to.eql(24)
        id = res.body[0]._id
        done()
      })    
  })

  it('retrieves an object', function(done){
    superagent.get('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body._id.length).to.eql(24)        
        expect(res.body._id).to.eql(id)        
        done()
      })
  })

  it('retrieves a collection', function(done){
    superagent.get('http://localhost:3000/collections/test')
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(res.body.length).to.be.above(0)
        expect(res.body.map(function (item){return item._id})).to.contain(id)        
        done()
      })
  })

  it('updates an object', function(done){
    superagent.put('http://localhost:3000/collections/test/'+id)
      .send({name: 'Peter'
        , email: 'peter@yahoo.com'})
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body.msg).to.eql('success')        
        done()
      })
  })

  it('checks an updated object', function(done){
    superagent.get('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body._id.length).to.eql(24)        
        expect(res.body._id).to.eql(id)        
        expect(res.body.name).to.eql('Peter')        
        done()
      })
  })    
  it('removes an object', function(done){
    superagent.del('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body.msg).to.eql('success')    
        done()
      })
  })      
})

To run the tests we can use the $ mocha express.test.js command.

Dependencies

In this tutorial we’ll utilize Mongoskin, a MongoDB library which is a better alternative to the plain good old native MongoDB driver for Node.js. In additition Mongoskin is more light-weight than Mongoose and schema-less. For more insight please check out Mongoskin comparison blurb.

Express.js is a wrapper for the core Node.js HTTP module objects. The Express.js framework is build on top of Connect middleware and provided tons of convenience. Some people compare the framework to Ruby’s Sinatra in terms of how it’s non-opinionated and configurable.

If you’ve create a rest-api folder in the previous section Test Coverage, simply run these commands to install modules for the application:

npm install express@3.5.1 --save
npm install mongoskin@1.3.20 --save

Implementation

First things first, so let’s define our dependencies:

var express = require('express')
  , mongoskin = require('mongoskin')

After the version 3.x, Express streamlines the instantiation of its app instance, in a way that this line will give us a server object:

var app = express()

To extract params from the body of the requests we’ll use bodyParser() middleware which looks more like a configuration statement:

app.use(express.bodyParser())

Middleware (in this and other forms) is a powerful and convenient pattern in Express.js and Connect to organize and re-use code.

As with the bodyParser() method that saves us from the hurdles of parsing a body object of HTTP request, Mongoskin makes possible to connect to the MongoDB database in one effortless line of code:

var db = mongoskin.db('localhost:27017/test', {safe:true});

Note: If you wish to connect to a remote database, e.g., MongoHQ instance, substitute the string with your username, password, host and port values. Here is the format of the URI string: mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]

The app.param() method is another Express.js middleware. It basically says “do something every time there is this value in the URL pattern of the request handler”. In our case we select a particular collection when request pattern contains a sting collectionName prefixed with a colon (you’ll see it later in the routes):

app.param('collectionName', function(req, res, next, collectionName){
  req.collection = db.collection(collectionName)
  return next()
})

Merely to be user-friendly let’s put a root route with a message:

app.get('/', function(req, res) {
  res.send('please select a collection, e.g., /collections/messages')
})

Now the real work begins, here is how we retrieve a list of items sorted by _id and which has a limit of 10:

app.get('/collections/:collectionName', function(req, res) {
  req.collection.find({},{limit:10, sort: [['_id',-1]]}).toArray(function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

Have you noticed a :collectionName string in the URL pattern parameter? This and the previous app.param() middleware is what gives us the req.collection object which points to a specified collection in our database.

The object creating endpoint is slightly easier to grasp since we just pass the whole payload to the MongoDB (method a.k.a. free JSON REST API):

app.post('/collections/:collectionName', function(req, res) {
  req.collection.insert(req.body, {}, function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

Single object retrieval functions are faster than find(), but they use different interface (they return object directly instead of a cursor), so please be aware of that. In addition, we’re extracting the ID from :id part of the path with req.params.id Express.js magic:

app.get('/collections/:collectionName/:id', function(req, res) {
  req.collection.findOne({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send(result)
  })
})

PUT request handler gets more interesting because update() doesn’t return the augmented object, instead it returns us a count of affected objects.

Also {$set:req.body} is a special MongoDB operator (operators tend to start with a dollar sign) that sets values.

The second {safe:true, multi:false} parameter is an object with options that tell MongoDB to wait for the execution before running the callback function and to process only one (first) item.

app.put('/collections/:collectionName/:id', function(req, res) {
  req.collection.update({_id: req.collection.id(req.params.id)}, {$set:req.body}, {safe:true, multi:false}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

Finally, the DELETE method which also output a custom JSON message:

app.del('/collections/:collectionName/:id', function(req, res) {
  req.collection.remove({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

Note: The delete is an operator in JavaScript, so Express.js uses app.del instead.

The last line that actually starts the server on port 3000 in this case:

app.listen(3000)

Just in case something is not working quite well here is the full code of express.js file:

var express = require('express')
  , mongoskin = require('mongoskin')

var app = express()
app.use(express.bodyParser())

var db = mongoskin.db('localhost:27017/test', {safe:true});

app.param('collectionName', function(req, res, next, collectionName){
  req.collection = db.collection(collectionName)
  return next()
})
app.get('/', function(req, res) {
  res.send('please select a collection, e.g., /collections/messages')
})

app.get('/collections/:collectionName', function(req, res) {
  req.collection.find({},{limit:10, sort: [['_id',-1]]}).toArray(function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

app.post('/collections/:collectionName', function(req, res) {
  req.collection.insert(req.body, {}, function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

app.get('/collections/:collectionName/:id', function(req, res) {
  req.collection.findOne({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send(result)
  })
})
app.put('/collections/:collectionName/:id', function(req, res) {
  req.collection.update({_id: req.collection.id(req.params.id)}, {$set:req.body}, {safe:true, multi:false}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})
app.del('/collections/:collectionName/:id', function(req, res) {
  req.collection.remove({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

app.listen(3000)

Exit your editor and run this in your terminal:

$ node express.js

And in a different window (without closing the first one):

$ mocha express.test.js

If you really don’t like Mocha and/or BDD, CURL is always there for you. :-)

For example, CURL data to make a POST request:

$ curl -d "" http://localhost:3000

GET requests also work in the browser, for example http://localhost:3000/test.

In this tutorial our tests are longer than the app code itself so abandoning test-driven development might be tempting, but believe me the good habits of TDD will save you hours and hours during any serious development when the complexity of the applications you work one is big.

Conclusion

The Express.js and Mongoskin libraries are great when you need to build a simple REST API server in a few line of code. Later, if you need to expand the libraries they also provide a way to configure and organize your code.

NoSQL databases like MongoDB are good at free-REST APIs where we don’t have to define schemas and can throw any data and it’ll be saved.

The full code of both test and app files: https://gist.github.com/azat-co/6075685.

If you like to learn more about Express.js and other JavaScript libraries take a look at the series Intro to Express.js tutorials.

Note: In this example I’m using semi-colon less style. Semi-colons in JavaScript are absolutely optional except in two cases: in the for loop and before expression/statement that starts with parenthesis (e.g., Immediately-Invoked Function Expression).

Author: Azat

Techies, entrepreneur, 20+ years in tech/IT/software/web development expert: NodeJS, JavaScript, MongoDB, Ruby on Rails, PHP, SQL, HTML, CSS. 500 Startups (batch Fall 2011) alumnus. http://azat.co http://github.com/azat-co

37 thoughts on “Tutorial: Node.js and MongoDB JSON REST API server with Mongoskin and Express.js”

  1. hmm, very nice post. But where are you showing how you’er test driving the real code? Here you only show the BDD integration tests but where is the follow-through with each BDD test to the multiple TDD tests that implemented that code that made your BDD tests pass? Because I interpret this is that you’re not TDD’ing your app which is what BDD + TDD is all about, they go hand-in-hand and so when you don’t show the TDD tests with asserts that forced you to create the real production code I have to assume you added tests after.

  2. Pingback: Ra Puke Moana
  3. Thanks for the tutorial. I am getting the following errors consistently. Any ideas?

    TypeError: Object # has no method ‘id’
    at /Users/jonathan/Projects/rest-api/express.js:34:47
    at callbacks (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:164:37)
    at param (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:138:11)
    at param (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:135:11)
    at paramCallback (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:150:30)
    at /Users/jonathan/Projects/rest-api/express.js:11:10
    at paramCallback (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:151:7)
    at param (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:133:11)
    at pass (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:145:5)
    at Router._dispatch (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:173:5)

  4. Thanks for the mocka and superagent intruduction, I appreciate it. Isn’t there a way to make the tests independent? I mean, if the first test fails then the second one will fail as well as the id variable will be undefined.

  5. 1. JSON is a data format. It stands for JavaScript Object Notation
    2. None, Java and JavaScript is like ham and hamster
    3. This blog and Rapid Prototyping with JS book, Quora, StackOverflow and official JSON docs
    4. JSON is not a database so there’s no query language. MongoDB is a database, but it’s a NoSQL database, so there’s no need for SQL at all!

    You’re welcome!

  6. Hi Azat,

    Basically i am completely new to java and iam tryin to learn mongodb which has very min requirements i guess. But anyways what i want is
    1. what is JSON?
    2. How much of Java should i be knowing to work with these JSON docs?
    3. are there any articles which i can read in my free time to gain knowledge ?
    4. if incase there is a query language how can i map it with SQL so i could have better understanding>
    Any kind of Info or help would very much Appreciated thanks in advance.

  7. Hi Donald,

    You can use something like this:


    app.get('/collections/:collectionName/:key/:value', function(req, res) {
    var query = {}
    query[req.params.key] = req.params.value
    req.collection.findOne(query, function(e, result){
    if (e) return next(e)
    res.send(result)
    })
    })

    Or assuming collectionName is provided and we have collections in req already:


    app.get('/collections/:collectionName/:key/:value', function(req, res) {
    var query = {}
    query[req.params.key] = req.params.value
    req.collections[req.params.collectionName].findOne(query, function(e, result){
    if (e) return next(e)
    res.send(result)
    })
    })

  8. This moght be begineer question, but how do I search MonogDB using different parameters?

    Say I have a field called name, and I want to search by that name. Something like this:
    eventCollection.findOne({“name”:eventName},
    I use it currently and it works

    How can I modify your example which is this:
    app.get(‘/collections/:collectionName/:id’, function(req, res) {
    req.collection.findOne({_id: req.collection.id(req.params.id)},
    function(e, result){
    if (e) return next(e)
    res.send(result)
    })
    })

    So it supports search by custom fields :)

    Thank you

  9. Hi Azat,

    great post and a nice introduction for building a RESTful API- thanks.

    Actually I think “next” is missing in each route callback. If not set, you get an


    ReferenceError: next is not defined

    in case of an error. Or am I missing something?

    Cheers

    Andy

  10. Hi Chris,

    Great comments, thank you for the suggestions — definitely ‘-g’ or ‘./node_modules/mocha/bin/mocha’ which could be better sometime because of the specificity of the version from package.json. :-)

  11. Three comments:
    1. “$ npm install mocha” should read “$ npm install -g mocha” to install mocha globally (necessary for the mocha executable to be exposed)

    2. It would be helpful to note that installing and running mongo is a prerequisite. (I’m sure this seems very obvious to you, but the rest of your tutorial is sufficiently detailed and step-by-step that this feels more like an omission.)
    http://docs.mongodb.org/manual/tutorial/install-mongodb-on-os-x/ may be helpful for OSX users, for example.

    3. Otherwise, great tutorial,! I might not have bothered with Mocha in my current project if not for your efforts here.

    Thank you!

    Sincerely,
    cweekly

  12. Great Tutorial,

    did’t know about mongoskin always used mongoose.

    Just a quick tip :

    replace the {} in

    app.post('/collections/:collectionName', function(req, res) {
    req.collection.insert(req.body, {}, function(e, results){
    if (e) return next(e)
    res.send(results)
    })
    })

    with req.query then you can filter the query f.e.

    collection/users?name=Jon

  13. I went back and cleaned some stuff up (N00B error on the DB connection error refused issue)…

    But now I am still getting the following error:

    Uncaught Error: expected 1 to be above 1

    at Assertion.assert (/…/…/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:99:13)
    at Assertion.greaterThan.Assertion.above (/Users/zezo/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:279:10)
    at Function.scope (/…/…/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:480:17)
    at superagent.put.send.name (/…/…/Desktop/node_dev/rest-api/express.test.js:39:39)
    at Request.callback (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:628:30)
    at Request. (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:131:10)
    at Request.EventEmitter.emit (events.js:96:17)
    at IncomingMessage.Request.end (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:773:12)
    at IncomingMessage.EventEmitter.emit (events.js:126:20)
    at IncomingMessage._emitEnd (http.js:366:10)
    at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:149:23)
    at Socket.socketOnData (http.js:1367:20)
    at TCP.onread (net.js:403:27)

  14. When I run the express.test.js, I am getting “.double callback!” alerts and all tests failed…

    I went back and used your code from the gist and still getting same error. Any ideas as to why?

    Here’s some of what the error looks like:

    ․double callback!
    ․double callback!
    ․double callback!
    ․double callback!
    ․double callback!
    ․double callback!

    0 passing (28ms)
    6 failing

    1) express rest api server post object:
    Uncaught Error: expected { code: ‘ECONNREFUSED’,
    errno: ‘ECONNREFUSED’,
    syscall: ‘connect’ } to sort of equal null
    at Assertion.assert (/…/…/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:99:13)
    at Assertion.eql (/…/…/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:214:10)
    at superagent.put.send.name (/…/…/Desktop/node_dev/rest-api/express.test.js:14:22)
    at Request.callback (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:628:30)
    at ClientRequest. (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:596:10)
    at ClientRequest.EventEmitter.emit (events.js:96:17)
    at Socket.socketErrorListener (http.js:1331:9)
    at Socket.EventEmitter.emit (events.js:96:17)
    at Socket._destroy.self.errorEmitted (net.js:328:14)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)

  15. You should put “return” when there is an error, otherwise express will try to send an error and a response, and won’t be happy (OR using “else”).
    if (e) return next(e);
    res.send(results);

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.