Tutorial: Node.js and MongoDB JSON REST API server with Mongoskin and Express.js

Update3: Expess 4 version of this tutorial is available at Express.js 4, Node.js and MongoDB REST API Tutorial, and github.com/azat-co/rest-api-express (master branch). This tutorial will work with Express 3.x.

Update2: “Mongoskin removed ‘db.collection.id’ and added some actionById methods” from this pull request with this code changes. To use the code in this post, just install an older version of Mongoskin (0.5.0?). The code in the GitHub will work with Mongoskin 1.3.20.

Update2: “Mongoskin removed ‘db.collection.id’ and added some actionById methods” from this pull request with this code changes. To use the code in this post, just install an older version of Mongoskin (0.5.0?)

Update: use the enhanced code from this repository github.com/azat-co/rest-api-express (express3 branch).

Note: This text is a part of Express.js Guide: The Comprehensive Book on Express.js.

This tutorial will walk you through writing test using the Mocha and Super Agent libraries and then use them in a test-driven development manner to build a Node.js free JSON REST API server utilizing Express.js framework and Mongoskin library for MongoDB. In this REST API server, we’ll perform create, read, update and delete (CRUD) operations and harness Express.js middleware concept with app.param() and app.use() methods.

Test Coverage

Before anything else let’s write functional tests that make HTTP requests to our soon-to-be-created REST API server. If you know how to use Mocha or just want to jump straight to the Express.js app implementation feel free to do so. You can use CURL terminal commands for testing too.

Assuming we already have Node.js, NPM and MongoDB installed, let’s create a new folder (or if you wrote the tests use that folder):

mkdir rest-api
cd rest-api

We’ll use Mocha, Expect.js and Super Agent libraries. To install them run these command from the project folder:

$ npm install mocha@1.18.2 --save-dev
$ npm install expect.js@0.3.1 --save-dev 
$ npm install superagent@0.17.0 --save-dev

Now let’s create express.test.js file in the same folder which will have six suites:

  • Creating a new object
  • Retrieving an object by its ID
  • Retrieving the whole collection
  • Updating an object by its ID
  • Checking an updated object by its ID
  • Removing an object by its ID

HTTP requests are just a breeze with Super Agent’s chained functions which we’ll put inside of each test suite. Here is the full source code for the express.test.js file:

var superagent = require('superagent')
var expect = require('expect.js')

describe('express rest api server', function(){
  var id

  it('post object', function(done){
    superagent.post('http://localhost:3000/collections/test')
      .send({ name: 'John'
        , email: 'john@rpjs.co'
      })
      .end(function(e,res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(res.body.length).to.eql(1)
        expect(res.body[0]._id.length).to.eql(24)
        id = res.body[0]._id
        done()
      })    
  })

  it('retrieves an object', function(done){
    superagent.get('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body._id.length).to.eql(24)        
        expect(res.body._id).to.eql(id)        
        done()
      })
  })

  it('retrieves a collection', function(done){
    superagent.get('http://localhost:3000/collections/test')
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(res.body.length).to.be.above(0)
        expect(res.body.map(function (item){return item._id})).to.contain(id)        
        done()
      })
  })

  it('updates an object', function(done){
    superagent.put('http://localhost:3000/collections/test/'+id)
      .send({name: 'Peter'
        , email: 'peter@yahoo.com'})
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body.msg).to.eql('success')        
        done()
      })
  })

  it('checks an updated object', function(done){
    superagent.get('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body._id.length).to.eql(24)        
        expect(res.body._id).to.eql(id)        
        expect(res.body.name).to.eql('Peter')        
        done()
      })
  })    
  it('removes an object', function(done){
    superagent.del('http://localhost:3000/collections/test/'+id)
      .end(function(e, res){
        // console.log(res.body)
        expect(e).to.eql(null)
        expect(typeof res.body).to.eql('object')
        expect(res.body.msg).to.eql('success')    
        done()
      })
  })      
})

To run the tests we can use the $ mocha express.test.js command.

Dependencies

In this tutorial we’ll utilize Mongoskin, a MongoDB library which is a better alternative to the plain good old native MongoDB driver for Node.js. In additition Mongoskin is more light-weight than Mongoose and schema-less. For more insight please check out Mongoskin comparison blurb.

Express.js is a wrapper for the core Node.js HTTP module objects. The Express.js framework is build on top of Connect middleware and provided tons of convenience. Some people compare the framework to Ruby’s Sinatra in terms of how it’s non-opinionated and configurable.

If you’ve create a rest-api folder in the previous section Test Coverage, simply run these commands to install modules for the application:

npm install express@3.5.1 --save
npm install mongoskin@1.3.20 --save

Implementation

First things first, so let’s define our dependencies:

var express = require('express')
  , mongoskin = require('mongoskin')

After the version 3.x, Express streamlines the instantiation of its app instance, in a way that this line will give us a server object:

var app = express()

To extract params from the body of the requests we’ll use bodyParser() middleware which looks more like a configuration statement:

app.use(express.bodyParser())

Middleware (in this and other forms) is a powerful and convenient pattern in Express.js and Connect to organize and re-use code.

As with the bodyParser() method that saves us from the hurdles of parsing a body object of HTTP request, Mongoskin makes possible to connect to the MongoDB database in one effortless line of code:

var db = mongoskin.db('localhost:27017/test', {safe:true});

Note: If you wish to connect to a remote database, e.g., MongoHQ instance, substitute the string with your username, password, host and port values. Here is the format of the URI string: mongodb://[username:password@]host1[:port1][,host2[:port2],...[,hostN[:portN]]][/[database][?options]]

The app.param() method is another Express.js middleware. It basically says “do something every time there is this value in the URL pattern of the request handler”. In our case we select a particular collection when request pattern contains a sting collectionName prefixed with a colon (you’ll see it later in the routes):

app.param('collectionName', function(req, res, next, collectionName){
  req.collection = db.collection(collectionName)
  return next()
})

Merely to be user-friendly let’s put a root route with a message:

app.get('/', function(req, res) {
  res.send('please select a collection, e.g., /collections/messages')
})

Now the real work begins, here is how we retrieve a list of items sorted by _id and which has a limit of 10:

app.get('/collections/:collectionName', function(req, res) {
  req.collection.find({},{limit:10, sort: [['_id',-1]]}).toArray(function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

Have you noticed a :collectionName string in the URL pattern parameter? This and the previous app.param() middleware is what gives us the req.collection object which points to a specified collection in our database.

The object creating endpoint is slightly easier to grasp since we just pass the whole payload to the MongoDB (method a.k.a. free JSON REST API):

app.post('/collections/:collectionName', function(req, res) {
  req.collection.insert(req.body, {}, function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

Single object retrieval functions are faster than find(), but they use different interface (they return object directly instead of a cursor), so please be aware of that. In addition, we’re extracting the ID from :id part of the path with req.params.id Express.js magic:

app.get('/collections/:collectionName/:id', function(req, res) {
  req.collection.findOne({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send(result)
  })
})

PUT request handler gets more interesting because update() doesn’t return the augmented object, instead it returns us a count of affected objects.

Also {$set:req.body} is a special MongoDB operator (operators tend to start with a dollar sign) that sets values.

The second {safe:true, multi:false} parameter is an object with options that tell MongoDB to wait for the execution before running the callback function and to process only one (first) item.

app.put('/collections/:collectionName/:id', function(req, res) {
  req.collection.update({_id: req.collection.id(req.params.id)}, {$set:req.body}, {safe:true, multi:false}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

Finally, the DELETE method which also output a custom JSON message:

app.del('/collections/:collectionName/:id', function(req, res) {
  req.collection.remove({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

Note: The delete is an operator in JavaScript, so Express.js uses app.del instead.

The last line that actually starts the server on port 3000 in this case:

app.listen(3000)

Just in case something is not working quite well here is the full code of express.js file:

var express = require('express')
  , mongoskin = require('mongoskin')

var app = express()
app.use(express.bodyParser())

var db = mongoskin.db('localhost:27017/test', {safe:true});

app.param('collectionName', function(req, res, next, collectionName){
  req.collection = db.collection(collectionName)
  return next()
})
app.get('/', function(req, res) {
  res.send('please select a collection, e.g., /collections/messages')
})

app.get('/collections/:collectionName', function(req, res) {
  req.collection.find({},{limit:10, sort: [['_id',-1]]}).toArray(function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

app.post('/collections/:collectionName', function(req, res) {
  req.collection.insert(req.body, {}, function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

app.get('/collections/:collectionName/:id', function(req, res) {
  req.collection.findOne({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send(result)
  })
})
app.put('/collections/:collectionName/:id', function(req, res) {
  req.collection.update({_id: req.collection.id(req.params.id)}, {$set:req.body}, {safe:true, multi:false}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})
app.del('/collections/:collectionName/:id', function(req, res) {
  req.collection.remove({_id: req.collection.id(req.params.id)}, function(e, result){
    if (e) return next(e)
    res.send((result===1)?{msg:'success'}:{msg:'error'})
  })
})

app.listen(3000)

Exit your editor and run this in your terminal:

$ node express.js

And in a different window (without closing the first one):

$ mocha express.test.js

If you really don’t like Mocha and/or BDD, CURL is always there for you. :-)

For example, CURL data to make a POST request:

$ curl -d "" http://localhost:3000

GET requests also work in the browser, for example http://localhost:3000/test.

In this tutorial our tests are longer than the app code itself so abandoning test-driven development might be tempting, but believe me the good habits of TDD will save you hours and hours during any serious development when the complexity of the applications you work one is big.

Conclusion

The Express.js and Mongoskin libraries are great when you need to build a simple REST API server in a few line of code. Later, if you need to expand the libraries they also provide a way to configure and organize your code.

NoSQL databases like MongoDB are good at free-REST APIs where we don’t have to define schemas and can throw any data and it’ll be saved.

The full code of both test and app files: https://gist.github.com/azat-co/6075685.

If you like to learn more about Express.js and other JavaScript libraries take a look at the series Intro to Express.js tutorials.

Note: In this example I’m using semi-colon less style. Semi-colons in JavaScript are absolutely optional except in two cases: in the for loop and before expression/statement that starts with parenthesis (e.g., Immediately-Invoked Function Expression).

31 thoughts on “Tutorial: Node.js and MongoDB JSON REST API server with Mongoskin and Express.js

  1. Pingback: Ra Puke Moana

  2. Pingback: Erlang vs Node.js |

  3. Pingback: Node.js and Mobile App Development News Round-up – July 30, 2013 | Eclectic Consulting

  4. Pingback: Express.js 4, Node.js and MongoDB REST API Tutorial

  5. Jonathan

    Thanks for the tutorial. I am getting the following errors consistently. Any ideas?

    TypeError: Object # has no method ‘id’
    at /Users/jonathan/Projects/rest-api/express.js:34:47
    at callbacks (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:164:37)
    at param (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:138:11)
    at param (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:135:11)
    at paramCallback (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:150:30)
    at /Users/jonathan/Projects/rest-api/express.js:11:10
    at paramCallback (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:151:7)
    at param (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:133:11)
    at pass (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:145:5)
    at Router._dispatch (/Users/jonathan/Projects/rest-api/node_modules/express/lib/router/index.js:173:5)

  6. Nicolás Garnil

    Thanks for the mocka and superagent intruduction, I appreciate it. Isn’t there a way to make the tests independent? I mean, if the first test fails then the second one will fail as well as the id variable will be undefined.

  7. Pingback: Read List | Daisy & Joel

  8. Pingback: How to start with node.js : PHP, JavaScript, C++, JAVA, Programmer, software developer

  9. Azat Post author

    1. JSON is a data format. It stands for JavaScript Object Notation
    2. None, Java and JavaScript is like ham and hamster
    3. This blog and Rapid Prototyping with JS book, Quora, StackOverflow and official JSON docs
    4. JSON is not a database so there’s no query language. MongoDB is a database, but it’s a NoSQL database, so there’s no need for SQL at all!

    You’re welcome!

  10. Pingback: Bad Request error with Express.JS | Work'n'Me

  11. nikhil

    Hi Azat,

    Basically i am completely new to java and iam tryin to learn mongodb which has very min requirements i guess. But anyways what i want is
    1. what is JSON?
    2. How much of Java should i be knowing to work with these JSON docs?
    3. are there any articles which i can read in my free time to gain knowledge ?
    4. if incase there is a query language how can i map it with SQL so i could have better understanding>
    Any kind of Info or help would very much Appreciated thanks in advance.

  12. Pingback: JavaScript Allongé and Rapid Prototyping with JS | webapplog

  13. Azat Post author

    Hi Donald,

    You can use something like this:


    app.get('/collections/:collectionName/:key/:value', function(req, res) {
    var query = {}
    query[req.params.key] = req.params.value
    req.collection.findOne(query, function(e, result){
    if (e) return next(e)
    res.send(result)
    })
    })

    Or assuming collectionName is provided and we have collections in req already:


    app.get('/collections/:collectionName/:key/:value', function(req, res) {
    var query = {}
    query[req.params.key] = req.params.value
    req.collections[req.params.collectionName].findOne(query, function(e, result){
    if (e) return next(e)
    res.send(result)
    })
    })

  14. Pingback: Express.js Fundamentals | Flippin' Awesome

  15. DonaldM

    This moght be begineer question, but how do I search MonogDB using different parameters?

    Say I have a field called name, and I want to search by that name. Something like this:
    eventCollection.findOne({“name”:eventName},
    I use it currently and it works

    How can I modify your example which is this:
    app.get(‘/collections/:collectionName/:id’, function(req, res) {
    req.collection.findOne({_id: req.collection.id(req.params.id)},
    function(e, result){
    if (e) return next(e)
    res.send(result)
    })
    })

    So it supports search by custom fields :)

    Thank you

  16. Pingback: Express.js FUNdamentalswebapplog | webapplog

  17. Andy Wenk

    Hi Azat,

    great post and a nice introduction for building a RESTful API- thanks.

    Actually I think “next” is missing in each route callback. If not set, you get an


    ReferenceError: next is not defined

    in case of an error. Or am I missing something?

    Cheers

    Andy

  18. Azat Post author

    Hi Chris,

    Great comments, thank you for the suggestions — definitely ‘-g’ or ‘./node_modules/mocha/bin/mocha’ which could be better sometime because of the specificity of the version from package.json. :-)

  19. Chris Weekly

    Three comments:
    1. “$ npm install mocha” should read “$ npm install -g mocha” to install mocha globally (necessary for the mocha executable to be exposed)

    2. It would be helpful to note that installing and running mongo is a prerequisite. (I’m sure this seems very obvious to you, but the rest of your tutorial is sufficiently detailed and step-by-step that this feels more like an omission.)
    http://docs.mongodb.org/manual/tutorial/install-mongodb-on-os-x/ may be helpful for OSX users, for example.

    3. Otherwise, great tutorial,! I might not have bothered with Mocha in my current project if not for your efforts here.

    Thank you!

    Sincerely,
    cweekly

  20. Marco

    Great Tutorial,

    did’t know about mongoskin always used mongoose.

    Just a quick tip :

    replace the {} in

    app.post('/collections/:collectionName', function(req, res) {
    req.collection.insert(req.body, {}, function(e, results){
    if (e) return next(e)
    res.send(results)
    })
    })

    with req.query then you can filter the query f.e.

    collection/users?name=Jon

  21. mediabewm

    I went back and cleaned some stuff up (N00B error on the DB connection error refused issue)…

    But now I am still getting the following error:

    Uncaught Error: expected 1 to be above 1

    at Assertion.assert (/…/…/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:99:13)
    at Assertion.greaterThan.Assertion.above (/Users/zezo/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:279:10)
    at Function.scope (/…/…/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:480:17)
    at superagent.put.send.name (/…/…/Desktop/node_dev/rest-api/express.test.js:39:39)
    at Request.callback (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:628:30)
    at Request. (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:131:10)
    at Request.EventEmitter.emit (events.js:96:17)
    at IncomingMessage.Request.end (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:773:12)
    at IncomingMessage.EventEmitter.emit (events.js:126:20)
    at IncomingMessage._emitEnd (http.js:366:10)
    at HTTPParser.parserOnMessageComplete [as onMessageComplete] (http.js:149:23)
    at Socket.socketOnData (http.js:1367:20)
    at TCP.onread (net.js:403:27)

  22. mediabewm

    When I run the express.test.js, I am getting “.double callback!” alerts and all tests failed…

    I went back and used your code from the gist and still getting same error. Any ideas as to why?

    Here’s some of what the error looks like:

    ․double callback!
    ․double callback!
    ․double callback!
    ․double callback!
    ․double callback!
    ․double callback!

    0 passing (28ms)
    6 failing

    1) express rest api server post object:
    Uncaught Error: expected { code: ‘ECONNREFUSED’,
    errno: ‘ECONNREFUSED’,
    syscall: ‘connect’ } to sort of equal null
    at Assertion.assert (/…/…/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:99:13)
    at Assertion.eql (/…/…/Desktop/node_dev/rest-api/node_modules/expect.js/expect.js:214:10)
    at superagent.put.send.name (/…/…/Desktop/node_dev/rest-api/express.test.js:14:22)
    at Request.callback (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:628:30)
    at ClientRequest. (/…/…/Desktop/node_dev/rest-api/node_modules/superagent/lib/node/index.js:596:10)
    at ClientRequest.EventEmitter.emit (events.js:96:17)
    at Socket.socketErrorListener (http.js:1331:9)
    at Socket.EventEmitter.emit (events.js:96:17)
    at Socket._destroy.self.errorEmitted (net.js:328:14)
    at process.startup.processNextTick.process._tickCallback (node.js:244:9)

  23. Mod

    You should put “return” when there is an error, otherwise express will try to send an error and a response, and won’t be happy (OR using “else”).
    if (e) return next(e);
    res.send(results);

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>