Full Stack JavaScript

My new book Full Stack JavaScript (my 4th traditionally-published book) comes with a series of screencast videos for better immersion in a wonderful and mesmerizing world of Node.js, Backbone and MongoDB. It’s a one thing to read through the text and another to follow up with dynamic videos which walk you through the book’s projects.

Full Stack JavaScript
Full Stack JavaScript

The videos and the source code are open source, meaning they are publicly available. Therefore, you don’t have to buy a book—you can just watch the 14 videos on YouTube (playlist) and go through the code on GitHub (repository).

Continue reading “Full Stack JavaScript”

Time to Up Your Node.js Skills

Are you a software engineer who wants to master an effective and fun-to-use, new platform Node.js? Is your goal to stay on top of the curve with cutting-edge technology, and to stay ahead of the competitive software engineering industry by delivering high quality products faster?

Continue reading “Time to Up Your Node.js Skills”

MongoUI: Real-Time MongoDB Admin Web Interface (a la phpMyAdmin in Node.js)

MongoUI is a real-time web interface for Node.js and MongoDB written with DerbyJS—a real-time full-stack web framework. It’s a app that can be run locally and on the server. Similarly, you can view and manipulate data in a local or remote database. The MongoUI project is in beta so use caution, and please contribute your feedback.

MongoUI on NPM: https://www.npmjs.org/package/mongoui

MongoUI on GitHub: https://github.com/azat-co/mongoui

MongoUI description (this page): http://webapplog.com/mongoui

MongoUI features include:

  • Switch databases and collections
  • Search by field value (string, number, ObjectId)
  • Save search / filter results as a bookmark (each URL has a query)
  • Edit any fields’ values in a real-time editor
  • Get raw JSON objects

Here’s a one-and-a-half-minute video that shows filtering, editing, and switching collections:

Direct link to the YouTube video: http://youtu.be/l8Rfpow0f9A.

Continue reading “MongoUI: Real-Time MongoDB Admin Web Interface (a la phpMyAdmin in Node.js)”

Practical Node.js TOC

As the Apress team of technical reviewers and copy editors and I make progress on the Practical Node.js manuscript, the date of the publication approaches fast. Last time I checked it was June 2014.

Practical Node.js: Building Real-world Scalable Web Apps
Practical Node.js: Building Real-world Scalable Web Apps

Many people ask me: how is the process compared to self-publishing? Is it worth the hassle?

So far, I can say only good things about my editors and the process of traditional publishing itself. I’m impressed about so many things I’ve already learned about structuring and technical writing. I feel like it enormously improved my style. There is more on this in my new meta book&resource ProgWriter.

As a sneak peek, here’s the tentative Table of Contents for the Practical Node.js book:

  1. Setting up Node.js and Other Essentials
  2. Using Express.js to Create Node.js Web Apps
  3. TDD and BDD for Node.js with Mocha
  4. Template Engines: Jade and Handlebars
  5. Persistence with MongoDB and Mongoskin
  6. Using Sessions and OAuth to Authorize and Authenticate Users in Node.js Apps
  7. Boosting Your Node.js Data with the Mongoose ORM Library
  8. Building Node.js REST API Servers with Express.js and Hapi
  9. Real-time Apps with WebSockets, Socket.IO and DerbyJS
  10. Getting Node.js Apps Production Ready
  11. Deploying Node.js Apps
  12. Publishing Node.js Modules and Contributing to Open Source

The good thing is that people who want to get the book first don’t have to wait ’til the book is released. They can pre-order the book on Amazon, or even better get access to the alpha version at Apress!

The alpha version will be release chapter by chapter starting in the next few weeks!

Express.js 4, Node.js and MongoDB REST API Tutorial

Usually good things don’t stay the same, so our tutorial on building a JSON REST API server with Node.js and MongoDB using Mongoskin and Express.js, and testing it with Mocha and Superagent, has became a bit outdated with the new Express.js 4 version release. Here’s a brand new, revisited tutorial for Express.js 4, Node.js and MongoDB (Mongoskin) free-JSON RESTful API server.

The code for this new tutorial is available at github.com/azat-co/rest-api-express (master branch). The old tutorial’s code for Express 3.x, is still working and in the express3 branch.

Express.js 4 and MongoDB REST API Tutorial consists of these parts:

  1. Node.js and MongoDB REST API Overview
  2. REST API Tests with Mocha and Superagent
  3. NPM-ing Node.js Server Dependencies
  4. Express.js 4.x Middleware Caveat
  5. Express.js and MongoDB (Mongoskin) Implementation
  6. Running The Express.js 4 App and Testing MongoDB Data with Mocha
  7. Conclusion and Further Express.js and Node.js Reading

Instead of TL;DR:

If you’re only interested in a working code from the repository and know what to do, here are brief instructions on how to download and run the REST API server:

$ git clone git@github.com:azat-co/rest-api-express.git
$ npm install
$ node express.js

Start MongoDB with $ mongod. Then, in a new terminal window run the Mocha tests:

$ mocha express.test.js

Or, if you don’t have mocha installed globally:

$ ./node_modules/mocha/bin/mocha express.test.js

Node.js and MongoDB REST API Overview

This Node.js, Express.js and MongoDB (Mongoskin) tutorial will walk you through writing the test using the Mocha and Super Agent libraries. This is needed for a test-driven development building of a Node.js free JSON REST API server.

The server application itself will utilize Express.js 4.x framework and Mongoskin library for MongoDB. In this REST API server, we’ll perform create, read, update and delete (CRUD) operations and harness Express.js middleware concept with app.param() and app.use() methods.

First of all, make sure you have MongoDB installed. You can follow the steps on the official website.

We’ll be using the following versions of libraries:

  • express: ~4.1.1
  • body-parser: ~1.0.2
  • mongoskin: ~1.4.1
  • expect.js: ~0.3.1
  • mocha: ~1.18.2
  • superagent: ~0.17.0

If you try to attempt to use later or older versions the code might not work. :-(

Continue reading “Express.js 4, Node.js and MongoDB REST API Tutorial”

Blog Express.js Web Application Example

Here is one of the main examples that is featured in the book: the Blog application built with Express.js, MongoDB and Mongoskin. The application is subject to change, but if you’re an intermediate or advance Node.js programmer, go ahead and poke the source code, kick the tires of the server. It’s not a Ghost blogging platform, but blog-express can give valuable practical tips on how to implement:

Session-based authentication
Express.js middleware authorization
MVC-like pattern using lightweight MongoDB library called Mongoskin
REST API server
Express.js routes organization
Jade and server-side rendering

For the past few weeks, I’ve been writing a new book on Node.js. Hence the lack of new posts. This time, the book encompasses virtually all of the practical aspects of web development using Node.js: authentication, authorization, OAuth with third-party service providers, testing, libraries, frameworks, databases, ORMs and MVC-like patterns. In other words, if you have to read only one book on Node.js (which is not a good advice, but let’s assume), this will be the book that you have to read. :-)

The title and the alpha sign up are in secret for now (subscribe to get the news faster). The book is due in early spring 2014.

Here is one of the main examples that is featured in the book: the Blog application built with Express.js, MongoDB and Mongoskin. The application is subject to change, but if you’re an intermediate or advance Node.js programmer, go ahead and poke the source code, kick the tires of the server. It’s not a Ghost blogging platform, but blog-express can give valuable practical tips on how to implement:

  • Session-based authentication
  • Express.js middleware authorization
  • MVC-like pattern using lightweight MongoDB library called Mongoskin
  • REST API server
  • Express.js routes organization
  • Jade and server-side rendering
Blog Express.js App Home Page
Blog Express.js App Home Page

GitHub: http://github.com/azat-co/blog-express

Express.js and Mongoose Example: Building HackHall

Note: This text is a part of the Express.js Guide: The Comprehensive Book on Express.js.

The HackHall project was written using Backbone.js and Underscore for the front-end app, and Express.js, MongoDB via Mongoose for the back-end REST API server.

Continue reading “Express.js and Mongoose Example: Building HackHall”

The Release of Express.js Guide: The Comprehensive Book on Express.js

After weeks of writing and editing, Azat and his team are happy to announce the release of Express.js Guide: The Most Popular Node.js Framework Manual! The book is very approachable and suitable for beginners. If someone wants to save time searching the web and learn the best practices from the trenches, Express.js Guide is the book that has everything: Express.js API reference, quick start guides, 20+ meticulously explained examples and tutorials on over 270 pages with more than 60 illustrations.

Express.js is a de facto standard of Node.js development and the most popular NPM library as of today! However, as with any framework, sometimes the learning curve is steep. At HackReactor, I often asked the same questions about code organization, authentication, database connections and deployment.

Continue reading “The Release of Express.js Guide: The Comprehensive Book on Express.js”

Todo App with Express.js/Node.js and MongoDB

Todo apps are considered to be quintessential in showcasing frameworks akin to famous Todomvc.com for front-end JavaScript frameworks. In this example, we’ll use Jade, forms, LESS, AJAX/XHR and CSRF.

Note: This tutorial is a part of Express.js Guide: The Comprehensive Book on Express.js.

Todo apps are considered to be quintessential in showcasing frameworks akin to famous Todomvc.com for front-end JavaScript frameworks. In this example, we’ll use Jade, forms, LESS, AJAX/XHR and CSRF.

In our Todo app, we’ll intentionally not use Backbone.js or Angular to demonstrate how to build traditional websites with the use of forms and redirects. In addition to that, we’ll explain how to plug-in CSRF and LESS.

Example: All the source code is in the github.com/azat-co/todo-express for your convenience.

Continue reading “Todo App with Express.js/Node.js and MongoDB”

Tutorial: Node.js and MongoDB JSON REST API server with Mongoskin and Express.js

Tutorial: building a JSON REST API server with Node.js and MongoDB using Mongoskin and Express.js utilizing Mocha and Super Agent for BDD/TDD.

Update3: Expess 4 version of this tutorial is available at Express.js 4, Node.js and MongoDB REST API Tutorial, and github.com/azat-co/rest-api-express (master branch). This tutorial will work with Express 3.x.

Update2: “Mongoskin removed ‘db.collection.id’ and added some actionById methods” from this pull request with this code changes. To use the code in this post, just install an older version of Mongoskin (0.5.0?). The code in the GitHub will work with Mongoskin 1.3.20.

Update2: “Mongoskin removed ‘db.collection.id’ and added some actionById methods” from this pull request with this code changes. To use the code in this post, just install an older version of Mongoskin (0.5.0?)

Update: use the enhanced code from this repository github.com/azat-co/rest-api-express (express3 branch).

Note: This text is a part of Express.js Guide: The Comprehensive Book on Express.js.

This tutorial will walk you through writing test using the Mocha and Super Agent libraries and then use them in a test-driven development manner to build a Node.js free JSON REST API server utilizing Express.js framework and Mongoskin library for MongoDB. In this REST API server, we’ll perform create, read, update and delete (CRUD) operations and harness Express.js middleware concept with app.param() and app.use() methods.

Continue reading “Tutorial: Node.js and MongoDB JSON REST API server with Mongoskin and Express.js”

Nested Objects in Mongoose

There is a certain magic in ORMs like Mongoose. I learned it the hard way (as usual!), when I was trying to iterate over nested object’s properties…

There is a certain magic in ORMs like Mongoose. I learned it the hard way (as usual!), when I was trying to iterate over nested object’s properties. For example, here is a schema with a nested object features defines like this:

var User = module.exports = new Schema({
  features: { 
    realtime_updates: {
      type: Boolean
    },
    storylock: {
      type: Boolean
    },
    custom_embed_style: {
      type: Boolean
    },
    private_stories: {
      type: Boolean
    },
    headerless_embed:{
      type: Boolean
    }
};

Let’s say I want to overwrite object features_enabled with these properties:

if (this.features) { 
  for (var k in this.features) {
    features_enabled[k] = this.features[k];
  }
}
console.log(features_enabled)
return features_enabled;

Not so fast, I was getting a lot of system properties specific to Mongoose. Instead we need to use toObject(), e.g.:

if (this.features.toObject()) { 
  for (var k in this.features.toObject()) {
    console.log('!',k)
    features_enabled[k] = this.features.toObject()[k];
  }
}

Remember rule number one, computer is always right. If we think that it’s wrong — look up the rule number one. :-)

Intro to Express.js: Simple REST API app with Monk and MongoDB

After looking at Google Analytics stats I’ve realized that there is a demand for short Node.js tutorial and quick start guides. This is an introduction to probably the most popular (as of April 2013) Node.js framework Express.js.

Why?

After looking at Google Analytics stats I’ve realized that there is a demand for short Node.js tutorial and quick start guides. This is an introduction to probably the most popular (as of April 2013) Node.js framework Express.js.

Express.js — Node.js framework
Express.js — Node.js framework

mongoui

This app is a start of mongoui project. A phpMyAdmin counterpart for MongoDB written in Node.js. The goal is to provide a module with a nice web admin user interface. It will be something like Parse.com, Firebase.com, MongoHQ or MongoLab has but without trying it to any particular service. Why do we have to type db.users.findOne({'_id':ObjectId('...')}) any time we want to look up the user information? The alternative of MongoHub mac app is nice (and free) but clunky to use and not web based.

REST API app with Express.js and Monk

Ruby enthusiasts like to compare Express to Sinatra framework. It’s similarly flexible in the way how developers can build there apps. Application routes are set up in a similar manner, i.e., app.get('/products/:id', showProduct);. Currently Express.js is at version number 3.1. In addition to Express we’ll use Monk module.

We’ll use Node Package Manager which is usually come with a Node.js installation. If you don’t have it already you can get it at npmjs.org.

Create a new folder and NPM configuration file, package.json, in it with the following content:

{
  "name": "mongoui",
  "version": "0.0.1",
  "engines": {
    "node": ">= v0.6"
  },
  "dependencies": {
    "mongodb":"1.2.14",
    "monk": "0.7.1",
    "express": "3.1.0"
  }
}

Now run npm install to download and install modules into node_module folder. If everything went okay you’ll see bunch of folders in node_modules folders. All the code for our application will be in one file, index.js, to keep it simple stupid:

var mongo = require('mongodb');
var express = require('express');
var monk = require('monk');
var db =  monk('localhost:27017/test');
var app = new express();

app.use(express.static(__dirname + '/public'));
app.get('/',function(req,res){
  db.driver.admin.listDatabases(function(e,dbs){
      res.json(dbs);
  });
});
app.get('/collections',function(req,res){
  db.driver.collectionNames(function(e,names){
    res.json(names);
  })
});
app.get('/collections/:name',function(req,res){
  var collection = db.get(req.params.name);
  collection.find({},{limit:20},function(e,docs){
    res.json(docs);
  })
});
app.listen(3000)

Let break down the code piece by piece. Module declaration:

var mongo = require('mongodb');
var express = require('express');
var monk = require('monk');

Database and Express application instantiation:

var db =  monk('localhost:27017/test');
var app = new express();

Tell Express application to load and server static files (if there any) from public folder:

app.use(express.static(__dirname + '/public'));

Home page, a.k.a. root route, set up:

app.get('/',function(req,res){
  db.driver.admin.listDatabases(function(e,dbs){
      res.json(dbs);
  });
});

get() function just takes two parameters: string and function. The string can have slashes and colons, for example product/:id. The function must have two parapemets request and response. Request has all the information like query string parameters, session, headers and response is an object to with we output the results. In this case we do it by calling res.json() function. db.driver.admin.listDatabases() as you might guess give us a list of databases in async manner.

Two other routes are set up in a similar manner with get() function:

app.get('/collections',function(req,res){
  db.driver.collectionNames(function(e,names){
    res.json(names);
  })
});
app.get('/collections/:name',function(req,res){
  var collection = db.get(req.params.name);
  collection.find({},{limit:20},function(e,docs){
    res.json(docs);
  })
});

Express conveniently supports other HTTP verbs like post and update. In the case of setting up a post route we write this:

app.post('product/:id',function(req,res) {...});

Express also has support for middeware. Middleware is just a request function handler with three parameters: request, response, and next. For example:

app.post('product/:id', authenticateUser, validateProduct, addProduct);

function authenticateUser(req,res, next) {
  //check req.session for authentication
  next();
}

function validateProduct (req, res, next) {
   //validate submitted data
   next();
}

function addProduct (req, res) {
  //save data to database
}

validateProduct and authenticateProduct are middleware. They are usually put into separate file (or files) in a big projects.

Another way to set up middle ware in Express application is to use use() function. For example earlier we did this for static assets:

app.use(express.static(__dirname + '/public'));

We can also do it for error handlers:

app.use(errorHandler);

Assuming you have mongoDB installed this app will connect to it (localhost:27017) and display collection name and items in collections. To start mongo server:

$ mongod

to run app (keep the mongod terminal window open):

$ node .

or

$ node index.js

To see the app working, open http://localhost:3000 in Chrome with JSONViewer extension (to render JSON nicely).

Tom Hanks' The Polar Express
Tom Hanks’ The Polar Express

MongoDB migration with Node and Monk

Recently one of our top users complained that their Storify account is unaccessible. We’ve checked the production database and it appeared to be that the account might have been compromised and maliciously deleted by somebody using user’s account credentials. Thanks for a great MongoHQ service we had a backup database in less than 15 minutes.

Recently one of our top users complained that their Storify account was unaccessible. We’ve checked the production database and it appeares to be that the account might have been compromised and maliciously deleted by somebody using user’s account credentials. Thanks to a great MongoHQ service, we had a backup database in less than 15 minutes.
There were two options to proceed with the migration:

  1. Mongo shell script
  2. Node.js program

Because Storify user account deletion involves deletion of all related objects — identities, relationships (followers, subscriptions), likes, stories — we’ve decided to proceed with the latter option. It worked perfectly, and here is a simplified version which you can use as a boilerplate for MongoDB migration (also at gist.github.com/4516139).

Restoring MongoDB Records
Restoring MongoDB Records

Let’s load all the modules we need: Monk, Progress, Async, and MongoDB:

var async = require('async');
var ProgressBar = require('progress');
var monk = require('monk');
var ObjectId=require('mongodb').ObjectID;

By the way, made by LeanBoost, Monk is a tiny layer that provides simple yet substantial usability improvements for MongoDB usage within Node.JS.

Monk takes connection string in the following format:

username:password@dbhost:port/database

So we can create the following objects:

var dest = monk('localhost:27017/storify_localhost');
var backup = monk('localhost:27017/storify_backup');

We need to know the object ID which we want to restore:

var userId = ObjectId(YOUR-OBJECT-ID); 

This is a handy restore function which we can reuse to restore objects from related collections by specifying query (for more on MongoDB queries go to post Querying 20M-Record MongoDB Collection. To call it, just pass a name of the collection as a string, e.g., "stories" and a query which associates objects from this collection with your main object, e.g., {userId:user.id}. The progress bar is needed to show us nice visuals in the terminal.

var restore = function(collection, query, callback){
  console.info('restoring from ' + collection);
  var q = query;
  backup.get(collection).count(q, function(e, n) {
    console.log('found '+n+' '+collection);
    if (e) console.error(e);
    var bar = new ProgressBar('[:bar] :current/:total :percent :etas', { total: n-1, width: 40 })
    var tick = function(e) {
      if (e) {
        console.error(e);
        bar.tick();
      }
      else {
        bar.tick();
      }
      if (bar.complete) {
        console.log();
        console.log('restoring '+collection+' is completed');
        callback();                
      }
    };
    if (n>0){
      console.log('adding '+ n+ ' '+collection);
      backup.get(collection).find(q, { stream: true }).each(function(element) {
        dest.get(collection).insert(element, tick);
      });        
    } else {
      callback();
    }
  });
}

Now we can use async to call the restore function mentioned above:

async.series({
  restoreUser: function(callback){   // import user element
    backup.get('users').find({_id:userId}, { stream: true, limit: 1 }).each(function(user) {
      dest.get('users').insert(user, function(e){
        if (e) {
          console.log(e);
        }
        else {
          console.log('resored user: '+ user.username);
        }
        callback();
      });
    });
  },

  restoreIdentity: function(callback){  
    restore('identities',{
      userid:userId
    }, callback);
  },

  restoreStories: function(callback){
    restore('stories', {authorid:userId}, callback);
  }

  }, function(e) {
  console.log();
  console.log('restoring is completed!');
  process.exit(1);
});

The full code is available at gist.github.com/4516139 and here:

var async = require('async');
var ProgressBar = require('progress');
var monk = require('monk');
var ms = require('ms');
var ObjectId=require('mongodb').ObjectID;

var dest = monk('localhost:27017/storify_localhost');
var backup = monk('localhost:27017/storify_backup');

var userId = ObjectId(YOUR-OBJECT-ID); // monk should have auto casting but we need it for queries

var restore = function(collection, query, callback){
  console.info('restoring from ' + collection);
  var q = query;
  backup.get(collection).count(q, function(e, n) {
    console.log('found '+n+' '+collection);
    if (e) console.error(e);
    var bar = new ProgressBar('[:bar] :current/:total :percent :etas', { total: n-1, width: 40 })
    var tick = function(e) {
      if (e) {
        console.error(e);
        bar.tick();
      }
      else {
        bar.tick();
      }
      if (bar.complete) {
        console.log();
        console.log('restoring '+collection+' is completed');
        callback();                
      }
    };
    if (n>0){
      console.log('adding '+ n+ ' '+collection);
      backup.get(collection).find(q, { stream: true }).each(function(element) {
        dest.get(collection).insert(element, tick);
      });        
    } else {
      callback();
    }
  });
}

async.series({
  restoreUser: function(callback){   // import user element
    backup.get('users').find({_id:userId}, { stream: true, limit: 1 }).each(function(user) {
      dest.get('users').insert(user, function(e){
        if (e) {
          console.log(e);
        }
        else {
          console.log('resored user: '+ user.username);
        }
        callback();
      });
    });
  },

  restoreIdentity: function(callback){  
    restore('identities',{
      userid:userId
    }, callback);
  },

  restoreStories: function(callback){
    restore('stories', {authorid:userId}, callback);
  }

  }, function(e) {
  console.log();
  console.log('restoring is completed!');
  process.exit(1);
});
           

To launch it, run npm install/update and change hard-coded database values.