Tag Archives: JavaScript

Rapid Prototyping with JS: Agile JavaScript Development is done with v0.5 (1.0RC)!

Rapid Prototyping with JS: Agile JavaScript Development

Rapid Prototyping with JS: Agile JavaScript Development

This is a release candidate for 1.0 version which is going to be available to public in print. Page count has increase from 151 to 211 pages (PDF), which include the following updates:

  • More digestible (smaller) code examples with better comments
  • Express.js middleware section with an example
  • Express.js + MongoDB REST API server section with an example
  • Derby section with an example
  • Grammar and typo fixes
  • Illustrations
  • Summaries in the beginning of each chapter
  • Code examples formatting fixes

Download a free chapter Intro to Backbone.js or full PDF, ePub or Mobi copy at LeanPub.

Rapid Prototyping with JS touched topics such as:

  • jQuery
  • AJAX
  • CSS and LESS
  • JSON and BSON
  • Twitter Bootstrap
  • Node.js
  • MongoDB
  • Parse.com
  • Agile methodologies
  • Git
  • Heroku, MongoHQ and Windows Azure
  • REST API
  • Backbone.js
  • AMD and Require.js
  • Express.js
  • Monk
  • Derby

If you need in-depth knowledge or references, they are usually one click or one Google search away.

Practical aspect included building multiple versions of the Message Board app:

  • jQuery + Parse.com JS REST API
  • Backbone and Parse.com JS SDK
  • Backbone and Node.js
  • Backbone and Node.js + MongoDB

The Message Board application has all the foundation of a typical web/mobile application: fetching data, displaying it, submitting new data. Other examples include:

  • jQuery + Twitter RESP API “Tweet Analyzer”
  • Parse.com “Save John”
  • Node.js “Hello World”
  • MongoDB “Print Collections”
  • Derby + Express “Hello World”
  • Backbone.js “Hello World”
  • Backbone.js “Apple Database”
  • Monk + Expres.js “REST API Server”

Oh My JS: The Best JavaScript Articles (https://leanpub.com/ohmyjs/) is in the works!

Nested Objects in Mongoose

There is a certain magic in ORMs like Mongoose. I learned it the hard way (as usual!), when I was trying to iterate over nested object’s properties. For example, here is a schema with a nested object features defines like this:

var User = module.exports = new Schema({
  features: { 
    realtime_updates: {
      type: Boolean
    },
    storylock: {
      type: Boolean
    },
    custom_embed_style: {
      type: Boolean
    },
    private_stories: {
      type: Boolean
    },
    headerless_embed:{
      type: Boolean
    }
};

Let’s say I want to overwrite object features_enabled with these properties:

if (this.features) { 
  for (var k in this.features) {
    features_enabled[k] = this.features[k];
  }
}
console.log(features_enabled)
return features_enabled;

Not so fast, I was getting a lot of system properties specific to Mongoose. Instead we need to use toObject(), e.g.:

if (this.features.toObject()) { 
  for (var k in this.features.toObject()) {
    console.log('!',k)
    features_enabled[k] = this.features.toObject()[k];
  }
}

Remember rule number one, computer is always right. If we think that it’s wrong — look up the rule number one. :-)

Rapid Prototyping with JS Update v0.4

Rapid Prototyping with JS: Agile JavaScript Developement

Link to the new copy of Rapid Prototyping with JS at https://leanpub.com/rapid-prototyping-with-js.

Rapid Prototyping with JS: Agile JavaScript Developement

Rapid Prototyping with JS: Agile JavaScript Developement

TL;DR

Here is a list of the update for Rapid Prototyping with JS v0.4:

  • Brand new chapter: Intro to Backbone.js
  • Re-structured table of contents (chapters and parts)
  • Extended list of resources for further reading
  • Fixed code formatting and highlighting
  • Fixed grammar, style and typos mistakes
  • Brand new book cover optimized for ebook format
  • Reduced size of ebook (faster download)
  • Inspirational quotes in each chapter
  • Updated “Basics” chapter
  • New headline “Agile JavaScript Development”
  • Extended list of ways to reach us (Twitter, Facebook, Storify, etc.)

LeanPub Updates

LeanPub changed their purchasing. Now it’s even better for readers. Anybody can “return” the book they didn’t like within 45 days and get a full refund. Word return is in double quotes because all content is Digital Right Management (DRM) free. Does it mean that somebody can buy and download a book, get their money back, but keep the copy? Yes, but they will get bad karma for that! And not like a bad karma on Hacker News but a real bad karma. Everybody else deservers authors and publishers trust and respect.

Backbone.js

Why Backbone.js? Because a lot of people expressed desire to use it but being a framework Backbone has a learning curve. Not a steep one like Rails but still it takes time to master and learn Backbone. The new chapter “Intro to Backbone.js” will show readers how to:

  • Create Backbone architecture from scratch
  • Construct proper architecture with Routes
  • Use Collections
  • Apply Templates in Subviews
  • Split code into multiple JS files
  • Organize code into modules with AMD and Require.js
  • Optimize code for production with r.js library

Write and follow @RPJSbook.

Test-Driven Development in Node.js With Mocha

Who needs Test-Driven Development?

Imagine that you need to implement a complex feature on top of an existing interface, e.g., a ‘like’ button on a comment. Without tests you’ll have to manually create a user, log in, create a post, create a different user, log in with a different user and like the post. Tiresome? What if you’ll need to do it 10 or 20 times to find and fix some nasty bug? What if your feature breaks existing functionality, but you notice it 6 months after the release because there was no test!

Mocha: simple, flexible, fun

Mocha: simple, flexible, fun

Don’t waste time writing tests for throwaway scripts, but please adapt the habit of Test-Driven Development for the main code base. With a little time spent in the beginning, you and your team will save time later and have confidence when rolling out new releases. Test Driven Development is a really really really good thing.

Quick Start Guide

Follow this quick guide to set up your Test-Driven Development process in Node.js with Mocha.

Install Mocha globally by executing this command:

$ sudo npm install -g mocha

We’ll also use two libraries, Superagent and expect.js by LeanBoost. To install them fire up npm commands in your project folder like this:

$ npm install superagent
$ npm install expect.js   

Open a new file with .js extension and type:

var request = require('superagent');
var expect = require('expect.js');

So far we’ve included two libraries. The structure of the test suite going to look like this:

describe('Suite one', function(){
  it(function(done){
  ...
  });
  it(function(done){
  ...
  });
});
describe('Suite two', function(){
  it(function(done){
  ...
  });
});

Inside of this closure we can write request to our server which should be running at localhost:8080:

...
it (function(done){
  request.post('localhost:8080').end(function(res){
    //TODO check that response is okay
  });
});
...

Expect will give us handy functions to check any condition we can think of:

...
expect(res).to.exist;
expect(res.status).to.equal(200);
expect(res.body).to.contain('world');
...

Lastly, we need to add done() call to notify Mocha that asynchronous test has finished its work. And the full code of our first test looks like this:

var request = require('superagent');
var expect = require('expect.js');
  
describe('Suite one', function(){
 it (function(done){
   request.post('localhost:8080').end(function(res){
    expect(res).to.exist;
    expect(res.status).to.equal(200);
    expect(res.body).to.contain('world');
    done();
   });
  });
});

If we want to get fancy, we can add before and beforeEach hooks which will, according to their names, execute once before the test (or suite) or each time before the test (or suite):

before(function(){
  //TODO seed the database
});
describe('suite one ',function(){
  beforeEach(function(){
    //todo log in test user
  });
  it('test one', function(done){
  ...
  });
});

Note that before and beforeEach can be placed inside or outside of describe construction.

To run our test simply execute:

$ mocha test.js

To use different report type:

$ mocha test.js -R list
$ mocha test.js -R spec

Asynchronicity in Node.js

Non-Blocking I/O

One of the biggest advantages of using Node.js over Python or Ruby is that Node has a non-blocking I/O mechanism. To illustrate this, let me use an example of a line in a Starbucks coffee shop. Let’s pretend that each person standing in line for a drink is a task, and everything behind the counter — cashier, register, barista — is a server or server application. When we order a cup of regular drip coffee, like Pike, or hot tea, like Earl Grey, the barista makes it. The whole line waits while that drink is made, and the person is charged the appropriate amount.

Asynchronicity in Node.js

Asynchronicity in Node.js

Of course, we know that these kinds of drinks are easy to make; just pour the liquid and it’s done. But what about those fancy choco-mocha-frappe-latte-soy-decafs? What if everybody in line decides to order these time-consuming drinks? The line will be held up by each order, and it will grow longer and longer. The manager of the coffee shop will have to add more registers and put more baristas to work (or even stand behind the register him/herself). This is not good, right? But this is how virtually all server-side technologies work, except Node. Node is like a real Starbucks. When you order something, the barista yells the order to the other employee, and you leave the register. Another person gives their order while you wait for your state-of-the-art eye-opener in a paper cup. The line moves, the processes are executed asynchronously and without blocking the queue by waiting.

This is why Node.js blows everything else away (except maybe low-level C/C++) in terms of performance and scalability. With Node, you just don’t need that many CPUs and servers to handle the load.

Asynchronous Way of Coding

Asynchronicity requires a different way of thinking for programmers familiar with Python, PHP, C or Ruby. It’s easy to introduce a bug unintentionally by forgetting to end the execution of the code with a proper return expression.

Here is a simple example illustrating this scenario:

var test = function (callback) {
  return callback();  
  console.log('test') //shouldn't be printed
}

var test2 = function(callback){
  callback();
  console.log('test2') //printed 3rd
}

test(function(){
  console.log('callback1') //printed first
  test2(function(){
  console.log('callback2') //printed 2nd
  })
});

If we don’t use return callback() and just use callback() our string test2 will be printed (test is not printed).

callback1
callback2
tes2

For fun I’ve added a setTimeout() delay for the callback2 string, and now the order has changed:

var test = function (callback) {
  return callback();  
  console.log('test') //shouldn't be printed
}

var test2 = function(callback){
  callback();
  console.log('test2') //printed 2nd
}

test(function(){
  console.log('callback1') //printed first
  test2(function(){
    setTimeout(function(){
      console.log('callback2') //printed 3rd
    },100)
  })
});

Prints:

callback1
tes2
callback2

The last example illustrates that the two functions are independent of each other and run in parallel. The faster function will finish sooner than the slower one. Going back to our Starbucks examples, you might get your drink faster than the other person who was in front of you in the line. Better for people, and better for programs! :-)

Cheat Sheets for Web Development

Cheat sheets are great ways to organize frequently used information and keep it handy. I used cheat sheets for learning and memorizing during my crams at school, and use them now for reference.

Cheat sheet for Web Development

Cheat sheet for Web Development

Web development usually involves a large number of languages each with its own syntax, keywords, special sauce and magic tricks.
Here is a collection of web development cheat sheets, in no particular order, which I’ve amassed by browsing the Internet over many years of web development. They cover the following topics:

  • jQuery
  • CSS3
  • Git
  • Heroku
  • HTML5
  • Linux Command Line
  • Mod reWrite
  • CoffeeScript
  • JavaScript
  • CSS2
  • JavaScript DOM
  • Mac Glyphs
  • Node.js
  • PHP
  • RGB Hex
  • Sublime Text 2
  • SEO
  • WordPress

Get zip archive at http://www.webapplog.com/wp-content/uploads/web-dev-cheat-sheets.zip.

Full list of the files:


jquery12_colorcharge.png
cdiehl_firefox.pdf
CSS Help Sheet outlined.pdf
css-cheat-sheet-v2.pdf
css-cheat-sheet.pdf
CSS3 Help Sheet outlined.pdf
css3-cheat-sheet.pdf
dan-schmidt_jquery-utility-functions-type-testing.pdf
danielschmitz_jquery-mobile.pdf
davechild_css2.pdf
davechild_html-character-entities.pdf
davechild_javascript.pdf
davechild_linux-command-line.pdf
davechild_mod-rewrite.pdf
davechild_regular-expressions.pdf
dimitrios_coffeescript-cheat-sheet.pdf
gelicia_sublime-text-2-shortcuts-verbose-mac.pdf
Git_Cheat_Sheet_grey.pdf
heroku_cheatsheet.pdf
HTML Help Sheet 02.pdf
html5-cheat-sheet.pdf
i3quest_jquery.pdf
javascript_refererence.pdf
javascript-cheat-sheet-v1.pdf
JavaScript-DOM-Cheatsheet.pdf
javascriptcheatsheet.pdf
jQuery-1_3-Visual-Cheat-Sheet.pdf
Mac_Glyphs_All.pdf
Node Help Sheet.pdf
PHP Help Sheet 01.pdf
pyro19d_javascript.pdf
rgb-hex-cheat-sheet-v1.pdf
salesforce_git_developer_cheatsheet.pdf
samcollett_git.pdf
sanoj_web-programming.pdf
SEO_Web_Developer_Cheat_Sheet.pdf
skrobul_sublime-text-2-linux.pdf
WordPress-Help-Sheet.pdf
Help Sheets.zip

Decreasing 64-bit Tweet ID in JavaScript

As some of you might know, JavaScript is only able to handle integers up to 53-bit in size. This post, Working with large integers in JavaScript (which is a part of Numbers series) does a great job at explaining general concepts on dealing with large numbers in JS.

64-bit Tweet ID is "rounded" in JS

64-bit Tweet ID is “rounded” in JS

I had to do some research on the topic when I was re-writing some JavaScript code responsible for handling Twitter search in Storify editor: we had tweet duplicates in results! In this article, Working with Timelines, Twitter official documentation says:

Environments where a Tweet ID cannot be represented as an integer with 64 bits of precision (such as JavaScript) should skip this step.

So true, because id and id_str fields in a Twitter API response were different. Apparently, JavaScript engine just “rounds” inappropriately large numbers. :-( The task was complicated by the fact that I needed to subtract 1 from the last tweet’s ID to prevent its reappearance in a second search response. After the subtraction I could have easily passed the value to max_id parameter of Twitter API.

I’ve come across different solutions, but decided to write my own function which is simple to understand and not heavy on resources. Here is a script to decrease tweet ID which is a 64-bit number in JavaScript without libraries or recursion, to use with max_id or since_id in Twitter API:

function decStrNum (n) {
    n = n.toString();
    var result=n;
    var i=n.length-1;
    while (i>-1) {
      if (n[i]==="0") {
        result=result.substring(0,i)+"9"+result.substring(i+1);
        i --;
      }
      else {
        result=result.substring(0,i)+(parseInt(n[i],10)-1).toString()+result.substring(i+1);
        return result;
      }
    }
    return result;
}

To check if it works, you can run these logs:

console.log("290904187124985850");
console.log(decStrNum("290904187124985850"));
console.log("290904187124985851");
console.log(decStrNum("290904187124985851"));
console.log("290904187124985800");
console.log(decStrNum("290904187124985800"));
console.log("000000000000000001");
console.log(decStrNum("0000000000000000001"));

Alternative solution which I’ve found in a StackOverflow question was suggested by Bob Lauer, but it involves recursion and IMHO is more complicated:

function decrementHugeNumberBy1(n) {
    // make sure s is a string, as we can't do math on numbers over a certain size
    n = n.toString();
    var allButLast = n.substr(0, n.length - 1);
    var lastNumber = n.substr(n.length - 1);

    if (lastNumber === "0") {
        return decrementHugeNumberBy1(allButLast) + "9";
    }
    else {      
        var finalResult = allButLast + (parseInt(lastNumber, 10) - 1).toString();
        return trimLeft(finalResult, "0");
    }
}

function trimLeft(s, c) {
    var i = 0;
    while (i < s.length && s[i] === c) {
        i++;
    }

    return s.substring(i);
}

Now, if you’re the type of person who likes to shoot sparrows with a howitzer, there are full-blown libraries to handle operations on large numbers in JavaScript; just to name a few: BigInteger, js-numbers and javascript-bignum.

MongoDB migration with Node and Monk

Recently one of our top users complained that their Storify account was unaccessible. We’ve checked the production database and it appeares to be that the account might have been compromised and maliciously deleted by somebody using user’s account credentials. Thanks to a great MongoHQ service, we had a backup database in less than 15 minutes.
There were two options to proceed with the migration:

  1. Mongo shell script
  2. Node.js program

Because Storify user account deletion involves deletion of all related objects — identities, relationships (followers, subscriptions), likes, stories — we’ve decided to proceed with the latter option. It worked perfectly, and here is a simplified version which you can use as a boilerplate for MongoDB migration (also at gist.github.com/4516139).

Restoring MongoDB Records

Restoring MongoDB Records

Let’s load all the modules we need: Monk, Progress, Async, and MongoDB:

var async = require('async');
var ProgressBar = require('progress');
var monk = require('monk');
var ObjectId=require('mongodb').ObjectID;

By the way, made by LeanBoost, Monk is a tiny layer that provides simple yet substantial usability improvements for MongoDB usage within Node.JS.

Monk takes connection string in the following format:

username:password@dbhost:port/database

So we can create the following objects:

var dest = monk('localhost:27017/storify_localhost');
var backup = monk('localhost:27017/storify_backup');

We need to know the object ID which we want to restore:

var userId = ObjectId(YOUR-OBJECT-ID); 

This is a handy restore function which we can reuse to restore objects from related collections by specifying query (for more on MongoDB queries go to post Querying 20M-Record MongoDB Collection. To call it, just pass a name of the collection as a string, e.g., "stories" and a query which associates objects from this collection with your main object, e.g., {userId:user.id}. The progress bar is needed to show us nice visuals in the terminal.

var restore = function(collection, query, callback){
  console.info('restoring from ' + collection);
  var q = query;
  backup.get(collection).count(q, function(e, n) {
    console.log('found '+n+' '+collection);
    if (e) console.error(e);
    var bar = new ProgressBar('[:bar] :current/:total :percent :etas', { total: n-1, width: 40 })
    var tick = function(e) {
      if (e) {
        console.error(e);
        bar.tick();
      }
      else {
        bar.tick();
      }
      if (bar.complete) {
        console.log();
        console.log('restoring '+collection+' is completed');
        callback();                
      }
    };
    if (n>0){
      console.log('adding '+ n+ ' '+collection);
      backup.get(collection).find(q, { stream: true }).each(function(element) {
        dest.get(collection).insert(element, tick);
      });        
    } else {
      callback();
    }
  });
}

Now we can use async to call the restore function mentioned above:

async.series({
  restoreUser: function(callback){   // import user element
    backup.get('users').find({_id:userId}, { stream: true, limit: 1 }).each(function(user) {
      dest.get('users').insert(user, function(e){
        if (e) {
          console.log(e);
        }
        else {
          console.log('resored user: '+ user.username);
        }
        callback();
      });
    });
  },

  restoreIdentity: function(callback){  
    restore('identities',{
      userid:userId
    }, callback);
  },

  restoreStories: function(callback){
    restore('stories', {authorid:userId}, callback);
  }

  }, function(e) {
  console.log();
  console.log('restoring is completed!');
  process.exit(1);
});

The full code is available at gist.github.com/4516139 and here:

var async = require('async');
var ProgressBar = require('progress');
var monk = require('monk');
var ms = require('ms');
var ObjectId=require('mongodb').ObjectID;

var dest = monk('localhost:27017/storify_localhost');
var backup = monk('localhost:27017/storify_backup');

var userId = ObjectId(YOUR-OBJECT-ID); // monk should have auto casting but we need it for queries

var restore = function(collection, query, callback){
  console.info('restoring from ' + collection);
  var q = query;
  backup.get(collection).count(q, function(e, n) {
    console.log('found '+n+' '+collection);
    if (e) console.error(e);
    var bar = new ProgressBar('[:bar] :current/:total :percent :etas', { total: n-1, width: 40 })
    var tick = function(e) {
      if (e) {
        console.error(e);
        bar.tick();
      }
      else {
        bar.tick();
      }
      if (bar.complete) {
        console.log();
        console.log('restoring '+collection+' is completed');
        callback();                
      }
    };
    if (n>0){
      console.log('adding '+ n+ ' '+collection);
      backup.get(collection).find(q, { stream: true }).each(function(element) {
        dest.get(collection).insert(element, tick);
      });        
    } else {
      callback();
    }
  });
}

async.series({
  restoreUser: function(callback){   // import user element
    backup.get('users').find({_id:userId}, { stream: true, limit: 1 }).each(function(user) {
      dest.get('users').insert(user, function(e){
        if (e) {
          console.log(e);
        }
        else {
          console.log('resored user: '+ user.username);
        }
        callback();
      });
    });
  },

  restoreIdentity: function(callback){  
    restore('identities',{
      userid:userId
    }, callback);
  },

  restoreStories: function(callback){
    restore('stories', {authorid:userId}, callback);
  }

  }, function(e) {
  console.log();
  console.log('restoring is completed!');
  process.exit(1);
});
           

To launch it, run npm install/update and change hard-coded database values.