Ship Code That Matters

Recently I’ve been thinking a lot about the meaning of life and our purpose on planet Earth. Just kidding! I was thinking why so many programmers are unhappy. Programming is fun and creative after all. The main reason for the unhappiness is that such engineers don’t believe in the software product they’re building. They don’t see how it can benefit end-users. Maybe they don’t even know who their end-users are.

This feeling was not unfamiliar to me at times when I was working for government agencies, but it’s foreign to me now. At Storify our team (myself included) love the product. We see the direct impact it brings to the users, e.g., The White House, ABC News and CBS News. Storify enables people to tell their stories which might otherwise get lost in the sea of social media noise. This contemplation brought me to the conclusion that it’s important to ship code that matters.

Recently I’ve been thinking a lot about the meaning of life and our purpose on planet Earth. Just kidding! I was thinking why so many programmers are unhappy. Programming is fun and creative after all. The main reason for the unhappiness is that such engineers don’t believe in the software product they’re building. They don’t see how it can benefit end-users. Maybe they don’t even know who their end-users are.

Continue reading “Ship Code That Matters”

LeanPub Compilation and Sanitizing Tool Written in Node.js

LeanPub Tool — Node.js script for sanitizing and compiling a book’s manuscript. LeanPub uses multi-file book format and a special markdown markup (that confuses Marked app). I found that to convert to MS Word (for editing and other needs) this workflow is the best: 1. run leanpub-tool.js with node.js (node leanpub-tool.js) 2. open file in Marked app and convert to HTML 3. open HTML in MS Word and enjoy.

LeanPub is an awesome publishing platform, but often we need to communicate with other  team members involved in the book production, e.g., content and copy editors. In such times, I’ve learned that Guy Kawasaki was right — when we suggested in his book APE: Author, Publisher, Entrepreneur — that author must use MS Word!

The problem is that LeanPub uses multi-file book format and a special markdown markup (that confuses Marked app). I was wrapping up a new revision of the Rapid Prototyping with JS book and previous experience of handing a txt file to an editor was poor. Luckily, Node.js came to help! Through some trail and error attempts, I found this workflow to be the best for converting LeanPub manuscript to a one MS Word file (for editing and other needs):

  1. Run leanpub-tool.js with Node.js (node leanpub-tool.js)
  2. Open file in Marked app and convert to HTML
  3. Open HTML in MS Word and enjoy.

The full code of leanpub-tool.js which is also available at https://gist.github.com/azat-co/5674684:


var fs = require('fs');
//change these to YOUR filenames
//probably we can read it from Book.txt but I was too lazy to implement it :-) and somebody might want to compile only sertain parts of the book
var book =[
"frontmatter.txt",
"mainmatter.txt",
"part1.txt",
"chapter1.txt",
"part2.txt",
"chapter2.txt",
"backmatter.txt",
"acknowledgment.txt"
];
 
var sanitizeText = [
  '{frontmatter}',
  '{backmatter}',
  '{mainmatter}',
  "I>## Note",
  "T>## Tip",
  "W>## Warning",
  '{lang="javascript"}',
  '{:lang="javascript"}',
  '{lang="css"}',
  '{:lang="css"}',
  '{lang="json"}',
  '{lang="ruby"}',
  '{lang="php"}',
  '{lang="text"}',
  '{lang="bash"}',
  '{lang="html"}',
  "I>",
  "T>",
  "W>"
]
 
 
var str = '';
//read files
book.forEach(function(chapter){
  str +=  fs.readFileSync(chapter,'utf8');
})
//sanitize LeanPub specific Markdown tags
sanitizeText.forEach(function(text){
//  console.log(text)
  //this loops through while there is not matches
  while (new RegExp (text).test(str)) {
    str = str.replace(text,'','gm')
  };
})
 
//write output to a file
fs.writeFileSync('leanpub-tool.txt',str);

Rapid Prototyping with JS: Agile JavaScript Development is done with v0.5 (1.0RC)!

This is a release candidate for 1.0 version which is going to be available to public in print. Page count has increase from 151 to 211 pages (PDF), which include the following updates:

More digestible (smaller) code examples with better comments
Express.js middleware section with an example
Express.js + MongoDB REST API server section with an example
Derby section with an example
Grammar and typo fixes
Illustrations
Summaries in the beginning of each chapter
Code examples formatting fixes

Rapid Prototyping with JS: Agile JavaScript Development
Rapid Prototyping with JS: Agile JavaScript Development

This is a release candidate for 1.0 version which is going to be available to public in print. Page count has increase from 151 to 211 pages (PDF), which include the following updates:

  • More digestible (smaller) code examples with better comments
  • Express.js middleware section with an example
  • Express.js + MongoDB REST API server section with an example
  • Derby section with an example
  • Grammar and typo fixes
  • Illustrations
  • Summaries in the beginning of each chapter
  • Code examples formatting fixes

Download a free chapter Intro to Backbone.js or full PDF, ePub or Mobi copy at LeanPub.

Rapid Prototyping with JS touched topics such as:

  • jQuery
  • AJAX
  • CSS and LESS
  • JSON and BSON
  • Twitter Bootstrap
  • Node.js
  • MongoDB
  • Parse.com
  • Agile methodologies
  • Git
  • Heroku, MongoHQ and Windows Azure
  • REST API
  • Backbone.js
  • AMD and Require.js
  • Express.js
  • Monk
  • Derby

If you need in-depth knowledge or references, they are usually one click or one Google search away.

Practical aspect included building multiple versions of the Message Board app:

  • jQuery + Parse.com JS REST API
  • Backbone and Parse.com JS SDK
  • Backbone and Node.js
  • Backbone and Node.js + MongoDB

The Message Board application has all the foundation of a typical web/mobile application: fetching data, displaying it, submitting new data. Other examples include:

  • jQuery + Twitter RESP API “Tweet Analyzer”
  • Parse.com “Save John”
  • Node.js “Hello World”
  • MongoDB “Print Collections”
  • Derby + Express “Hello World”
  • Backbone.js “Hello World”
  • Backbone.js “Apple Database”
  • Monk + Expres.js “REST API Server”

Oh My JS: The Best JavaScript Articles (https://leanpub.com/ohmyjs/) is in the works!

Nested Objects in Mongoose

There is a certain magic in ORMs like Mongoose. I learned it the hard way (as usual!), when I was trying to iterate over nested object’s properties…

There is a certain magic in ORMs like Mongoose. I learned it the hard way (as usual!), when I was trying to iterate over nested object’s properties. For example, here is a schema with a nested object features defines like this:

var User = module.exports = new Schema({
  features: { 
    realtime_updates: {
      type: Boolean
    },
    storylock: {
      type: Boolean
    },
    custom_embed_style: {
      type: Boolean
    },
    private_stories: {
      type: Boolean
    },
    headerless_embed:{
      type: Boolean
    }
};

Let’s say I want to overwrite object features_enabled with these properties:

if (this.features) { 
  for (var k in this.features) {
    features_enabled[k] = this.features[k];
  }
}
console.log(features_enabled)
return features_enabled;

Not so fast, I was getting a lot of system properties specific to Mongoose. Instead we need to use toObject(), e.g.:

if (this.features.toObject()) { 
  for (var k in this.features.toObject()) {
    console.log('!',k)
    features_enabled[k] = this.features.toObject()[k];
  }
}

Remember rule number one, computer is always right. If we think that it’s wrong — look up the rule number one. :-)

Git Reflog to Rescue after Hard Resets

On benefits of using Git reflog command

Today I was resetting a local Git repository to previous commits,, in order to trace back some changes which lead to Continious Integration tests failures. I did a few trail and error runs with:


git reset --hard SHA

Where SHA was taken from GitHub or git log. These commands were performed on my local branch which had some new changes commited locally, but not yet pushed to GitHub. Then, I found the working old commit I realized that I am stuck on it! There was a SHA for my latest commit and the command:


git log

wasn’t showing my latest commit either, because I moved the HEAD of the branch behind it. Thansk for an advice, this command saved me:


git reflog BRANCHNAME

It had a short SHA of my latest commit and I was able to reset HEAD back with reset. I knew that Git had my change save, but I wasn’t sure how to get them. As usual Git is a great technology… with a terrible user interface! :-)

Git reflog — a lifesaver!
Git reflog — a lifesaver!

More on git reflog:

Intro to Express.js: Parameters, Error Handling and Other Middleware

Express.js is a node.js framework that among other things provides a way to organize routes. Each route is defined via a method call on an application object with a URL patter as a first parameter (RegExp is also supported). The best practice is to keep router lean and thin by moving all the logic into corresponding external modules/files. This way important server configuration parameters will be neatly in one place, right there when you need them! :-)

Note: This text is a part of Express.js Guide: The Comprehensive Book on Express.js.

Express.js is one of the most popular and mature Node.js frameworks. You can read more about in Intro to Express.js series on webapplog.com:

To learn how to create an application from scratch please refer to the earlier post.

Request Handlers

Express.js is a node.js framework that among other things provides a way to organize routes. Each route is defined via a method call on an application object with a URL patter as a first parameter (RegExp is also supported), for example:

app.get('api/v1/stories/', function(res, req){
  ...
})

or, for a POST method:

app.post('/api/v1/stories'function(req,res){
  ...
})

It’s needless to say that DELETE and PUT methods are supported as well.
The callbacks that we pass to get() or post() methods are called request handlers, because they take requests (req), process them and write to response (res) objects. For example:

app.get('/about', function(req,res){
  res.send('About Us: ...');
});

We can have multiple request handlers, hence the name middleware. They accept a third parameter next calling which (next()) will switch the execution flow to the next handler:

app.get('/api/v1/stories/:id', function(req,res, next) {
  //do authorization
  //if not authorized or there is an error 
  // return next(error);
  //if authorized and no errors
  return next();
}), function(req,res, next) {
  //extract id and fetch the object from the database
  //assuming no errors, save story in the request object
  req.story = story;
  return next();
}), function(req,res) {
  //output the result of the database search
  res.send(res.story);
});

ID of a story in URL patter is a query string parameter which we need for finding a matching items in the database.

Parameters Middleware

Parameters are values passed in a query string of a URL of the request. If we didn’t have Express.js or similar library, and had to use just the core Node.js modules, we’d had to extract parameters from HTTP.request object via some require('querystring').parse(url) or require('url').parse(url, true) functions trickery.

Thanks to Connect framework and people at VisionMedia, Express.js already has support for parameters, error handling and many other important features in the form of middlewares. This is how we can plug param middleware in our app:

app.param('id', function(req,res, next, id){
  //do something with id
  //store id or other info in req object
  //call next when done
  next();
});

app.get('/api/v1/stories/:id',function(req,res){
  //param middleware will be execute before and
  //we expect req object already have needed info
  //output something
  res.send(data);
});

For example:

app.param('id', function(req,res, next, id){
  req.db.get('stories').findOne({_id:id}, function (e, story){
    if (e) return next(e);
    if (!story) return next(new Error('Nothing is found'));
    req.story = story;
    next();
  });
});

app.get('/api/v1/stories/:id',function(req,res){
  res.send(req.story);
});

Or we can use multiple request handlers but the concept remains the same: we can expect to have req.story object or an error thrown prior to the execution of this code so we abstract common code/logic of getting parameters and their respective objects:

app.get('/api/v1/stories/:id', function(req,res, next) {
  //do authorization
  }),
  //we have an object in req.story so no work is needed here
  function(req,res) {
  //output the result of the database search
  res.send(story);
});

Authorization and input sanitation are also good candidates for residing in the middlewares.

Function param() is especially cool because we can combine different keys, e.g.:

app.get('/api/v1/stories/:storyId/elements/:elementId',function(req,res){
  res.send(req.element);
});

Error Handling

Error handling is typically used across the whole application, therefore it’s best to implement it as a middleware. It has the same parameters plus one more, error:

app.use(function(err, req, res, next) {
  //do logging and user-friendly error message display
  res.send(500);
})

In fact, the response can be anything:

JSON string

app.use(function(err, req, res, next) {
  //do logging and user-friendly error message display
  res.send(500, {status:500, message: 'internal error', type:'internal'});
})

Text message

app.use(function(err, req, res, next) {
  //do logging and user-friendly error message display
  res.send(500, 'internal server error');
})

Error page

app.use(function(err, req, res, next) {
  //do logging and user-friendly error message display
  //assuming that template engine is plugged in
  res.render('500');
})

Redirect to error page

app.use(function(err, req, res, next) {
  //do logging and user-friendly error message display
  res.redirect('/public/500.html');
})

Error HTTP response status (401, 400, 500, etc.)

app.use(function(err, req, res, next) {
  //do logging and user-friendly error message display
  res.end(500);
})

By the way, logging is also should be abstracted in a middleware!

To trigger an error from within your request handlers and middleware you can just call:

next(error);

or

next(new Error('Something went wrong :-(');

You can also have multiple error handlers, and use named instead of anonymous functions as its shows in Express.js Error handling guide.

Other Middleware

In addition to extracting parameters, it can be used for many things, like authorization, error handling, sessions, output, and others.

res.json() is one of them. It conveniently outputs JavaScript/Node.js object as a JSON. For example:

app.get('/api/v1/stories/:id', function(req,res){
  res.json(req.story);
});

is equivalent to (if req.story is an Array and Object):

app.get('/api/v1/stories/:id', function(req,res){
  res.send(req.story);
});

or

app.get('api/v1/stories/:id',function(req,res){
  res.set({
    'Content-Type': 'application/json'
  });
  res.send(req.story);
});

Abstraction

Middleware is flexible. You can use anonymous or named functions, but the best thing is to abstract request handlers into external modules based on the functionality:

var stories = require.('./routes/stories');
var elements = require.('./routes/elements');
var users = require.('./routes/users');
...
app.get('/stories/,stories.find);
app.get('/stories/:storyId/elements/:elementId', elements.find);
app.put('/users/:userId',users.update);

routes/stories.js:

module.exports.find = function(req,res, next) {
};

routes/elements.js:

module.exports.find = function(req,res,next){
};

routes/users.js:

module.exports.update = function(req,res,next){
};

You can use some functional programming tricks, like this:

function requiredParamHandler(param){
  //do something with a param, e.g., check that it's present in a query string
  return function (req,res, next) {
    //use param, e.g., if token is valid proceed with next();
    next();
  });
}

app.get('/api/v1/stories/:id', requiredParamHandler('token'), story.show);

var story  = {
  show: function (req, res, next) {
    //do some logic, e.g., restrict fields to output
    return res.send();
  }
}   

As you can see middleware is a powerful concept for keeping code organized. The best practice is to keep router lean and thin by moving all the logic into corresponding external modules/files. This way important server configuration parameters will be neatly in one place, right there when you need them! :-)

Node.js MVC: Express.js + Derby.js Hello World Tutorial

Express.js is a popular node frameworks which uses middleware concept to enhance functionality of applications. Derby is a new sophisticated Model View Controller (MVC) framework which is designed to be used with Express as it’s middleware. Derby also comes with the support of Racer, data synchronization engine, and Handlebars-like template engine among many other features.

DerbyJS — Node.js MVC Framework

Express.js is a popular node frameworks which uses middleware concept to enhance functionality of applications. Derby is a new sophisticated Model View Controller (MVC) framework which is designed to be used with Express as it’s middleware. Derby also comes with the support of Racer, data synchronization engine, and Handlebars-like template engine among many other features.

Derby.js Installation

Let’s set up a basic Derby application architecture without the use of scaffolding. Usually project generators are confusing when people just start to learn a new comprehensive framework. This is a bare minimum “Hello World” application tutorial that still illustrates Derby skeleton and demonstrates live-templates with websockets.

Of course we’ll need Node.js and NPM which can be obtained at nodejs.org. To install derby globally run:

$ npm install -g derby

To check the installation:

$ derby -V

My version as of April 2013 is 0.3.15. We should be good to go to creating our first app!

File Structure in a Derby.js App

This is the project folder structure:

project/
  -package.json
  -index.js
  -derby-app.js
  views/
    derby-app.html
  styles/
    derby-app.less

Dependencies for The Derby.js Project

Let’s include dependencies and other basic information in package.json file:

 {
  "name": "DerbyTutorial",
  "description": "",
  "version": "0.0.0",
  "main": "./server.js",
  "dependencies": {
    "derby": "*",
    "express": "3.x"
  },
  "private": true
}

Now we can run npm install which will download our dependencies into node_modules folder.

Views in Derby.js

Views must be in views folder and they must be either in index.html under a folder which has the same name as your derby app JavaScript file, i.e., views/derby-app/index.html, or be inside of a file which has the same name as your derby app JS file, i.e., derby-app.html.

In this example “Hello World” app we’ll use <Body:> template and {message} variable. Derby uses mustach-handlebars-like syntax for reactive binding. index.html looks like this:

<Body:>
  <input value="{message}"><h1>{message}</h1>

Same thing with Stylus/LESS files, in our example index.css has just one line:

h1 {
  color: blue;
}

To find out more about those wonderful CSS preprocessors check out documentation at Stylus and LESS.

Building The Main Derby.js Server

index.js is our main server file, and we begin it with an inclusion of dependencies with require() function:

var http = require('http'),
  express = require('express'),
  derby = require('derby'),
  derbyApp = require('./derby-app');

Last line is our derby application file derby-app.js.

Now we’re creating Express.js application (v3.x has significant differences between 2.x) and an HTTP server:

var expressApp = new express(),
  server = http.createServer(expressApp);

Derby uses Racer data synchronization library which we create like this:

var store = derby.createStore({
  listen: server
});

To fetch some data from back-end to the front-end we instantiate model object:

var model = store.createModel();

Most importantly we need to pass model and routes as middlewares to Express.js app. We need to expose public folder for socket.io to work properly.

expressApp.
  use(store.modelMiddleware()).
  use(express.static(__dirname + '/public')).
  use(derbyApp.router()).
  use(expressApp.router);

Now we can start the server on port 3001 (or any other):

server.listen(3001, function(){
  model.set('message', 'Hello World!');
});

Full code of index.js file:

var http = require('http'),
  express = require('express'),
  derby = require('derby'),
  derbyApp = require('./derby-app');

var expressApp = new express(),
  server = http.createServer(expressApp);

var store = derby.createStore({
  listen: server
});

var model = store.createModel();

expressApp.
  use(store.modelMiddleware()).
  use(express.static(__dirname + '/public')).
  use(derbyApp.router()).
  use(expressApp.router);

server.listen(3001, function(){
  model.set('message', 'Hello World!');
});

Derby.js Application

Finally, Derby app file which contains code for both a front-end and a back-end. Front-end only code is inside of app.ready() callback. To start, let’s require and create an app. Derby uses unusual construction (not the same familiar good old module.exports = app):

var derby = require('derby'),
  app = derby.createApp(module);

To make socket.io magic work we need to subscribe model attribute to its visual representation, in other words bind data and view. We can do it in the root route, and this is how we define it (patter is /, a.k.a. root):

app.get('/', function(page, model, params) {
  model.subscribe('message', function() {
    page.render();  
  })  
});

Full code of derby-app.js file:

var derby = require('derby'),
  app = derby.createApp(module);

app.get('/', function(page, model, params) {
  model.subscribe('message', function() {
    page.render();  
  })  
});  

Launching Hello World App

Now everything should be ready to boot our server. Execute node . or node index.js and open a browser at http://localhost:3001. You should be able to see something like this: http://cl.ly/image/3J1O0I3n1T46.

Derby + Express.js Hello World App

Passing Values to Back-End in Derby.js

Of course static data is not much, so we can slightly modify our app to make back-end and front-end pieces talks with each other.

In the server file index.js add store.afterDb to listen to set events on message attribute:

server.listen(3001, function(){
  model.set('message', 'Hello World!');
  store.afterDb('set','message', function(txn, doc, prevDoc, done){
    console.log(txn)
    done();
  }) 
});

Full code of index.js after modifications:

var http = require('http'),
  express = require('express'),
  derby = require('derby'),
  derbyApp = require('./derby-app');

var expressApp = new express(),
  server = http.createServer(expressApp);

var store = derby.createStore({
  listen: server
});

var model = store.createModel();

expressApp.
  use(store.modelMiddleware()).
  use(express.static(__dirname + '/public')).
  use(derbyApp.router()).
  use(expressApp.router);

server.listen(3001, function(){
  model.set('message', 'Hello World!');
  store.afterDb('set','message', function(txn, doc, prevDoc, done){
    console.log(txn)
    done();
  })   
});

In Derby application file derby-app.js add model.on() to app.ready():

  app.ready(function(model){
	    model.on('set', 'message',function(path, object){
	    console.log('message has been changed: '+ object);
	  })
  });

Full derby-app.js file after modifications:

var derby = require('derby'),
  app = derby.createApp(module);

app.get('/', function(page, model, params) {
  model.subscribe('message', function() {
    page.render();
  })
});

app.ready(function(model) {
  model.on('set', 'message', function(path, object) {
    console.log('message has been changed: ' + object);
  })
});

Now we’ll see logs both in the terminal window and in the browser Developer Tools console. The end result should look like this in the browser: http://cl.ly/image/0p3z1G3M1E2c, and like this in the terminal: http://cl.ly/image/322I1u002n38.

Hello World App: Browser Console Logs

Hello World App: Terminal Console Logs

For more magic in the persistence area, check out Racer’s db property. With it you can set up an automatic synch between views and database!

Let me know if you’re interested in any specific topic for future blog post and don’t forget to checkout my JavaScript books:

The full code of all the files in this Express.js + Derby Hello World app is available as a gist at https://gist.github.com/azat-co/5530311.

Day 6 of StartupBus 2013: All-Star Finals

On the sixth day six after departing from San Francisco, buspreneurs gathered to socialize and to let steam off at a happy hour organized by Twillio. GhostPost brought a projector to the bar to show their anonymous live chat. The funny story is that the projector was sold to them by of a fellow busmate (from the Grassroots.io team) whose Austin-based friend the Grassroots guy to handle the Craigslist projector ad. :-) Apparently the GhostPost team weren’t happy with their defeat (and who would be?) and hustled their way through the competition to become an All-Star wildcard — and by the evening they were indeed selected as a wildcard!

On the sixth day six after departing from San Francisco, buspreneurs gathered to socialize and to let steam off at a happy hour organized by Twillio. GhostPost brought a projector to the bar to show their anonymous live chat. The funny story is that the projector was sold to them by of a fellow busmate (from the Grassroots.io team) whose Austin-based friend the Grassroots guy to handle the Craigslist projector ad. :-) Apparently the GhostPost team weren’t happy with their defeat (and who would be?) and hustled their way through the competition to become an All-Star wildcard — and by the evening they were indeed selected as a wildcard!

StartupBus 2013 West Coast - random stop
StartupBus 2013 West Coast – random stop

Rackspace bought out the Champions sports bar in downtown Austin. GhostPost and other teams pitched during the evening to Dave McCure, Robert Scobler, and other important startup personas. The sound quality wasn’t very good. Every now and then somebody would shush the drunk, happy and tired crowd, but that didn’t help much. Despite it being very entertaining to watch Dave McClure rip apart startups and Robert (because he saw the evolution during the span of 3 days) explain and sometime defend them — in the end the decision was the same. They announced that the winner was CareerMob, and the runner up was NextChaptr.

StartupBus 2013 — Dave McClure and Rober Scoble
StartupBus 2013 — Dave McClure and Rober Scoble

Summary of StartupBus 2013

Overall StartupBus is a great experience but I can’t say that it has changed my life. :-) There are similarities to a real startup life:

  • Scarcity of resources, balance of risks and trade-offs, ample creativity to solve problems
  • Building, motivating and selling the concept to your team while pitching your idea to judges
  • Human drama: communication issues, interactions among team members in close quarters, under stress, without enough sleep, etc.
  • Startup lifestyle: exhaustion, abundance of stress

But some things are far from the reality. Mainly, in an actual startup:

  • Founders can focus not only on consumer segment, but also on small business and/or enterprise customers. Obviously, due to the lack of time and resource constraints buspreneurs targeted consumer audience.
  • Team needs to be serious and to pick not just fun and sexy ideas to get the most buzz. Solid business models usually come from ugly and boring, though concrete and painful problems.
  • These days, anyone who wants to start up a business has full-time reliable and even speedy Internet access without having to get stranded in the middle of a desert or having your mobile hot-spot picking up Mexican cell phone carriers. :-)

My conclusion is that a StartupBus trip was a good experience, but it’s not exactly the same as building a real startup.

Real StartupBus Tattoo by @claco
Real StartupBus Tattoo by @claco