Speedy Browserifying with Multiple Bundles

NOTE: Much of the information here is outdated. Watchify eliminates the need for a vendor bundle entirely. Use it with Gulp via vinyl-source-stream.

Last month I talked about one of my favorite tools for JavaScript on the front end, Browserify, which allows you to create modular code for the browser using CommonJS modules and npm. It does this by combining the dependencies into an all-in-one bundle. In development, typically you will watch your JavaScript files for changes and then recompile the bundle.

If you're including some large dependencies in your bundle, though, you may have noticed that it can take several seconds to regenerate that bundle each time you edit a module. This usually isn't a big problem, but when you want to rapidly hack on a UI feature and frequently refresh to see your changes this several second display can end up seriously slowing you down.

One of the most effective things you can do to speed up this process is build multiple bundles - one for your application code and one for your dependencies. When your application code changes you only have to recompile that bundle, not all of your dependencies.

On the command line

If you're using the command line interface, creating an additional bundle is as easy as invoking browserify with the --require option. You don't have to point it at a source file, you can simply pass -r my-module for each module you want to include in the bundle. This will pull the requirement into the bundle and make that requirement available from outside the bundle with require('my-module').

$ browserify -r react -r q -r loglevel > build/vendor.js

If you include vendor.js from the example in your HTML, you can then require('react') or require('q') from anywhere. You may notice, however, that your application bundle is still pulling these requirements in, defeating the whole purpose. This brings us to the next command line option we need to use, --external.

The --external option tells browserify that the given module will be provided externally and doesn't need to be included in this bundle. When used with your application bundle, it will filter out the dependencies that you will be compiling into your vendor bundle.

$ browserify -x react -x q -x loglevel src/index.js > build/app.js

Make sure to include the vendor.js file before app.js in your HTML, so that the dependencies will be available when app.js is loaded.

With Gulp

Gulp has recently overtaken Grunt as our task manager of choice. We typically set Gulp up with our major tasks in a gulp/ subfolder, and then we require each of those modules in our gulpfile.js:

'use strict';
var gulp = require('gulp');

require('./gulp/app');
require('./gulp/serve');
require('./gulp/vendor');
require('./gulp/watch');

gulp.task('build', [
  'app',
  'vendor',
]);

gulp.task('default', ['build'], function() {
  return gulp.start('serve', 'watch');
});

The vendor bundle

I'll start with our vendor bundle, and break it up to explain a few things.

'use strict';
var browserify = require('gulp-browserify');
var gulp = require('gulp');
var rename = require('gulp-rename');
var uglify = require('gulp-uglify');

var libs = [
  'react',
  'react/lib/ReactCSSTransitionGroup',
  'react/lib/cx',
  'q',
  'underscore',
  'loglevel'
];

I assign an array of dependency names to a variable, because we'll be using this in a couple of places.

gulp.task('vendor', function() {
  var production = (process.env.NODE_ENV === 'production');

I'm using the NODE_ENV environment variable to determine whether this is a production build or not.

  // A dummy entry point for browserify
  var stream = gulp.src('./gulp/noop.js', {read: false})

Since Gulp is a file-based build system, it needs a file to open the stream with. We don't really need this on the vendor bundle, so my workaround is to point it to an empty file called noop.js.

    // Browserify it
    .pipe(browserify({
      debug: false,  // Don't provide source maps for vendor libs
    }))

    .on('prebundle', function(bundle) {
      // Require vendor libraries and make them available outside the bundle.
      libs.forEach(function(lib) {
        bundle.require(lib);
      });
    });

This is where the magic happens. The gulp-browserify plugin doesn't have an option to handle the --require command, so I simply listen for the "prebundle" event that it sends, and interact with browserify's API directly. The bundle.require() method is documented here. I iterate over the list of dependencies, and call bundle.require() for each one.

  if (production) {
    // If this is a production build, minify it
    stream.pipe(uglify());
  }

  // Give the destination file a name, adding '.min' if this is production
  stream.pipe(rename('vendor' + (production ? '.min' : '') + '.js'))

    // Save to the build directory
    .pipe(gulp.dest('build/'));

  return stream;

});

exports.libs = libs;

The rest of the task is pretty basic. I minify it if this is a production build, give the bundle an appropriate name, and save it to the build directory. I assign the list of dependencies to exports.libs so that it will be available to other modules, like our application bundle.

The application bundle

The application bundle follows a very similar pattern:

'use strict';
var browserify = require('gulp-browserify');
var gulp = require('gulp');
var libs = require('./vendor').libs;
var pkg = require('../package.json');
var rename = require('gulp-rename');
var uglify = require('gulp-uglify');

I import the list of dependencies that I exported from the vendor bundle with require('./vendor').libs.

gulp.task('app', function() {
  var production = (process.env.NODE_ENV === 'production');

  var stream = gulp.src('src/index.js', {read: false})

    // Browserify it
    .pipe(browserify({
      debug: !production,  // If not production, add source maps
      transform: ['reactify'],
      extensions: ['.jsx']
    }))

I include some settings for browserify, including a transform and additional extension definition for working with ".jsx" modules from React.

    .on('prebundle', function(bundle) {
      // The following requirements are loaded from the vendor bundle
      libs.forEach(function(lib) {
        bundle.external(lib);
      });
    });

Just as I did with the vendor bundle, I iterate over the list of dependencies. This time, however, I use bundle.external(). It's documented (briefly) here.

  if (production) {
    // If this is a production build, minify it
    stream.pipe(uglify());
  }

  // Give the destination file a name, adding '.min' if this is production
  stream.pipe(rename(pkg.name + (production ? '.min' : '') + '.js'))

    // Save to the build directory
    .pipe(gulp.dest('build/'));

  return stream;
});

The rest of the task is identical to the vendor bundle.

Watching for changes

Now, when something changes I can rebuild only the affected bundle. Here's an example of my watch.js:

'use strict';
var gulp = require('gulp');
var gutil = require('gulp-util');
var livereload = require('gulp-livereload');

gulp.task('watch', function() {
  var reloadServer = livereload();

  var app = gulp.watch('src/**/{*.js,*.jsx}');
  app.on('change', function(event) {
    gulp.start('app', function() {
      gutil.log(gutil.colors.bgGreen('Reloading...'));
      reloadServer.changed(event.path);
    });
  });

  var vendor = gulp.watch('node_modules/**/*.js');
  vendor.on('change', function(event) {
    gulp.start('vendor', function() {
      gutil.log(gutil.colors.bgGreen('Reloading...'));
      reloadServer.changed(event.path);
    });
  });

  gutil.log(gutil.colors.bgGreen('Watching for changes...'));
});

It's a little verbose, because I'm using the "change" event in order to start the task and then trigger a liveReload as a callback. It works great, though! On one of our applications, the vendor bundle takes 5 or 6 seconds to compile, but the app bundle takes less than a second. This makes active development quite speedy!

With Grunt

For Grunt, we like the load-grunt-config plugin that loads configuration blocks from modules in a grunt/ folder, similar to how we're handling Gulp above.

'use strict';
module.exports = function(grunt) {

  // Look for grunt config files in the 'grunt' directory
  require('load-grunt-config')(grunt);

  grunt.registerTask('default', [
    'browserify:vendor',
    'browserify:app',
    'watch'
  ]);
};

The browserify task

Both the app and vendor bundles are configured inside the browserify.js task file:

'use strict';

module.exports = {
  options: {
    debug: true,
    transform: ['reactify'],
    extensions: ['.jsx'],
    external: [
      'react',
      'react/lib/ReactCSSTransitionGroup',
      'react/lib/cx',
      'q',
      'underscore',
      'loglevel'
    ]
  },
  app: {
    files: {
      'build/app.js': ['src/app.js']
    }
  },

The app bundle uses the "external" config option from grunt-browserify. I add the app bundle settings as the default, top-level options because I sometimes have more than one bundle that uses very similar settings. I don't add the dependencies to an array, because - as you'll see in the next step - I can't reuse the array.

Here's the configuration for the vendor bundle:

  vendor: {
    // External modules that don't need to be constantly re-compiled
    src: ['.'],
    dest: 'build/vendor.js',
    options: {
      debug: false,
      alias: [
        'react:',
        'react/lib/ReactCSSTransitionGroup:',
        'react/lib/cx:',
        'q:',
        'underscore:',
        'loglevel:'
      ],
      external: null  // Reset this here because it's not needed
    }
  }
};

The bundle.require() API is exposed through grunt-browserify's "alias" configuration. The reason I can't reuse the array is because the plugin uses a colon to separate the module name from an optional alias (which corresponds to the "expose" property from browserify's bundle.require() method).

The watch task

The watch task uses grunt-contrib-watch, and the configuration is quite simple:

module.exports = {
  options: {livereload: true},
  app: {
    files: ['src/**/*.js', 'src/**/*.jsx'],
    tasks: ['browserify:app']
  },
  test: {
    files: ['node_modules/**/*.js'],
    tasks: ['browserify:vendor']
  }
};

Now you're ready to hack away, and your app bundle can be regenerated in a fraction of the time it took before!