Sunday, September 2

Rhino + Maven for JavaScript compression @ build time

AJAX-based Rich Internet Applications (RIAs) make heavy use of JavaScript. To improve the user experience of RIAs we can minimize the number of (JavaScript) file requests and reduce their size.

These concatenation and compression operations can be done on-the-fly or at build time.
On-the-fly techniques intercept requests and perform merging and/or compression in real-time. They are simple(r) to implement, often involving URL rewriting combined with a compress script, but:

  • don't scale, since compression costs grow proportionately to the number and size of the files;
  • add latency, by consuming CPU resources during (precious) server response time;
  • are harder to test and debug;
  • rely heavily on caching for continued performance improvements, when research indicates 40-60% of users browse with an empty cache;
  • don't work on all browsers

Compression at build time allows the use of more robust, often slower, compression methods (e.g.: a proper JavaScript interpreter instead of a regular expression) and/or combination of different methods (e.g.: run YUI Compressor after Rhino).
At build time we're unconstrained by the "real-time performance pressures".

Here's a step-by-step guide for a Maven/Ant build file using Mozilla's Rhino to serve pre-compressed JavaScript and significantly reduce application load time.

1.Setup
  1. Download the rhino.jar from Dojo ShrinkSafe
  2. Download, install, and configure Maven
  3. Copy the rhino.jar to your Maven repository
  4. Have a Maven-based Web-application ready

2.Add rhino.jar as a dependency to the POM

<dependency>
<groupId>rhino</groupId>
<artifactId>custom_rhino</artifactId>
<version>0.1</version>
<properties>
<war.bundle>true</war.bundle>
</properties>
</dependency>

3.Setup some useful variables on maven.xml


<!-- destination for the compressed JavaScript files -->
<j:set var="js.compression.dir" value="${typically_the_war_src_dir}"/>
<j:set var="js.compression.skip" value="false"/> <!-- enable/disable compression -->
<j:set var="js.compressor.lib.path" value=""/> <!-- path to the compressor jar -->

4.Fetch compressor and make it globally available

    <goal name="get-compressor" description="Set path to JavaScript compressor library">
<!-- get compressor from dependencies, to allow standalone use of goal -->
<j:if test="${empty(js.compressor.lib.path)}">
<ant:echo message="Fetching JavaScript compressor from repository" />
<j:forEach var="lib" items="${pom.artifacts}">
<j:if test="${lib.dependency.artifactId == 'custom_rhino'}">
<lib dir="${maven.repo.local}">
<include name="${lib.path}"/>
<j:set var="js.compressor.lib.path" value="${lib.path}"/>
</lib>
</j:if>
</j:forEach>
</j:if>
<ant:echo>
Path to JavaScript compressor library is ${js.compressor.lib.path}
</ant:echo>
</goal>


This way, get-compressor can be reused as a pre-requisite of all compression tasks.

5.Aggregate compression sub-tasks

<goal name="compress-js" description="Compress JavaScript across all site components">
<!-- Different sections might have different compression requirements -->
<j:if test="${js.compression.skip == 'false'}">
<attainGoal name="compress-site-js"/>
<attainGoal name="compress-forum-js"/>
...
</j:if>
</goal>


6.Compression task (to compress a single file)

<goal name="compress-site-js" prereqs="get-compressor" description="Compress JavaScript files for 'site'">
<ant:echo message="Compressing JavaScript files for 'site'" />
<j:set var="stripLinebreaks" value="true" />
<j:set var="js.dir" value="${path_to_javascript_files}"/>
<concat destfile="${js.dir}/site-concat.temp" force="yes">
<filelist dir="${js.dir}"
files="file_to_merge.js, another_file_to_merge.js, etc.js"/>
</concat>
<ant:java
jar="${js.compressor.lib.path}"
failonerror="true"
fork="yes"
output="${js.dir}/site-breaks.temp">
<ant:arg value="-c"/>
<ant:arg value="${js.dir}/site-concat.temp"/>
</ant:java>
<!-- move compressed files back from dest to src dir -->
<ant:move file="${js.dir}/site-breaks.temp" tofile="${js.dir}/site_c.js" filtering="true">
<j:if test="${stripLinebreaks == 'true'}">
<ant:echo message="Removing line breaks" />
<filterchain>
<striplinebreaks/>
</filterchain>
</j:if>
</ant:move>
<delete file="${js.dir}/site-concat.temp"/>
</goal>


This goal concatenates the 3 JavaScript files file_to_merge.js, another_file_to_merge.js,
and etc.js into a single (temporary) file site-concat.temp.
It then compresses site-concat.temp into site-breaks.temp (-breaks suffix indicates compressed file still contains line brakes).
Finally, it moves the content of site-breaks.temp into site_c.js (optionally removing line breaks) and cleans-up any temporary files and directories.

7.Compression task (to compress many independent files)

<goal name="compress-forum-js" prereqs="get-compressor" description="Compress JavaScript files for 'forum'">
<j:set var="stripLinebreaks" value="true"/>
<j:set var="src.dir" value="${path_to_javascript_files}"/>
<!-- Delete previously compressed files -->
<ant:delete>
<ant:fileset dir="${src.dir}" includes="*_c.js"/>
</ant:delete>
<ant:echo message="Compressing 'forum' JavaScript files from ${src.dir}" />
<!-- create temp dir for compressed files -->
<ant:mkdir dir="${src.dir}/_compressedjs"/>
<j:set var="dest.dir" value="${src.dir}/_compressedjs"/>
<!-- compile JavaScript files to compress -->
<ant:fileScanner var="forumJSFiles">
<ant:fileset dir="${src.dir}" casesensitive="yes">
<ant:include name="file_to_compress.js"/>
<ant:include name="another_file_to_compress.js"/>
<ant:exclude name="*_c.js"/>
</ant:fileset>
</ant:fileScanner>
<!-- loop through files and compress using compressor set in 'get-compressor' goal -->
<j:forEach var="jsFile" items="${forumJSFiles.iterator()}">
<ant:echo message="Compressing ${jsFile.name}" />
<ant:java
jar="${js.compressor.lib.path}"
failonerror="true"
fork="yes"
output="${dest.dir}/${jsFile.name}">
<ant:arg value="-c"/>
<ant:arg value="${src.dir}/${jsFile.name}"/>
</ant:java>
</j:forEach>
<!-- move compressed files back from dest to src dir -->
<ant:move todir="${src.dir}" filtering="true">
<ant:fileset dir="${dest.dir}" casesensitive="yes">
<ant:include name="*.js"/>
</ant:fileset>
<j:if test="${stripLinebreaks == 'true'}">
<ant:echo message="Removing line breaks" />
<filterchain>
<striplinebreaks/>
</filterchain>
</j:if>
<ant:mapper type="glob" from="*.js" to="*_c.js"/>
</ant:move>
<!-- delete temp dir -->
<ant:delete dir="${dest.dir}"/>
</goal>


This goal loops through a list of JavaScript files compressing them one by one.

8.Caveats

Rhino compression removes all comments, so beware of IE-specific conditional compilation statements such as the snippet below (taken from http://jibbering.com/2002/4/httprequest.html):


...
var xmlhttp=false;
/*@cc_on @*/
/*@if (@_jscript_version >= 5)
// JScript gives us Conditional compilation, we can cope with old IE versions.
// and security blocked creation of the objects.
try {
xmlhttp = new ActiveXObject("Msxml2.XMLHTTP");
} catch (e) {
try {
xmlhttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (E) {
xmlhttp = false;
}
}
@end @*/
...


There are many possible ways around this issue, namely:
  • Use Ant to concatenate the critical code sections with the compressed files after compression
  • Move all critical sections to a separate, uncompressed file
  • Move all critical sections inline
  • Fix the compressor and submit your patch to Dojo :)


9.Improvements

When I implemented the solution above, more than a year ago, a common complaint from fellow developers was that the compression task had the undesirable side-effect of slowing down the build -biiig time. A colleague pointed me towards Ant's Uptodate task, which I then used to implement conditional compression. This means files were only compressed if they had been changed. Conditional compression reduced the automated compression time from about 1 minute to 5 seconds on average.

To use conditional compression, replace the ellipsis in the script below with the content of any of the compression goals above.


<goal name="conditional-compress-js" prereqs="get-compressor"
description="Conditional compression of JavaScript files">
<j:set var="src.dir" value="${path_to_javascript_files}"/>
<!-- check timestamps to see if compression is required -->
<ant:echo message="Checking timestamps of JavaScript files from ${src.dir}" />
<ant:fileScanner var="jsFiles">
<ant:fileset dir="${src.dir}" casesensitive="yes">
<ant:include name="**/*.js"/>
<ant:exclude name="**/*_c.js"/>
</ant:fileset>
</ant:fileScanner>
<j:forEach var="jsFile" items="${jsFiles.iterator()}">
<ant:echo message="Checking last-modified-date of ${jsFile.name}" />
<uptodate property="js.modified" targetfile="${src.dir}/${jsFile.name}">
<srcfiles dir="${src.dir}" includes="**/*_c.js" />
</uptodate>
</j:forEach>
<j:if test="${js.modified}">
<j:set var="compression.required" value="true"/>
</j:if>
<j:if test="${compression.required}">
<ant:echo message="Modified JavaScript files. Compression required." />

<!-- insert compression block here -->
...
</j:if>
</goal>


10.Further (potential) Improvements
  • Clever use of Caching, to avoid unnecessary downloading of unmodified resources.
  • Request parameter trigger to alternate between compressed and uncompressed JavaScript in real-time, a feature that is very useful for testing and debugging.
  • Very clever use of Versioning to aid with caching; my friend and former colleague Robert, responsible for an ingenious solution, can write about it in a post of his own.
  • In specific and controlled situations JavaScript namespaces can be shortened with tools like Ant, e.g.: <replace file="${file.js}" token="the.long.api.namespace" value="__u._a"/>.
  • Clever use of HTTP compression


Comments

  • I dislike the verbosity (by the very nature of XML) and obtrusiveness in the build configuration file.
  • I believe the Maven plugin approach (in combination with Julien Lecomte's excellent YUI Compressor) is a better solution.
  • With a few changes the scripts above can also be used to compress CSS resources at build time.

(Vaguely) Related Posts:
Measuring client-side performance of Web-apps
CSS clean-up @ build time


References
Custom on-the-fly compression
Make your pages load faster by combining and compressing javascript and css files
Minify
pack:tag

On-the-fly server compression
gzip, where have you been all my life..?

Compression @ build time
YUI compressor
Maven plugin for the YUI Compressor

Relevant Articles
YUI Performance Research - Part 1
YUI Performance Research - Part 2
Minification v Obfuscation
Serving JavaScript Fast
Using The XML HTTP Request
Response Time: Eight Seconds Plus Or Minus Two
Optimizing Page Load Time
Speed Web delivery with HTTP compression

8 comments:

Anonymous said...

I like the maven2 plugin approache better as well. However, as you pointed out in your approach, it also wipes out conditional comments at the moment.

Unknown said...

Yeah, and it should be trivial to add conditional compression (quite useful!) and optional namespace obfuscation to the plugin code

Anonymous said...

Another free online tool for compressing javascript is http://www.compressjavascript.com

cheers,
blogging developer
http://www.bloggingdeveloper.com

Unknown said...

Thanks for that!
Cheers
j

Unknown said...

Hi!
In your interesting post you mention two ways to approach minification (on the fly and at build time).
I have created an open source library for java apps that uses a third approach: at-startup.
Its name is Jawr.

It minifies using JSMin or the YUI compressor, but it does more than that. You can combine (join) your scripts in any way you want, you can have .license files that solve the problem of license comment destruction, and you can switch from development mode (uncompressed and uncombined scripts) to production mode by changing a flag in a config file.


There is no XML to write (only a simple .properties file).
Finally, Jawr also provides a solution for most of the 'further improvements' you mention: each link has a generated version path segment, caching is used (304 returned for if-modified-since, long time Etags, etc.), and gzip is used in a configurable way.

Hope you find it interesting, regards.
Jordi.

Neil's Dev Stuff said...

or check out my blog and do it from the ide at build time....http://nnbs.blogspot.com

Mike said...

I recently wrote a mod_perl output filter which sits inside Apache. It intercepts requests for .css files and then “compresses” them on the fly before sending. It’s not gzip compression, what it does is strip whitespace, comments, newlines etc. Check it out here: Compressing CSS on the fly

Hipotecas con ASNEF o RAI said...

I’ve recently started a web site, the info you offer on this web site has helped me greatly. Thank you for all of your time & work. “The murals in restaurants are on par with the food in museums.” by Peter De Vries.