(Neal Ford once told me about a bioengineering project that collects roughly 5 exabytes of data every few days, and as scientists they never want to throw anything away. Their simplest solution for storing it all may be to build a dedicated hard drive factory next door.)

The real issue, though, is not how much data you have. It’s what you do with it. How do you understand your situation when the data is voluminous, but may contain errors, omissions, and other problems?

This is not a new problem, and fortunately we have a great field of examples to use as a model. If you want to understand how to get insight from large amounts of data, look at baseball.

Even a brief summary of the history of sabermetrics (a term coined by Bill James himself, based on SABR — the Society for American Baseball Research) would go well beyond the scope of a simple blog post. If you’ve seen the Moneyball movie or read the book, you’ve got the general idea. Suffice it to say that sabermetrics is an attempt to understand how to win baseball games by focusing on the data, rather than on conventional wisdom.

Some of the interesting observations made by the stats community include ideas that seem blindingly obvious but were ignored for years:

1. Outs are everything, so don’t give them away unnecessarily. Baseball doesn’t have a time limit, and you can’t stall or play a prevent defense. You keep playing until 27 outs are recorded.

2. That means if you steal at less than about a 70% success rate, don’t run. It doesn’t matter how many bases you steal if you’re giving up too many by getting caught.

3. Similarly, bunting is done way, way too often. Moving a runner over from first to second (or sometimes from second to third) isn’t worth giving away an out. Or, as the stat guys say, if you play for one run, that’s likely all you’ll get, and most of the time not even that.

4. Pitcher wins are highly dependent on elements outside the pitcher’s control, like runs scored by his offense or errors made by the defense. Wins are a lousy measure of pitcher skill. Frankly, ERA (earned run average) isn’t that much better, because there are many subjective judgements in there, too (try looking at the difference between earned and unearned runs sometime and the arbitrariness of it all will make your head spin).

5. RBIs are so dependent on where you are in the batting order that they’re a lousy measure of hitting skill. If you are a major leaguer and you bat fourth, fifth, or sixth, you’re going to get a lot of RBIs no matter what you do. That’s why OBP (on base percentage, which measures how often you avoid giving up an out) is far better correlated with wins.

Incidentally, that was one of themes of the Moneyball movie. The idea was to get the players to get on base as much as possible, placing a higher than normal value on walks. Since OBP was much more important to wins than RBIs, the Oakland A’s could replace stars that left by filling in their on-base contributions rather than worrying about runs batted in.

As it turned out, they placed too little emphasis on defense. In the movie they moved Scott Hatteberg to first base where he had no defensive skills at all just to keep his bat in the lineup. Nowadays they would care more about the defense they gave up.

Actually, the real theme of Moneyball was to identify players that were undervalued based on traditional metrics and use them to build a successful team cheaply. At the time, OBP was a market inefficiency.

Rather than go on, I want to mention that there’s a reason I’m writing this post today. Tonight is the first game of the World Series between the Detroit Tigers and the San Francisco Giants, and from a sabermetric point of view it has all the elements of a potential classic.

I want to mention a couple baseball writers and what they’ve said about the upcoming series. First, Jonah Keri (author of The Extra 2%) wrote a great preview article today at Grantland discussing the Giants. Rany Jazayerli (a medical doctor cursed by following the Kansas City Royals) wrote a similar article on the Tigers. Both are well worth reading.

My favorite writer of all, however, is Joe Sheehan. He’s brilliant, and controversial, and fascinating to follow on Twitter. He writes a newsletter that I’m very happy to subscribe to, and his World Series preview came out today. I hope he won’t mind if I quote from it a bit, just to demonstrate the difference between mindlessly quoting statistics and drawing true insight from them.

The Tigers beat you by striking you out and not letting you exploit their poor defense.

He builds up to this by showing how the strike-out rate of the Detroit pitching staff is second in the league, and that they exceeded the league record in total strikeouts (coming in second to the Rays, actually). He also points out that their defense is so bad (they are the worst defense ever to reach the World Series) that the high strike out numbers are partly padded by having to face more batters than they should have had to, given the bad defense.

The Giants beat you by putting the ball in play and making you chase it.

The Giants were last in MLB in home runs. There were second in batting average and fourth on OBP. They have a park that suppresses homers and they play accordingly. On the other hand, they have a very low strike out rate. By the stat called equivalent average, they were the third best offense in baseball despite never hitting home runs.

As it turned out, the Tigers — after winning a weak division — caught two postseason opponents ill-equipped to take advantage of their poor defense.

The A’s struck out a lot and the Yankees were almost entirely driven by homers. The Giants will be completely opposite.

There’s plenty more, but this is the flavor of the observations. Note that none of the articles I mentioned are full of statistics. There are few choice stats presented with the goal of making a clear argument. That’s insight.

By the way, most organizations in baseball know this. The conflict between the scouts and the stats guys demonstrated in Moneyball is largely over (with the great exception being the Kansas City Royals, who not coincidentally keep coming in last). There’s one major group who still doesn’t “get it”, though, and that’s the media.

I cannot fathom why the networks continually put “analysts” in the broadcast booth who are completely unaware of the last twenty years of baseball research. In fact, they often disdain anything learned from studying the game as tricks with statistics. The greatest irony (as mentioned by Joe Sheehan many times) is that no stats guy is anywhere near as wedded to a particular metric as the so-called “traditionalists” care about RBIs or pitcher wins. If you watch the game broadcasts, you’ll see meaningless statistic after meaningless statistic based on small sample size (batter A is 2 for 7 against pitcher B) paraded out as though it meant anything, completely ignoring what’s now viewed as truly important.

For example, you’re guaranteed tonight to hear tons about the Giants’ “momentum” (which demonstrably doesn’t exist) or the effects of the Tigers’ long layoff since winning the ALCS, neither of which matter at all. Instead, the real story is whether the Giants can put enough balls in play to take advantage of the Tigers’ poor defense, or whether the Tigers’ great strike-out pitchers can keep them from doing it. It’s also whether the Tigers’ line up, which is dominated by a few stars, will be able to beat the Giants’ pitching, which has relatively weak starters but lots of bullpen depth, and the excellent San Francisco defense.

I have no vested interest in either team. Frankly, if it wasn’t for what I learn by reading those writers and others like them, I probably wouldn’t care at all. Now I’m excited to watch the interplay of two diametrically contrasting styles. Even better, I’m looking forward to the snarky comments by the great writers I follow on Twitter during the games (nobody does snark like a good baseball writer).

Do I have a prediction? Please. I’m just happy the Yankees got swept. That almost, but not quite, made up for the disaster that was the Red Sox season. Still, one of the Red Sox owners claims part of the team’s problem was that they didn’t listen to Bill James enough, so that gives me hope for next year.

Also, I’m really looking forward to the articles written after each game, which will be beautiful demonstrations of how to make decisions based on insight rather than just quoting statistics as though they were significant in themselves. I’ll just have the mute button ready whenever Tim McCarver starts talking.

—-

Joe Sheehan’s Newsletter is sent via email. An annual subscription is available at http://joesheehanbaseball.blogspot.com.

Twitter handles:
Joe Sheehan (@joe_sheehan)
Rany Jazayerli (@jazayerli)
Johan Keri (@jonahkeri)
Keith Law (@keithlaw)
Joe Posnanski (@JPosnanski)

Originally submitted at O’Reilly

Best Practices for Experienced Grails Developers

Programming Grails

Excellent look under the hood of Grails

Pros: Accurate, Helpful examples, Easy to understand

Best Uses: Intermediate, Expert

Describe Yourself: Developer, Educator

Yes, this “early access” edition is very early, but the content so far is fantastic. The Hibernate chapter is the best discussion of how Grails handles Hibernate anywhere, and Burt clearly discusses how and when to use the second-level cache.

If the book consisted of that chapter alone it would be worth the price, but the other available chapters are also great. I have every reason to believe this will be an outstanding book for experienced Grails developers.

(legalese)

About a week ago, someone contacted me about the source code from my book, Making Java Groovy. The source code is located at my GitHub repository, https://github.com/kousen/Making-Java-Groovy. The requester noted that I didn’t have any sort of license file on my code and wondered what the terms were.

Leaving aside the wonder that (1) somebody found the book code useful (ack! humblebrag alert!) and (2) he actually asked permission to use it, it occurred to me I really ought to have something in place for that eventuality. I asked my editor at Manning about it and she didn’t answer right away, so I interpreted that as freedom to do whatever I wanted.

A friend on a mailing list suggested that the Apache 2 license is appropriate if I don’t care too much how the code is used (and I don’t), so I decided to add that license to each source file. That brings me, at long last, to the original subject of this post: how do I add a license statement to the top of a large number of source files nested in many subdirectories?

I thought I would solve the problem with the eachFileRecurse method that Groovy adds to Java’s java.io.File class. I quickly realized, though, that there were directories I wanted to skip, and that lead me to the traverse method, which takes a Map of parameters.

Here’s the result:

import static groovy.io.FileType.*
import static groovy.io.FileVisitResult.*
String license = '''/* ===================================================
 * Copyright 2012 Kousen IT, Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 * ========================================================== */
'''
dir = '/Users/kousen/mjg'
new File(dir).traverse(
    type       : FILES, 
    nameFilter : ~/.*(java|groovy)$/,
    preDir     : { if (it.name == '.metadata') return SKIP_SUBTREE }) { file ->
    // only add license if not already there
    if (!file.text.contains(license)) {
        def source = file.text
        file.text = "$license$source"
    }
    assert file.text.contains(license)
}

I used static imports for the FileType and FileVisitResult classes. The FILES constant comes from FileType, and the SKIP_SUBTREE constant comes from FileVisitResult. The parameters I used return only files whose name ends in either ‘java’ or ‘groovy’ and aren’t in any directory tree including ‘.metadata’.

Ultimately everything is based on the getText and setText methods that the Groovy JDK adds to the java.io.File class. Both are called using the text property. The getText method returns all the existing source code, and the setText method acts as an alias for the write method, which automatically closes (and therefore flushes) the file when finished. I used a multiline string for the license and included the training carriage return, so writing the license followed by the source did the trick.

The documentation for these methods is, shall we say, a little thin. I therefore did what I normally do in these situations: I found the test case in the Groovy source distribution. The test in question is called FileTest and can be viewed in the Groovy GitHub repository here. The test cases showed how to use all of the methods, including traverse, so it was just a question of looking for the right example.

(Incidentally, one of the less publicized but really sweet features of GitHub is the code browser. Just find the project you want and dig into the directories until you find a file, and then GitHub provides syntax highlighting and everything. It’s a great, great feature, especially if you don’t want to clone the source of every project you care about onto your own local disk.)

Since I hadn’t known about the traverse method ahead of time, and I messed up the regular expressions for a while (sigh), solving the problem took longer than I expected. Still, it’s hard to beat a solution that takes less than a dozen lines. Hopefully someone else will find this helpful as well. And regarding the job interview situation described above, to paraphrase John Lennon in the rooftop concert at the end of Let It Be, on behalf of Groovy and myself, I hope I passed the audition.

Actually, my larger theme is that understanding Groovy (and the Groovy JDK in particular) is helped considerably by thinking about operator overloading. I don’t do a lot of operator overloading in my own code (present example excepted), but it appears all over the place in the Groovy library. I find it helps Java developers understand Groovy better if they know that whenever they see any operator, they should mentally convert it to a method. For example, + is the plus method, – is minus, and, better yet, [...] is getAt or putAt and even as is really asType.

This helps Java devs understand that by reading the groovydocs for String, they realize they can do things like:

String s = 'this is a string'
assert 'this' == s[0..3]
assert 'ing' == s[-3..-1]
assert 'gni' == s[-1..-3]
assert 'thisisastring' == s[0..3, 5..6, 8, -6..-1]
assert 'th is a string' == s - 'is'

and so on.

A groovy range is simply two values separated by a pair of dots, as shown in the previous example. Many classes can be used in a range, as in::

assert [1, 2, 3, 4, 5] == 1..5
assert ['a', 'b', 'c'] == 'a'..'c'
Date now = new Date()
Date then = now + 2
assert ['Jun 16', 'Jun 17', 'Jun 18'] == (now..then)*.format('MMM dd')

(Adjust the last one based on when you run the script, of course. And how cool is it that the Groovy JDK adds a format method to the Date class?)

As the most excellent book Groovy in Action points out (2nd edition available through the Manning Early Access Program), any class can be used in a range if it:

• implements the java.util.Comparable interface
• has a next method
• has a previous method

That’s all you need. I wanted to show an example of this, and based on all my traveling I decided to use train stations. Here’s my first pass at it:

class Station implements Comparable<Station> {
    String name
    Station next
    Station previous
    int position
    
    Station next() { next }
    Station previous() { previous }
    
    int compareTo(Station s) {
        this.position - s.position
    }
    String toString() { name }
}

My Station class is really a node in a doubly-linked list. It has a name and pointers to the next station and the previous station. To make the class implement Comparable, I decided to assign each station an integer position as I added it to a track.

My next step was to use put stations together. At first I was going to use an addStation method, and then I realized that’s really what the plus method was all about. So instead I did this:

    Station plus(Station s) {
       s.position = ++position
       s.previous = this
       this.next = s
       return s 
    }

Here’s a script using the Station class:

Station dc = new Station(name:'DC')
Station phl = new Station(name:'PHL')
Station nyc = new Station(name:'NYC')
Station bos = new Station(name:'BOS')

// operator overloading to make a route:
dc + phl + nyc + bos

// Stations are a range in each direction:
def northBound = (dc..bos)*.name
def southBound = (bos..dc)*.name

assert northBound == ['DC', 'PHL', 'NYC', 'BOS']
assert southBound == ['BOS', 'NYC', 'PHL', 'DC']

That’s all there is to it. I could easily add a minus method to Station in order to remove a node, and if I ever have to use this class in a real system I probably will. The position value is only used for comparison, so the actual number doesn’t matter, but I can imagine that if I have to add and remove lots of stations I’ll need some way to make that more rigorous. I also can’t escape the feeling that a better design would involve a Track class to hold the resulting route, but I didn’t need it for this simple demonstration.

Finally, those of us who occasionally travel the Acela route up and down the northeast corridor know that I left out a lot of stations, but I suppose I can dream that someday our trains will be in the same class (no pun intended) as their European or Asian counterparts.

(Obligatory marketing: My book Making Java Groovy, also available through the Manning Early Access Program, just went out for its 2/3rds review. I hope to be “text complete” by the end of the summer, for a dead treeware release just in time for the holiday gift-giving season.)

While it was annoying enough to set up a browser to surf the web, that’s not sufficient to access remote sites programmatically. For example, the client does a daily download of exchange rate data from a central site, which they process and store in a local db. I wanted to demonstrate that using Groovy.

Normally, to use a proxy I set the host and port on the command line. I’ve done that in Java (and Groovy) many times:

groovy -DproxyHost=10.x.x.x -DproxyPort=8080 myscript.groovy

Most of the time, that’s all you need. In this particular case, however, I also needed to submit a username and a password for authentication on the proxy server.

There are several sites that show you how to do that in Java. Here’s one of them, and it shows that you need to extend the java.net.Authenticator class and override the getPasswordAuthentication method. Here’s an example in Java:

import java.net.Authenticator;
import java.net.PasswordAuthentication;

public class MyAuthenticator extends Authenticator {
    @Override
    protected PasswordAuthentication getPasswordAuthentication() {
        return new PasswordAuthentication("username","password".toCharArray());
    }
}

Then, in your program, set the new authenticator as the default.

Authenticator.setDefault(new MyAuthenticator());

and you’re good to go.

Of course, I couldn’t leave it at that. I was teaching a Groovy class anyway, so I wanted to make the solution groovier. Here’s what I ultimately used:

Authenticator.default = {
    new PasswordAuthentication('username','password' as char[])
} as Authenticator

I switched from using the setDefault method to setting a property, and coerced a closure with the required method into the proper class. Since the authentication mechanism only calls the getPasswordAuthentication method, I can use the single closure as the implementation. Normally I use closure coercion for interfaces, and then generally if they only have a single method, but it was too easy in this case to ignore.

Besides, showing the simplicity of the Groovy solution made the demo a teachable moment, which at least tried to make some lemonade out of the paranoid security lemons. My favorite part was how I hard-wired both the username and password directly into the script, in clear text no less. I could have found a way around that, but I was on a guest account anyway and it felt nicely subversive to do so.

Lately there have been several episodes in the IT industry of boys behaving badly. A recent article in Mother Jones has a good summary. In short, a few guys in this industry have had a tendency to do and say things that have been particularly insensitive to women, for a variety of reasons ranging from simple foolishness to (potentially) outright bias.

Maybe this isn’t terribly surprising. After all, geeks aren’t always the most agile of social butterflies, and one of the downsides of getting guys together in groups is that the collective IQ tends to be lower than the individual ones. Still, I don’t want to rant about that, especially since lamenting poor behavior is almost as much of a cliche as the behavior itself.

Instead, I want to focus on the Groovy programming language, and a decision made during its early formation that could easily have been avoided if more women developers had been available.

As anyone who works with Groovy knows, the language has two types of strings. One uses single-quotes ('this is a string') and represents and instance of regular old java.lang.String. The other uses double-quotes ("this is a string with a ${variable}"), and allows for variables to be interpolated into it. The latter is a really useful class and comes up all the time.

By all the standard conventions, the class representing double-quoted strings should be called GroovyString. There are thirty or forty different classes in the Groovy API that start with the word Groovy, including such common classes as GroovyObject, GroovyShell, GroovyServlet, and GroovyTestCase. A GroovyString class would feel right at home there.

Unfortunately, though, the original development team decided to call the class representing double-quoted strings GString, because they thought it was funny. Even worse, since you substitute values into it using a $, the inevitable follow-up joke is that you do string interpolation by “putting a dollar in a GString”.

Now, I’m sure no malice was intended at the time. It was a joke, and I chuckled the first time I heard it too. So have many women I know when they first hear it. It’s actually a pretty clever joke.

But that doesn’t mean a name like that belongs in the standard library.

I think of it this way. The name of that class is yet one more reason why we need more women in this field. I’m sure that if a woman had been part of the core Groovy team at the time, she would have said something like, “ha ha, very funny, guys, but let’s not hard-wire that gag into the standard library, okay? You know, the same library that’s going to be used by every Groovy developer ever?” I fully expect that the rest of the team would have thought about it and almost certainly (if sheepishly) agreed. After all, it’s the kind of joke that’s funny once, and only for a few minutes. After that it’s mostly a nuisance. It’s hard enough to get a language named Groovy taken seriously by the Fortune 500 without going there, too.

Please don’t get me wrong, though. I’m not blaming or condemning anybody. That’s the sort of gag that comes up all the time. The mistake was making it part of the standard. Clearly there wasn’t a woman available at the time to provide some perspective.

I also draw a strong distinction between silly mistakes and outright bias. Everybody makes mistakes. I make them all the time. I try not to say anything offensive in class (or in life), partly because I really do try to respect people different from myself, and partly because I’m pretty much of a coward and don’t want to get in trouble. I certainly don’t want to make anyone uncomfortable for no good reason.

Oversensitivity leads to its own problems, too. I was an undergrad at MIT back in the 80′s. Despite the popular image of the place, the actual male/female ratio there was about 55%/45%, which wasn’t all that bad despite my complaints at the time. (The women there used to joke that the ratio of “acceptable” men to women was closer to 50/50.) The women at MIT tended to be pretty strong personalities, too, which I always thought was a good thing. I’ve always welcomed the company of women I could take seriously, or men too for that matter. The problem was that like so many academic environments, MIT had a marked tendency to overreact to any accusation of bias. Once someone complained of bias, rationality went out the window and any hope of getting to the actual truth was completely lost.

By the way, I believe this is partly what happened in the late 80′s and early 90′s, when oversensitivity to bias was labeled “political correctness” and summarily dismissed. I think we’re still paying the price for that.

That’s really unfortunate, too, because every woman I’ve ever met, without exception, has a story to tell that qualifies as bias or even harassment by any reasonable definition. Sometimes all you have to do is ask to hear stories that will make you shudder. Overreactions desensitize people against actual injustices that should cause outrage.

I don’t think the Groovy class in question qualifies on that scale. It’s a mistake, in my opinion, and a natural result of having too few women in the field. I’m not calling for a movement to rename the class or criticize anyone who uses the established name. But, then again, I’m not a woman. It’s entirely possible I don’t “get it”.

As a result, from now on I’m going to call double-quoted strings GroovyStrings. It’s easy enough for me to do, and while it may not change anything, hopefully it will make some women developers feel slightly more welcome. If you want to join me, great, but if not, that’s fine too. I’ve made my decision, though, and hopefully this post will make my reasons clear.

The appeal of groovlets is both their simplicity and the environment in which they run. They’re simple because they’re just Groovy scripts. The convenience comes from the environment, which has a whole set of implicit variables available, like params for the request parameters, headers for the request headers, and the normal request, response, session, and application references.

The way groovlets work is from a combination of classes in Groovy called groovy.servlet.GroovyServlet and groovy.lang.ServletBinding. The GroovyServlet instantiates a GroovyScriptEngine and a ServletBinding, sets up the implicit variables, finds the groovlet, and then executes it. That’s why groovlets are deployed as simple scripts, in source code form, rather than as compiled classes. They’re loaded and executed by the GroovyScriptEngine inside the GroovyServlet.

Here’s my simple example, taken from my Groovy Baseball application (less than 30 days to baseball season and counting). First I set up the GroovyServlet in the usual way.

  <servlet>
      <servlet-name>GroovyServlet</servlet-name>
      <servlet-class>groovy.servlet.GroovyServlet</servlet-class>
  </servlet>
  <servlet-mapping>
      <servlet-name>GroovyServlet</servlet-name>
      <url-pattern>*.groovy</url-pattern>
  </servlet-mapping>

Here is the groovlet that I use in the application, called GameService.groovy:

import beans.GameResult
import beans.Stadium
import service.GetGameData

def month = params.month
def day = params.day
def year = params.year

m = month.toInteger() < 10 ? '0' + month : month
d = day.toInteger() < 10 ? '0' + day : day
y = year

results = new GetGameData(month:m,day:d,year:y).games
response.contentType = 'text/xml'
html.games {
    results.each { g ->
        game(
            outcome:"$g.away $g.aScore, $g.home $g.hScore",
            lat:g.stadium.latitude,
            lng:g.stadium.longitude
        )
    }
}

The classes GameResult, Stadium, and GetGameData are internal to my application. For a given date, the application goes to the corresponding games directory maintained by Major League Baseball, determines which games were played that day, and downloads the box scores for each game in XML form. The main processor then extracts the results and creates a GameResult, which holds the home team name and score, the away team name and score, and a Stadium POGO that I use to hold the address and latitude and longitude.

The interesting part, or at least the part that lead to this blog post, is the last part of the script. The groovlet uses the built-in MarkupBuilder, called html, to write out the results in XML form.

Groovlets have had a built-in MarkupBuilder from the very beginning. The builder generates XML and writes the results directly to the output writer. Some of the early examples used the MarkupBuilder to write out a whole web page, but we’ve known that was a bad idea for years. The builder does have an excellent use case, though, which is to transform Java or Groovy objects into a format that can be read by other languages.

My web page builds a Google Map from the results which contains a dot, centered at the home stadium, for each game*.

*Real map geeks called this process “red dot fever”.

Here’s a picture from today, showing yesterday’s pre-season results.

Why write out my data in XML? The Google Maps API is written in JavaScript, and my result objects are in Groovy. JavaScript can’t read Groovy (or Java) objects, so I need a format to go between them, and hey, everybody can read and write strings. Even better, every language has an XML parsing and generation library. The Document Object Model is a W3C specification, after all. The presence of the MarkupBuilder in groovlets makes the process of converting my Groovy objects into XML trivial, and the resulting XML elements are sufficiently simple that parsing them in the JavaScript code is very easy.

Still, if I was writing this application today, I’d probably prefer JavaScript Object Notation (JSON) instead. JSON objects are native JavaScript, so the JavaScript code can deal with them immediately without further parsing. The problem is that groovlets don’t have a built-in JSON builder.

At least, they didn’t used to. The groovy.json.JsonBuilder class was added in Groovy 1.8. That means I can just import the class into my groovlet, instantiate the builder, and write out the data in JSON format as easily as I currently do in XML format. I suddenly realized, however, that other people might like that capability, it might be easy to implement, and then — here’s the cool part — I could make my first actual commit to the Groovy code base.

I therefore dug into the implementation of GroovyServlet and ServletBinding, and discovered that all I would need to do is to modify about half a dozen lines of code in ServletBinding.java and update the corresponding test class.

The fact that the ServletBinding class is implemented in Java is significant (whoa, holy foreshadowing, Batman).

I was inspired by Tim Yates’s great blog post showing exactly how to contribute to the language. I therefore forked the Groovy repository, created a local branch for my changes, added what I needed, and ran the test case. It took more work than I expected, but I finally got the tests to pass. As Tim suggested, I then raised a JIRA issue and sent a pull request from my GitHub repository.

Today I got my reply, from Guillaume Laforge himself (the head of the Groovy project, and a very nice guy, as his comment will show). Here is what he said:

“I had to modify the pull request a bit, as JsonBuilder being a Groovy class, it’s compiled after ServletBinding.
So on a clean build, the build was broken because of that.
I used a trick of “loading” the class lazily with this.getClass().getClassLoader().loadClass(“groovy.json.JsonBuilder”) which is not ideal – ideally, the Groovy build should compile with the joint compiler for the core sources.”

In other words, two things happened:
1. My cool addition of an implicit object called json of type JsonBuilder was accepted. (Whoo hoo!)
2. I broke the freakin’ build. Yup, the whole groovy core build failed, because of me. (Hangs head in shame)

Okay, I didn’t realize the Groovy core was built by compiling the Java sources first, and then the Groovy sources. I always use the joint compiler for my combined projects and never have an issue like that. So it’s not really my fault.

(Except I should have run the full build and discovered this. Sigh.)

In the end, though, Guillaume just fixed it, thanked me for my contribution (whoo hoo! again), and moved on. Hey, I’m now a contributor to the Groovy language! Combined with my one commit to Grails (thanks to Jeff Brown for that one), I now have contributed to the two open source projects I care most about. And this one I did all by myself (obviously, or I wouldn’t broken the stupid build — see what a mixed result this is?).

For the record, here’s what my groovlet would look like with the new json object available:

// ...stuff from before...
response.contentType = 'application/json'
json.games {
    results.each { g ->
        game(
            outcome:"$g.away $g.aScore, $g.home $g.hScore",
            lat:g.stadium.latitude,
            lng:g.stadium.longitude
        )
    }
}
out << json

Unlike the MarkupBuilder, JsonBuilder doesn’t automatically output its results, so I have to send it to the output writer myself.

I have some cool integration tests in the project, run by Gradle, but that’s another blog post. If you want to see the source code for the Groovy Baseball application, it’s part of the GitHub repository for my book. The application is in the ch02 directory (called “Groovy by Example” in the book) in the groovybaseball directory.

I’m not exactly sure which version of Groovy will have the json implicit object in groovlets. It’s part of the current build, which I suppose means Groovy 2.0. Anyway, feel free to use it, and if you do you can both thank me and laugh at me at the same time.

Consider a trivial POGO (Plain Old Groovy Object) called Course:

class Course
    String name
    int days
    String toString() { "($name,$days)" }
}

The goal was to take a collection of courses and sort it by the number of days. That’s really easy in Groovy:

def courses = [
    new Course(name:'Groovy',days:4),
    new Course(name:'Grails',days:3),
    new Course(name:'Spring',days:4),
    new Course(name:'Hibernate',days:3)
]

assert courses.toString() == '[(Groovy,4), (Grails,3), (Spring,4), (Hibernate,3)]'

courses.sort { it.days }

assert courses*.days == [3, 3, 4, 4]

The sort method in the java.util.Collection class is part of the Groovy JDK, meaning it’s one of the methods Groovy adds to the standard Java libraries. It takes a closure of either one or two arguments. In this case, I’m using the one-argument closure, which is used to select a property on which to base the sort. By specifying it.days in the closure, I’m telling the sort method to sort the courses based on their days property. Then I verify that the sort worked by checking that the courses are the right order, using the spread-dot operator to just look at the number of days.

In class the question that always comes up is, can I sort by days and then by name? In other words, if two courses have the same number of days, can I then sort by the name property?

That’s what the two-argument closure on the sort method is for. The two arguments are references to any pair of courses, and the closure should return a negative number, zero, or positive number according to whether the first course is less than, equal to, or greater than the second.

Here’s where things get amusing. The sort I want is:

courses.sort { a,b ->
    a.days <=> b.days ?: a.name <=> b.name
}

assert courses.toString() == '[(Grails,3), (Hibernate,3), (Groovy,4), (Spring,4)]'

The body of the closure on sort uses the spaceship operator <=>, which returns -1, 0, or 1 depending on whether the left side is less than, equal to, or greater than the right side. I use spaceship to compare the days properties. Then I add the Elvis operator ?: which means if the days comparison is not zero, use it, but otherwise use the following comparison, which uses another spaceship to compare by name.

It’s only after writing the code in class that one of the students pointed out that I had Elvis in between two spaceships, leading to the following observations:

1. The spaceships are there to return Elvis to his home planet
2. It takes two spaceships, working in tandem, to carry Elvis away, in much the same way two swallows can carry a coconut in tandem
3. Therefore, the Elvis being carried away must be the fat Elvis from the 70s, rather than the thin, cool Elvis from the 50s

Either way, after the sort is finished, Elvis has left the building.

Thank you, thank you very much.

Here’s the issue: I have a class that uses one of Google’s web services, and I want to test my class even when I’m not online. That means I need to mock the dependency, which isn’t all that hard. The problem is that there’s no explicit way to get my mock object into my own service. Yet, with Groovy’s MockFor and StubFor classes, I can mock a dependency, even when it’s instantiated as a local variable inside my class.

Let me show you the code. I’ll start with a simple POGO called Stadium:

class Stadium {
    String street
    String city
    String state
    double latitude
    double longitude
    
    String toString() {
        "($street,$city,$state,$latitude,$longitude)"
    }
}

I use this in my Groovy Baseball application, which accesses MLB box scores online and displays the daily results on a Google Map. The Stadium class holds location data for an individual baseball stadium. When I use it, I set the street, city, and state and have the service compute latitude and longitude for me.

My Geocoder class is based on Google’s restful geocoding service.

class Geocoder {
    String base = 'http://maps.google.com/maps/api/geocode/xml?'
   
    void fillInLatLng(Stadium stadium) {
        String urlEncodedAddress =
                [stadium.street, stadium.city, stadium.state].collect {
                    URLEncoder.encode(it,'UTF-8')
                }.join(',+')
        String url = base + [sensor:false, address:urlEncodedAddress].collect { it }.join('&')
        def response = new XmlSlurper().parse(url)
        String latitude = response.result.geometry.location.lat[0] ?: "0.0"
        String longitude = response.result.geometry.location.lng[0] ?: "0.0"
        stadium.latitude = latitude.toDouble()
        stadium.longitude = longitude.toDouble()
    }
}

First I take the stadium’s street, city, and state and add them to a list. Then the collect method is used to apply a closure to each element of the list, returning the transformed list. The closure runs each value through Java’s URLEncoder. In looking at the Google geocoder example, I see that they separate the encoded street from the city and the city from the state using “,+“, so I do the same using the join method.

The Google geocoding service requires a parameter called sensor, which is true if the request is coming from a GPS-enabled device and false otherwise. In the Groovy map, I set its value to false, and set the value of the address parameter to the string from the previous line. The collect method on the map converts each entry to “key=value“, so joining with an ampersand and appending to the base value gives me the complete URL for the stadium.

Most restful web services try to provide their data in a format requested by the user. The content negotiation is usually done through an “Accept” header in the HTTP request, but in this case Google does something different. They support only XML and JSON output data, and let the user select which one they want through separate URLs. The base URL in the service above ends in xml. Google lists the JSON version as preferred, but that’s no doubt because they expect the requests to come through their own JavaScript API. I’m making the request using Groovy, and the XmlSlurper class makes parsing the result trivial.

(Since Groovy 1.8, the JsonSlurper class also makes parsing JSON trivial, and I have a version that does that, too, but the difference really isn’t significant here.)

The resulting block of XML that is returned by the service is fairly elaborate, but as you can see from my code, the data is nested through the elements GeocodeResponse/result/geometry/location/lat and GeocodeResponse/result/geometry/location/lng. After invoking the parse method (which returns the root element, GeocodeResponse), I just walk the tree to get the values I want.

I do have to protect myself somewhat. The Google service isn’t nearly as deterministic as I would have expected. Sometimes I get multiple locations when I search, so I added the zero index to make sure I always get the first one. Also, some time during the last year Google decided to start throttling their service. I have a script that uses the Geocoder for all 30 MLB stadiums, and unfortunately it runs too fast (!) for the Google limits. I know this because I start getting back null results if I don’t artificially introduce a delay. That’s why I put in the Elvis operator with the value 0.0 if I don’t get a good answer.

So much for the service; now I need a test. If I know I’m going to be online, I can write a simple integration test as follows:

import static org.junit.Assert.*;
import org.junit.Test;

class GeocoderIntegrationTest {
    Geocoder geocoder = new Geocoder()
    
    @Test
    public void testFillInLatLng() {
        Stadium google = new Stadium(street:'1600 Ampitheatre Parkway',
            city:'Mountain View',state:'CA')
        geocoder.fillInLatLng(google)
        assertEquals(37.422, google.latitude, 0.01)
        assertEquals(-122.083, google.longitude, 0.01)
    }
}

I’m using Google headquarters, because that’s the example in the documentation. I invoke my Geocoder’s fillInLatLng method and check the results within a hundredth of a degree.

(The fact that the Google geocoder returns answers to seven decimal places is evidence that their developers have a sense of humor.)

Now, finally, after all that introduction, I reach the subject of this post. What happens if I’m not online? More to the point, how do I test the business logic in the fillInLatLng method without requiring access to Google?

What I need is a mock object, or, more precisely, a stub.

(A stub stands in for the external dependency, called a collaborator. A mock does the same, but also validates that the caller accesses the collaborator’s methods the right number of times in the right order. A stub helps test the caller, and a mock tests the interaction of the caller with the collaborator, sometimes called the protocol. Insert obligatory link to Martin Fowler’s “Mocks Aren’t Stubs” post here.)

In my Geocoder class, the Google service is accessed through the parse method of the XmlSlurper. Even worse (from a testing point of view), the slurper is instantiated as a local variable inside my fillInLatLng method. There’s no way to isolate the dependency and set it from outside, either as an argument to the method, a setter in the class, a constructor argument, or whatever.

That’s where Groovy’s StubFor (or MockFor) class comes in. Let me show it in action. First, I’ll set up the answer I want.

String xml = '''
    <root><result><geometry>
        <location>
            <lat>37.422</lat>
            <lng>-122.083</lng>
        </location>
    </geometry></result></root>'''
    
def correctRoot = new XmlSlurper().parseText(xml)

The xml variable has the right answer in it, nested appropriately. I use the XmlSlurper to parse the text and return the root of the tree I want.

Now I need a Stadium to update. I deliberately put in the wrong street, city, and state to make sure that I only get the right latitude and longitude if I insert the stub correctly.

Stadium wrongStadium = new Stadium(
    street:'1313 Mockingbird Lane',
    city:'Mockingbird Heights',state:'CA')

(Yes, that’s a pretty obscure reference. For details, see here. And yes, I’m old. If you’re in the right age group, though, you now have the show’s theme song going through your head. Sorry.)

The next step is to create the stub and set the expectations.

def stub = new StubFor(XmlSlurper)
stub.demand.parse { correctRoot }

The StubFor constructor takes a class as an argument and builds a stub around it. Then I use the demand property to say that when the parse method is called, my previously computed root is returned rather than actually going over the web and parsing anything.

To put the stub into play, invoke its use method, which takes a closure:

stub.use {
    geocoder.fillInLatLng(wrongStadium)
}

That’s all there is to it. By the magic of Groovy metaprogramming, when I invoke the parse method of the XmlSlurper — even when it’s instantiated as a local variable — inside the use block the stub steps in and returns the expected value.

For completeness, here’s the complete test class:

import static org.junit.Assert.*
import groovy.mock.interceptor.StubFor
import org.junit.Test

class GeocoderUnitTest {
    Geocoder geocoder = new Geocoder()
    
    @Test
    public void testFillInLatLng() {
        Stadium wrongStadium = new Stadium(
            street:'1313 Mockingbird Lane',
            city:'Mockingbird Heights',state:'CA')
        
        String xml = '''
        <root><result><geometry>
            <location>
                <lat>37.422</lat>
                <lng>-122.083</lng>
            </location>
        </geometry></result></root>'''
        
        def correctRoot = new XmlSlurper().parseText(xml)
        
        def stub = new StubFor(XmlSlurper)
        stub.demand.parse { correctRoot }
        
        stub.use {
            geocoder.fillInLatLng(wrongStadium)
        }
        assertEquals(37.422, wrongStadium.latitude, 0.01)
        assertEquals(-122.083, wrongStadium.longitude, 0.01)
    }
}

Since I’m only calling one method and only calling that method once, a stub is perfectly fine in this case. I could invoke the verify method if I wanted to (MockFor invokes verify automatically but StubFor doesn’t), but it doesn’t really add anything here.

There is one limitation to this technique, which only comes up when you’re doing the sort of Groovy/Java integration that is the subject of my book. You can only use StubFor or MockFor on a class written in Groovy.

All this code is available in the book’s Github repository at http://github.com/kousen/Making-Java-Groovy. This particular example is part of Chapter 5: Testing Java and Groovy.

As a final note, I should say that I dug into all of this, worked out all the details, and tried it in several examples. Then I finally did what I should have done all along, which was look it up in Groovy In Action. Of course, there it was laid out in detail over about five pages. Someday that will stop happening to me.

As a first attempt, consider the following simple example that adds the fyi and omg methods to java.util.logging.Logger:

import java.util.logging.Logger

Logger.metaClass.fyi = { msg -> delegate.info msg }
Logger.metaClass.omg = { msg -> delegate.severe msg }

For those who haven’t used Groovy much, the metaClass property is associated with every class in Groovy, and allows you to add methods and properties to the class. Here the fyi method is defined by assigning it to a one-argument closure whose implementation is to invoke the (existing) info method in Logger, with the msg argument. Likewise, omg is assigned to the severe method. Therefore, an invocation like:

Logger log = Logger.getLogger(this.class.name)
log.fyi 'for your information'
log.omg 'oh my goodness'

results in

Dec 12, 2011 10:09:02 PM java_util_logging_Logger$info call
INFO: for your information
Dec 12, 2011 10:09:02 PM java_util_logging_Logger$severe call
SEVERE: oh my goodness

The methods work, but the output isn’t really what I want. The messages get passed through, but the output shows INFO and SEVERE rather than FYI and OMG.

It turns out it takes a bit of work to define a custom log level. Levels are defined using the java.util.logging.Level class, which predefines levels like Level.INFO, Level.WARNING, and Level.SEVERE. The Level class has a protected constructor which can be used to make new levels. I therefore adding a class called CustomLevel, as follows:

import java.util.logging.Level

class CustomLevel extends Level {
    CustomLevel(String name, int val) {
        super(name,val)
    }
}

Each level gets an integer value. On my Windows 7 system (sorry) using JDK 1.6, the actual values of some of the defined levels are:

import java.util.logging.Level

println "$Level.INFO: ${Level.INFO.intValue()}"
println "$Level.WARNING: ${Level.WARNING.intValue()}"
println "$Level.SEVERE: ${Level.SEVERE.intValue()}"

INFO: 800
WARNING: 900
SEVERE: 1000


My second attempt was then to define a Groovy category, so that I could replace a couple of the existing levels in a controlled fashion.

import java.util.logging.Level
import java.util.logging.Logger

class SlangCategory {
    static String fyi(Logger self, String msg) {
        return self.log(new CustomLevel('FYI',Level.INFO.intValue()),msg)
    }
    static String lol(Logger self, String msg) {
        return self.log(new CustomLevel('LOL',Level.WARNING.intValue()),msg)
    }
}

Logger log = Logger.getLogger(this.class.name)
use(SlangCategory) {
    log.fyi 'this seems okay'
    log.lol('snicker')
}

Each of the logging methods in the Logger class (like info() or warning()) delegate to the log() method, which takes two arguments — an instance of Level, and a message String. I therefore used the category to replace the INFO and WARNING levels with FYI and LOL. The output is now:

Dec 12, 2011 10:20:29 PM sun.reflect.NativeMethodAccessorImpl invoke0
FYI: this seems okay
Dec 12, 2011 10:20:29 PM sun.reflect.NativeMethodAccessorImpl invoke0
LOL: snicker

Once again, this is just replacing existing levels, though it does at least have the new level name in the output string.

To really do this right, though, I wanted to be able to define new levels arbitrarily without having to hardwire them. That meant overriding the methodMissing method in the metaClass, using what Jeff Brown describes as the “intercept, cache, invoke” pattern for metaprogramming. Here’s the result, which I’ll explain after the code.

import java.util.logging.*

Logger.metaClass.methodMissing = { String name, args ->
    def impl = { Object... varArgs ->
        int val = Level.WARNING.intValue() +
            (Level.SEVERE.intValue() - Level.WARNING.intValue()) * Math.random()
        def level = new CustomLevel(name.toUpperCase(),val)
        delegate.log(level,varArgs[0])
    }
    Logger.metaClass."$name" = impl
    impl(args)
}

Logger log = Logger.getLogger(this.class.name)
log.wtf 'no effin way'
log.whoa 'dude, seriously'
log.rofl "you're kidding, right?"

The methodMissing method of the metaClass takes two arguments: the name of the method, and the arguments passed to it. Whenever you invoke a method that doesn’t exist, methodMissing gets invoked. That’s the “intercept” part.

The implementation is to define a closure that takes any number of arguments. Inside the closure, I computed a random value between Level.WARNING and Level.SEVERE, and then used that value to instantiate a custom level. The defined level and the new value were used in the log method on the closure’s delegate property (in this case, the logger) to log the message at the new level.

Finally, the "$name" method to the metaClass (which evaluates the name variable — otherwise the method added would just be called name) is assigned to the impl closure. That’s the “cache” part. Finally, the implementation is called, which is the “invoke” part.

Now I can use a log level with whatever name I want. The output of this script is
Dec 12, 2011 10:31:35 PM sun.reflect.NativeMethodAccessorImpl invoke0
WTF: no effin way
Dec 12, 2011 10:31:35 PM sun.reflect.NativeMethodAccessorImpl invoke0
WHOA: dude, seriously
Dec 12, 2011 10:31:35 PM sun.reflect.NativeMethodAccessorImpl invoke0
ROFL: you're kidding, right?

The demos are nice, but this really ought to be tested. I’m reasonably comfortable with this level (snicker) of metaprogramming, but I’d feel a lot better if I had a real test for it.

That took a fair amount of digging. It turns out that the inimitable Dierk Koenig (lead author of Groovy in Action, known as #regina on Twitter; second edition available now through the Manning Early Access Program) wrote a class called groovy.lang.GroovyLogTestCase. That class has a static method called stringLog. The GroovyDocs say:

“Execute the given Closure with the according level for the Logger that is qualified by the qualifier and return the log output as a String. Qualifiers are usually package or class names. Existing log level and handlers are restored after execution.”

It took me a while figure out how to use that, but eventually I got it to work. It automatically captures the console appender output, so the resulting test looks like this:

import java.util.logging.Level
import java.util.logging.Logger

class LoggingTests extends GroovyLogTestCase {
    String baseDir = 'src/main/groovy/metaprogramming'
    
    void testWithoutCustomLevel() {
        def result = stringLog(Level.INFO, without_custom_levels.class.name) {
            GroovyShell shell = new GroovyShell()
            shell.evaluate(new File("$baseDir/without_custom_levels.groovy"))
        }
        assert result.contains('INFO: for your information')
        assert result.contains('SEVERE: oh my goodness')
    }
    
    void testSlangCategory() {
        def result = stringLog(Level.INFO, use_slang_category.class.name) {
            GroovyShell shell = new GroovyShell()
            shell.evaluate(new File("$baseDir/use_slang_category.groovy"))
        }
        assert result.contains('FYI: this seems okay')
        assert result.contains('LOL: snicker')
    }

    void testEMC() {
        def result = stringLog(Level.INFO, use_emc.class.name) {
            GroovyShell shell = new GroovyShell()
            shell.evaluate(new File("$baseDir/use_emc.groovy"))
        }
        assert result.contains('WTF: no effin way')
        assert result.contains('WHOA: dude, seriously')
        assert result.contains("ROFL: you're kidding, right?")
    }
}

I should mention a couple of minor points. First, the class associated with a script has the same name as the file containing it, so the classes in the stringLog method are the script names. Second, in case you were wondering, the emc part of the script name stands for ExpandoMetaClass.

I spent a very pleasant evening working on this, and learned a few things:
1. Groovy metaprogramming is fun,
2. Now I know how to use GroovyLogTestCase, which is not at all well documented, and
3. I’ll go to all sorts of trouble to avoid working on what I’m supposed to be working on, especially if it involves a joke.

I added all this to my book’s source code. What book, you ask? Why, Making Java Groovy, available through the Manning Early Access Program (MEAP) at http://manning.com/kousen. I don’t know, however, if I’ll have room in the text to include all this.

log.marketing('Please forgive the mandatory advertising for my book')