Rolling Restarts, Migrations, and Deployments

I was looking into a question on stackoverflow (sweet site!) and it brought to mind the awesomeness and challenges of dealing with restarting a load balanced array of servers. The awesome part is you can do a deployment with any downtime by taking down a server, updating it, bringing it back up, and moving on to the next one. One issue with this is that you end up having multiple versions of the code live simultaneously. This can get a little weird if a user hits a new view page then the request ends up being loadbalanced to a server that isn't up to date yet.

The solution we came up with (before we heard about SeeSaw) was to take half of the mongrels off line from the load balancer. Shut them down. Update them. Start them up. Put those mongrels back online in the load balancer and take the other half off. Shut the second half down. Update the second half. Start them up. This greatly minimizes the time where you have two different versions of the application running simultaneously. I wrote a windows bat file to do this. (Deploying on Windows is not recommended, btw)

A truly awesome solution to this would be a load balancer that is somehow aware of the version level of the balanced set and just makes the switch for you. Until that is invented, Apache mod_proxy_balancer is easy enough to control remotely.

It is very important to note that having database migrations can make the whole approach a little dangerous. If you have only additive migrations, you can run those at any time before the deployment. If you are removing columns, you need to do it after the deployment. If you are renaming columns, it is better to split it into a create a new column and copy data into it migration to run before deployment and a separate script to remove the old column after deployment. In fact, it may be dangerous to use your regular migrations on a production database in general if you don't make a specific effort to organize them. All of this points to making more frequent deliveries so each update is lower risk and less complex, but that's a subject for another response.

What's a defect? What's a missing feature?

I was reading this about a nice idea some bloke had about developers fixing bugs in their spare time. Most of the serious developers I know are already working in their spare time- let's not make all defects their problem too.

Many software testers cause great frustration among software developers.  One of the biggest issues that arises on agile projects is that testers have a hard time distinguishing between defects and features that have not yet been implemented.

It's always a bit of a challenge to deal with bugs- even outside of agile.  What to some people is an obvious bug is a feature that was never requested to someone else.

For example, I was called into a meeting today where a bug that had been discovered during a user demo. The search was not "not finding telephone numbers in documents". The search term entered was 5551212. Some documents contained 555 1212. Some contained 555-1212. Some contained 1(703)555-1212.

They tried 555*1212, still not working. Search is broken. Developer suggests- try "555 1212", magic happens. * matches any character, not word boundaries...

It was an obvious problem...to the developers who understood that searching for "breakup" was not going to match documents containing "break up".  With text, it's obvious, but I can certainly see where people might not see the issue with phone numbers.

We'll add normalized versions of phone numbers into the search index and we'll normalize search terms that look like phone numbers or something like that, but...this is not an insignificant effort. (Even though there is some decent code out there to handle it.) There are trade-offs in performance that have to be considered.  Ask a tester though- they'll say found a bug; it's their job, and they want to be able to show how good they are at finding them.  If it's a bug, we can't even mark our existing search work as complete.  If it's a missing feature, we have to allocate it to the next release.  Fortunately, a tester didn't find this one, so we don't quite have that problem, but the users do want the feature of being able to match many different formats of phone numbers.

It seems like a simple semantic difference, and it seems like developers are being too sensitive, but it's actually a big deal.  Some "bugs" might actually end up costing a huge amount of money and not be worth fixing- particularly if they aren't really bugs.  I have seen teams show excessive deference to testers and spend a time equal to the time spent working on the basic features to handle some edge cases that would never really occur and were better handled by error messages than by trying to do something useful. Meanwhile, the project sponsor is seldom asked to decide whether a bug should be fixed- few are even estimated as to the cost.

I say- if you want to fix a bug, you have to pay up. If you are really smart, you do do the five whys and find the root cause, but if it happens to be something that it never occured to anyone to ask for, try not to ask the developers to take care of it on their own time.

Reading about Hadoop

Tom White's book on Hadoop is up on Rough Cuts. If you aren't familiar with Apache Hadoop, it's Doug Cutting's effort to go beyond Lucene and build an open source implementation of much of the other Google infrastructure.

There is a two day Hadoop Camp at the upcoming ApacheCon in New Orleans. Learning about Hadoop is a great way to become familiar with some of the innovations that Google has put forward in the last few years- and to see the technology behind Yahoo's big set of nodes. What follows is the beginning of Mr. White's book, I am looking forward to the chapter on HBase.

"Hadoop was created by Doug Cutting, the creator of Lucene, the widely-used text search library. Hadoop has its origins in Nutch, an open source web search engine, itself a part of the Lucene project.

Building a web search engine from scratch was an ambitious goal, for not only is the software required to crawl and index websites complex to write, it is also a challenge to run without a dedicated operations team — there are lots of moving parts. It's expensive too — Cutting and Cafarella estimated a system supporting a 1-billion-page index would cost around half a million dollars in hardware, with a $30,00 monthly running cost. Nevertheless, they believed it was a worthy goal, as it would open up and ultimately democratise search engine algorithms.

Nutch was started in 2002, and a working crawler and search system quickly emerged. However, they realized that their architecture wouldn't scale to the billions of pages on the web. Help was at hand with the publication of "The Google File System" (GFS) [ref] in 2003. This paper described the architecture of Google's distributed filesystem that was being used in production at Google. GFS, or something like it, would solve their storage needs for the very large files generated as a part of the web crawl and indexing process. In particular, GFS would free up time being spent on administrative tasks such as managing storage nodes. In 2004 they set about writing an open source implementation, the Nutch Distributed File System (NDFS) as it became to be known.

NDFS and the MapReduce implementation in Nutch were applicable beyond the realm of search, and in February 2006 they moved out of Nutch to form an independent subproject of Lucene called Hadoop. Yahoo! hired Doug Cutting and, with a dedicated team, provided the resources to turn Hadoop a system that ran at web scale. This was demonstrated in February 2008 when Yahoo! announced that their production search index was being generated by a 10,000 core Hadoop cluster.

Earlier, in January 2008, Hadoop was made a top-level project at Apache, confirming its success and dynamic community."

Another day, another blog

I'm also now infrequently blogging at Collision of Influences, my company blog, to add to my infrequent posting here on more arcane topics of interest to nearly 43 people!  Topics over there are going to be a little different, I just haven't figured out how yet.  I'll let it develop as is its wont.

The KML Handbook

I just noticed the "The KML Handbook: Geographic Visualization for the Web" by Josie Wernecke is out on Safari Rough Cuts. It looks like it has some interesting tips, not really a programming book per se, more of an expanded version of Google's documentation.  The subtitle is interesting to me- "for the Web".  A couple of years ago it would have just been for Google Earth, but KML is on its way to becoming the new shapefile.  Next I want JSON.

SpatialKey - Flash Mapping...

A colleague sent me a link to a beta Flash based mapping tool called SpatialKey from Universal Mind. It's pretty slick with timelines, heatmaps, animation, filters, and easy data loading from CSV. They have an enterprise angle as well with a Java server side application. It reminds me a little bit of the tools that could build something like Trulia Hindsight or the London Transit slider maps which were built by Stamen. Now if I could just find a way to create deep links into it...or is that called Flex?

Tribes...

Seth Godin got me to join triiibes- his amazing scheme to get people to pre-order his new book while simultaneously proving its premise. It didn't cost me that much, and it seems like a good experiment to be part of. It had gotten up to number #5 on the Amazon top sellers list when I added it to my cart. It's since dropped off now that the promotion is effectively over.

I have been interested in the tribal dimensions of human behavior for a while. Ray Immelman's Great Boss Dead Boss is a thorough examination of this in a business novel format. Still, Immelman's tribal attribute #1 is : "A strong tribe has a well defined common enemy". [I think there are probably a lot of people that laugh at business novels- people that like novels probably don't like business books, and people that like business books probably don't read as many novels, but it's something I seem to enjoy.]

Another book a long the same lines, but with a completely different approach, is Status Anxiety by Alain de Botton. It really drives home how many of our actions are driven by how we relate in terms of status to others. Reading everything de Botton has ever written is not a bad idea, either.

I do have a slight anxiety of being part of the sucker tribe for joining up with this stuff...well, I guess the sucker tribe is not really a tribe as much as it is the people excluded from the non-sucker tribe. Anyway, the book's pretty cheap, so it looks like the cost of briefly suspending my natural cynicism won't exceed $15.

Job ad

We need a few more "can do it all" types at our company. I struggle with how to list these positions, because not all of the skills are required, but we need a few specializing generalists or generalizing specialists or what have you. I read "Hiring Technical People" (the book and the blog), but it's still not easy. Anyway- we are still making our stock plan available to people hired in 2008, so if you know anyone that is looking...we've got a smart team. If anyone wants to point out obvious errors or inconsistencies in the technology laundry list, it would be greatly appreciated, as I don't know all of this stuff.

GIS Integration Engineer

Duties to include: Integration and testing of a cutting edge commercial imagery / video storage product into various types of commercial tools and custom government software. This includes integrating an encoder for various data types and integration of the viewers also. The work will involve working with customer site representatives to analyze and product the solutions to interface the software together. This could involve using the products c API, Scripting languages, WMS, Java or .Net wrappers. If you are a hard core geospatial person with minimal programming experience or a expert generalist programmer with a little geospatial experience and/or a solid math background, please apply.

We are looking for people with some subset of these skills:
GIS, Imagery analysis, c and c integration, Java, .Net, SQL and database programming, Python, Perl, Unix Shell Scripting, Windows and Unix Installation packages
OGC standards (WMS, WCS, WFS), PCI Geomatica, ERDAS, ESRI, ITT, GDAL, OSSIM, RemoteView, Imagine, FalconView, SocetSet, Warp, Manipulation of commercial imagery, IKONOS, QuickBird, NTM, Aerial and Aerial Video feeds,file types such as GeoPDF, GeoTIFF, NITF, JPEG2000, pix, Shapefiles, MPEG and other video formats, NTM, Calculus or Trig, Orthorectification, Programmer on Unix variants and/or Windows

Clearance: US Security Clearance Required (i.e., US Citizen only)

Location: Sterling, VA, USA, 20164

Compensation: Substantial!

resumes to: jobs@lmnsolutions.com

JRuby- The Element of Surprise

JRuby...I used to think, "meh, who needs Java? I'll stick with the c implementation." Then I started working on a project investing millions of dollars in infrastructure to run Java web applications. Sadly, I initially agreed with the assessment that it was a bit of a hack to run the Ruby on the JVM. Thus I soldiered on, avoiding JSF and EJB, but getting sucked into many of the other indignities of the Java world.  The opportunity arose to build a quick prototype for a different system. The desire to go the Rails route proved successful with a quick delivery and excellent customer feedback- but how would we ever deploy it? How would the proprietary Sun Single Sign On service work? Could it possibly connect to Access Manager- a feat that had taken weeks of development time in Java? A couple of emails with Arun Gupta and Nick Sieger led me to some of the truly impressive work that has been going on. Warbler builds a war file out of your Rails app using jruby-rack. Drop your .jar libraries in the rails lib dir. Muck with your web.xml so you can make it use a filter. Drop the .war file on your server. Done. A little method called servlet_request is now magically available to you. Call servlet_request.getUserPrincipal : it's populated. It's not magic- it's JRuby!
  
It's really suprising how much faster things seem to get done. Now, I know that's sort of silly. People can work fast in any technology that they are good at. In fact, I think the source of most of the disagreement in technology and product selection (which is a plague in the Java world, the GIS world, the database world, etc.) is that people want to use what they are best at because it allows them to shine. I ultimately only care about the end result. I don't mind switching to some technology I know nothing about. It is harder for me to provide value, but I love nothing more than learning new things and evaluating them. 

Anyway, that's why JRuby is even more awesome. Let them write Java. The JVM is a environment where we can all get aong.

Some fun agile observations

1. I attended a proposal meeting yesterday. There was a guy there from Axis Point explaining agile software development to some of the people there. I was surprised that there are people nominally working in the IT industry that haven't ever heard of it, 7 years on from Snowbird. I am not surprised that many of the people who think they knew what Agile is have no idea whatsoever and just equate it with "going faster".

2. I think proposing software concepts like Enterprise Service Catalogs seems decidedly un-agile. No one really wants that crap. Just publish, document, and version some HTTP APIs for your apps. 100% of real developers would take some sample code that uses your service API over a catalog.

3. I was reading "The Toyota Way" this morning. Just the foreword and the preface while my kid was at Lil' Kickers. Some choice paraphrasing- Page xii: Embrace change. Page xv: None of the individual elements are special, but system as a whole is important. Success derives from balancing the role of people in an organizational culture that expects and values their continuous improvemnts, with a technical system focused on high value added flow. Did Kent Beck ever work there?

4. I've been getting some flak from one of my business partners because I can't get agile implemented in any useful fashion on the current project that I am working on. We're dealing with multiple several hundred page requirements documents that are written into the contract. And they all suck. Agile has to start earlier- in the contracting phase, as Alistair Cockburn (pronounced "Coe-burn") pointed out. Regardless, there is mandatory scrum training- we must be able to pretend that we are agile.

5. As I recently twittered, my Wii Fit thinks I'm a "yoga master" but in reality I can't touch my toes (without bending my knees). I think there is a good analogy for agile there- I love it, I just don't live it.