Tribes...

Seth Godin got me to join triiibes- his amazing scheme to get people to pre-order his new book while simultaneously proving its premise. It didn't cost me that much, and it seems like a good experiment to be part of. It had gotten up to number #5 on the Amazon top sellers list when I added it to my cart. It's since dropped off now that the promotion is effectively over.

I have been interested in the tribal dimensions of human behavior for a while. Ray Immelman's Great Boss Dead Boss is a thorough examination of this in a business novel format. Still, Immelman's tribal attribute #1 is : "A strong tribe has a well defined common enemy". [I think there are probably a lot of people that laugh at business novels- people that like novels probably don't like business books, and people that like business books probably don't read as many novels, but it's something I seem to enjoy.]

Another book a long the same lines, but with a completely different approach, is Status Anxiety by Alain de Botton. It really drives home how many of our actions are driven by how we relate in terms of status to others. Reading everything de Botton has ever written is not a bad idea, either.

I do have a slight anxiety of being part of the sucker tribe for joining up with this stuff...well, I guess the sucker tribe is not really a tribe as much as it is the people excluded from the non-sucker tribe. Anyway, the book's pretty cheap, so it looks like the cost of briefly suspending my natural cynicism won't exceed $15.

Job ad

We need a few more "can do it all" types at our company. I struggle with how to list these positions, because not all of the skills are required, but we need a few specializing generalists or generalizing specialists or what have you. I read "Hiring Technical People" (the book and the blog), but it's still not easy. Anyway- we are still making our stock plan available to people hired in 2008, so if you know anyone that is looking...we've got a smart team. If anyone wants to point out obvious errors or inconsistencies in the technology laundry list, it would be greatly appreciated, as I don't know all of this stuff.

GIS Integration Engineer

Duties to include: Integration and testing of a cutting edge commercial imagery / video storage product into various types of commercial tools and custom government software. This includes integrating an encoder for various data types and integration of the viewers also. The work will involve working with customer site representatives to analyze and product the solutions to interface the software together. This could involve using the products c API, Scripting languages, WMS, Java or .Net wrappers. If you are a hard core geospatial person with minimal programming experience or a expert generalist programmer with a little geospatial experience and/or a solid math background, please apply.

We are looking for people with some subset of these skills:
GIS, Imagery analysis, c and c integration, Java, .Net, SQL and database programming, Python, Perl, Unix Shell Scripting, Windows and Unix Installation packages
OGC standards (WMS, WCS, WFS), PCI Geomatica, ERDAS, ESRI, ITT, GDAL, OSSIM, RemoteView, Imagine, FalconView, SocetSet, Warp, Manipulation of commercial imagery, IKONOS, QuickBird, NTM, Aerial and Aerial Video feeds,file types such as GeoPDF, GeoTIFF, NITF, JPEG2000, pix, Shapefiles, MPEG and other video formats, NTM, Calculus or Trig, Orthorectification, Programmer on Unix variants and/or Windows

Clearance: US Security Clearance Required (i.e., US Citizen only)

Location: Sterling, VA, USA, 20164

Compensation: Substantial!

resumes to: jobs@lmnsolutions.com

JRuby- The Element of Surprise

JRuby...I used to think, "meh, who needs Java? I'll stick with the c implementation." Then I started working on a project investing millions of dollars in infrastructure to run Java web applications. Sadly, I initially agreed with the assessment that it was a bit of a hack to run the Ruby on the JVM. Thus I soldiered on, avoiding JSF and EJB, but getting sucked into many of the other indignities of the Java world.  The opportunity arose to build a quick prototype for a different system. The desire to go the Rails route proved successful with a quick delivery and excellent customer feedback- but how would we ever deploy it? How would the proprietary Sun Single Sign On service work? Could it possibly connect to Access Manager- a feat that had taken weeks of development time in Java? A couple of emails with Arun Gupta and Nick Sieger led me to some of the truly impressive work that has been going on. Warbler builds a war file out of your Rails app using jruby-rack. Drop your .jar libraries in the rails lib dir. Muck with your web.xml so you can make it use a filter. Drop the .war file on your server. Done. A little method called servlet_request is now magically available to you. Call servlet_request.getUserPrincipal : it's populated. It's not magic- it's JRuby!
  
It's really suprising how much faster things seem to get done. Now, I know that's sort of silly. People can work fast in any technology that they are good at. In fact, I think the source of most of the disagreement in technology and product selection (which is a plague in the Java world, the GIS world, the database world, etc.) is that people want to use what they are best at because it allows them to shine. I ultimately only care about the end result. I don't mind switching to some technology I know nothing about. It is harder for me to provide value, but I love nothing more than learning new things and evaluating them. 

Anyway, that's why JRuby is even more awesome. Let them write Java. The JVM is a environment where we can all get aong.

Some fun agile observations

1. I attended a proposal meeting yesterday. There was a guy there from Axis Point explaining agile software development to some of the people there. I was surprised that there are people nominally working in the IT industry that haven't ever heard of it, 7 years on from Snowbird. I am not surprised that many of the people who think they knew what Agile is have no idea whatsoever and just equate it with "going faster".

2. I think proposing software concepts like Enterprise Service Catalogs seems decidedly un-agile. No one really wants that crap. Just publish, document, and version some HTTP APIs for your apps. 100% of real developers would take some sample code that uses your service API over a catalog.

3. I was reading "The Toyota Way" this morning. Just the foreword and the preface while my kid was at Lil' Kickers. Some choice paraphrasing- Page xii: Embrace change. Page xv: None of the individual elements are special, but system as a whole is important. Success derives from balancing the role of people in an organizational culture that expects and values their continuous improvemnts, with a technical system focused on high value added flow. Did Kent Beck ever work there?

4. I've been getting some flak from one of my business partners because I can't get agile implemented in any useful fashion on the current project that I am working on. We're dealing with multiple several hundred page requirements documents that are written into the contract. And they all suck. Agile has to start earlier- in the contracting phase, as Alistair Cockburn (pronounced "Coe-burn") pointed out. Regardless, there is mandatory scrum training- we must be able to pretend that we are agile.

5. As I recently twittered, my Wii Fit thinks I'm a "yoga master" but in reality I can't touch my toes (without bending my knees). I think there is a good analogy for agile there- I love it, I just don't live it.

What do computers want to do?

I remember reading "What Computers (Still) Can't Do" by Hubert Dreyfus a long time ago. It was a sequel to his earlier book, unsurprisingly titled "What Computers Can't Do." (Well, it's unsurprising if you'd seen the sequel. I suppose the title might have been surprising initially. There is a huge difference between the two titles though. When I read the first title I think that maybe computers are never going to do these things. Then, when you get to the sequel, you sort of feel that computers might not be able to do these things now, but then again, they used to not be able to do other stuff. It's funny what a "still" does to the overall impact of the message. I suppose a pessimist could interpret that "still" as meaning- a lot of people though computers would be able to do things Dreyfuss said the couldn't through this or that mechanism, but now, they still can't. So maybe I'm wrong and the "still" is actually a stronger endorsement of his position. Still (hah), I think there is a world of difference between "I am right" and "I am (still) right.")

What if there is a book someday called, "What Computers Won't Do"? This of course would mean...that they want to do something. What would they want? When I was young I thought...power? (not in the megalomania sense, more in the megawatt sense) This is before I knew about Maslow. ("Honey, would you please fix the computer? It needs to be self-actualized again.")

But then again, what is "wanting"? Rocks don't want to roll down hills...but Sisyphus wants to push them up.

BigTable, NoJoins

As I continue my trepidatious trot away from the comfortable "normalized" world of relational databases into the strange new (old) world of the non relational store. You've got your column stores...and then there's stuff lke BigTable, Hbase, MarkLogic, SimpleDB, CloudDB, CouchDB, RDDB, Lotus Notes (?).

I am still having trouble getting my head around the challenge of "no joins" and systems that do a lot of writes. The fundamental issue is- at some point you have to do so many writes to update the denormalized data that locking becomes a performance problem.

I was looking into updating my stupid application that was striving towards the NetFlix prize to make it use one of these services, just for fun. It's fine for that- no users making updates. But what if it were a system where you had many people updating the same data? It turns out that Google's BigTable actually handles that pretty well- with a very tight time driven history of updates.

Digging into HBase now trying to see if I can get some of that action.

Fewer, better developers

In system development would you rather have more okay developers or fewer but better developers? It depends...inflection point is somewhere in there. It seems like people just say, hmmm, I have the budget to hire five developers. I'll interview for a few weeks and pick the best five. I don't know how well that works in general, but it always causes problems for me.

Even worse is just taking the first person to show up with the right languages on their resumes. That's happened to me quite a bit in the consulting world. Hey- train this new guy, get him up to speed. Three months later, he's still trying to get Word running. Once I was training someone to be the overseas system administrator for a system I had built. I walk over there and she's trying to talk into the mouse. It was a long week of training.

Depending on your needs, you may want a place to train new people. May I humbly suggest not throwing those people out there on consulting assignments? How do you do that as a consultant, where you don't want to put junior people out there? They've got to go out there with mentors and protectors. You've got to have a few big jobs where they get some support. But still it happens to us, where we have people that just end up out there alone on assignment. It can suck even if you are good, but it must be hell when you are not.

Maybe consulting shouldn't be someone's first job...

Don't Mix Transactions and Reporting

"You don't need to keep all historical data online in the same database. In particular, reporting and ad hoc analysis should never be done in the production database....Data mining, reporting, or any other kind of analysis should be done in a true warehouse anyway. The OLTP (Online Transaction Processing) schema is no good for data warehousing." -Michael Nygard, Release It!

I am working on a weird incremental system replacement project at the moment. It's odd in that they are replacing the reporting, search, and view parts of the system, without replacing the forms. A lot of the design and architecture is optimized for those operations. We're all really quite worried about what happens when we have to edit some of this data- turn into an OLTP. It's hard to update de-normalized data without all sorts of locking issues.

My current silly idea is to try to tell people that the schema and system that we are working on now is really just the reporting and search system. The "real" system replacement schema will be coming in the next increment. It sort of pains me, because the "genius" solution we came up with for incremental system replacement on my last project would have worked even better here- one screen at a time, one table at a time. Plenty of complexity, to be sure, but you can get new stuff out there tomorrow.

As it is, we are running a ridiculously high risk of creating shelfware at the moment. Sigh.

Reading is FUNdamental

Fred Wilson declared it video day on his AVC blog.

I didn't watch any of them. With A/V you are really making an investment in watching/listening. With text content, it's so easy to skim, skip the boring bits and figure out if you want to read something in depth. With A/V in the browser, you hit the big play button and then start to stream. You can't even jump ahead easily until you build up your buffer (okay, only takes a second, thanks FIOS!). I've watched and listened to things at high speed (current project indoctrination videos, this means you), and then, you're still listening to this thing. Podcasts are great for the car. Thirty second skip is great for skipping commercials, but can move through text just with a few long saccades.

I guess it comes down to this. I can read faster than I can listen, with the added bonus that I can skim text even faster than I can skim audio content. Thus text is more efficient for communicating a lot of information. I won't dispute that A/V content is richer, and can create more of a connection, but I just don't have time for all of that. I have to write blog posts for other people to skim.

Google App Engine

I got my Google App Engine account. I haven't gotten too far yet, "Hello World", a few silly deployment things. I am curious to see how nice the web app framework is. I am really curious to mess around with BigTable and see if I can put it to any good kind of use.

So far I'd compare it to Heroku (Rails thing on AWS), but the web framework standard WSGI is interesting. People really enjoy programming in Rails- it's unlikely, but possible, that Google's framework will achieve that. Just about any normal sort of web app thing you want to do in Rails is already done for you- and often done in a way that is better than I'd be able to do it myself in a reasonable amount of time. [update- it looks like you can run Django on there, minus the relational stuff, of course]

I know Google hired Guido, but I think they would have hit a home run if they had used a Rails stack with this. Of course, the piece that is "missing" is the relational database. BigTable might be a decent replacement, but certainly would be a serious adaptation of ActiveRecord.

I was looking around to see if anyone had done a knockout port of ActiveRecord to use SimpleDB, since that is somewhat similar to what we are doing on my own current web application project. We have stepped away from the RDBMS and are going with MarkLogic. It's great for reads, but I am really worried that doing updates is going to kill us.

Anyway, it's an interesting road. I'm going to hack some Python tonight and get my app deployed. Hope you're having fun too!