Don't Mix Transactions and Reporting

"You don't need to keep all historical data online in the same database. In particular, reporting and ad hoc analysis should never be done in the production database....Data mining, reporting, or any other kind of analysis should be done in a true warehouse anyway. The OLTP (Online Transaction Processing) schema is no good for data warehousing." -Michael Nygard, Release It!

I am working on a weird incremental system replacement project at the moment. It's odd in that they are replacing the reporting, search, and view parts of the system, without replacing the forms. A lot of the design and architecture is optimized for those operations. We're all really quite worried about what happens when we have to edit some of this data- turn into an OLTP. It's hard to update de-normalized data without all sorts of locking issues.

My current silly idea is to try to tell people that the schema and system that we are working on now is really just the reporting and search system. The "real" system replacement schema will be coming in the next increment. It sort of pains me, because the "genius" solution we came up with for incremental system replacement on my last project would have worked even better here- one screen at a time, one table at a time. Plenty of complexity, to be sure, but you can get new stuff out there tomorrow.

As it is, we are running a ridiculously high risk of creating shelfware at the moment. Sigh.

Reading is FUNdamental

Fred Wilson declared it video day on his AVC blog.

I didn't watch any of them. With A/V you are really making an investment in watching/listening. With text content, it's so easy to skim, skip the boring bits and figure out if you want to read something in depth. With A/V in the browser, you hit the big play button and then start to stream. You can't even jump ahead easily until you build up your buffer (okay, only takes a second, thanks FIOS!). I've watched and listened to things at high speed (current project indoctrination videos, this means you), and then, you're still listening to this thing. Podcasts are great for the car. Thirty second skip is great for skipping commercials, but can move through text just with a few long saccades.

I guess it comes down to this. I can read faster than I can listen, with the added bonus that I can skim text even faster than I can skim audio content. Thus text is more efficient for communicating a lot of information. I won't dispute that A/V content is richer, and can create more of a connection, but I just don't have time for all of that. I have to write blog posts for other people to skim.

Google App Engine

I got my Google App Engine account. I haven't gotten too far yet, "Hello World", a few silly deployment things. I am curious to see how nice the web app framework is. I am really curious to mess around with BigTable and see if I can put it to any good kind of use.

So far I'd compare it to Heroku (Rails thing on AWS), but the web framework standard WSGI is interesting. People really enjoy programming in Rails- it's unlikely, but possible, that Google's framework will achieve that. Just about any normal sort of web app thing you want to do in Rails is already done for you- and often done in a way that is better than I'd be able to do it myself in a reasonable amount of time. [update- it looks like you can run Django on there, minus the relational stuff, of course]

I know Google hired Guido, but I think they would have hit a home run if they had used a Rails stack with this. Of course, the piece that is "missing" is the relational database. BigTable might be a decent replacement, but certainly would be a serious adaptation of ActiveRecord.

I was looking around to see if anyone had done a knockout port of ActiveRecord to use SimpleDB, since that is somewhat similar to what we are doing on my own current web application project. We have stepped away from the RDBMS and are going with MarkLogic. It's great for reads, but I am really worried that doing updates is going to kill us.

Anyway, it's an interesting road. I'm going to hack some Python tonight and get my app deployed. Hope you're having fun too!

Salad Container Topography

Release It!

While it might sound like a 37 Signals style plea to get software out there, Release It! from Michael Nygard is a totally different animal. It's a higher level "Code Complete" for large scale systems. It's about what to do so you can release it. A list of things you are doing that are increasing the risk of failure, and a list of things you can do to drop that risk.

Sample:
"Integration points are the number-one killer of systems. Every single one of those feeds presents a stability risk. Every socket, process, pipe, or remote procedure call can and will hang. Even database calls can hang, in ways obvious and subtle. Every feed into the system can hang it, crash it, or generate other impulses at the worst possible time. You’ll look at some of the specific ways these integration points can go bad and what you can do about them."

I wasn't really interested in the book until I heard the podcasts he did last year. A lot of sound advice in there...

Java for Rails developers

I am thinking of writing a short rejoinder to Rails for Java Developers. These are my notes so far.

Resist urge to yell: "Stop making everything so complicated!". Merely state it politely, as often as you can. Perhaps a recording would come in handy here.

Much of what Spring does with dependency injection is better handled by either reflection or simplifying code.

The DAO pattern makes no sense when compared with the majesty of ActiveRecord.

Oh no, they use XML for THAT!?!?!? How much XML? Oh no...

Why Maven Sucks (as compared to good old ant)

Maven is a build tool for Java. It suuuuuuuuuucks. I am not alone in thinking this.

Get out while you still can.
[Ship, 2007]

Any other community would be like "what the hell is this?"
[Rocher, 2008]

While you may think of the Maven 2 version you downloaded as building your code, it actually is just an engine that by itself, can't build squat. When you start Maven 2, it will try to download the latest and greatest of every core plugin, the bits that actually do the work. This means that even though two developers are using Maven 2.0.8, they could be using different plugin versions and therefore, the build is not consistent.
[Brown, 2008] (Maven patch submitter)

Maven suffers from a lack of flexibility and robustness
[Hanin, 2008]

It might cost you as much time to build your first Maven project as it does to LEARN Ant and build your first complex project.
[Carapetyan, 2008] (Maven evangelist)

Do they have a chapter called "Why do the repositories suck so much?" :)
[TSS Comment on free Maven book]

It introduces additional dependencies in your build process.
- Dependencies suck. The simple formula for operational availability is to multiply the availabilities of the interdependent systems. When you start with maven, there are a lot more things that have to be running- just to be able to compile some code. This is a measurable cost.

It introduces additional complexity in your build process.
- You are adding a lot of code to a project by adding Maven. You are adding dependent jars without necessarily knowing where they came from. If you build your own jar repository- it is a significant amount of work. If you don't, you are at the mercy of the Internet.

It is hard to debug when something is going wrong.

It took us a couple of days to get working. And then it immediately broke again. With completely different errors on three separate developer machines.

The side of evil will persistently keep trying to shoehorn maven into their projects and pretend they’re enjoying it. The side of good would rather gnaw off their own unmentionables than touch maven, and merrily keep getting things done faster, better, and lighter with ant (without paying Bruce Tate any denomination of any currency).
[Hani, 2004]

[Edit, 2009]
New graph of Maven adoption curve! 


Go check out the above link for more...

Business and IT, together again

“Running separate IT and business functions is not going to have a future, because there are companies out there that do it differently, do it faster and deliver business value. And as soon as folks start to learn that, they are going to start questioning their own IT set up.”
− Robert McGill, Standard Life, Edinburgh, Scotland

From Mary Poppendieck's slides here. The point being that this is IT+Business as an extension of the XP "onsite developer" concept.

The most rewarding period of my career was like this: working as part of a mission team, creating software for them, as it was needed to get our job done. Then came the CIO- in the name of killing off "duplicate" efforts they came at us. We had been smart and extracted a product from our work and had been offering it to other divisions in return for help in funding the organization. So was another group. Instead of taking a market based approach and letting the two applications find their niche- they had to merge us.

Oh, and at the same time as you merge the two systems, take these complex desktop applications and turn them into a web application that is 10 times slower and has almost none of the capabilities of the other systems. And use web standards that prevent adding usability features. And do everything with this XML services stuff that seems to be the future. Cue abject failure.

And then it happened again. And again. Soon the CIO is populated by tons of PowerPoint pushers with agendas and favorite technologies. Development talent is promoted into positions where they don't...make things. Budget is growing. No one I work with has seen a user for a long time. Does any of this software ever reach users? All we hear is they hate it and don't want it...So why are we doing it?

I've had the fortune to work on lots of good projects, but I have to say the whole concept of sucking IT out of the business and making it it's own shop that makes things for the business is not as effective as the opposite in 90% of cases I have seen. I do believe in core services of identity management, PKI, etc. But very very thin core services. I believe the CIO should have a budget to fund integrating core technologies and to sponsor the cost of implementing minimal standards across groups. But overall, it's a concept that doesn't make sense for most organizations.

According to Ms. Poppendieck, "You don’t get to be world class by chasing 'best practices'. You get there by inventing them." I don't care who invented it- let's just do it.

Yet another reason to raise my rates

Apparently, people drinking wines actually enjoy them more- based solely on the price.

The researchers discovered that people given two identical red wines to drink said they got much more pleasure from the one they were told had cost more. Brain scans confirmed that their pleasure centres were activated far more by the higher-priced wine.

I am not sure I want to activate customers' "pleasure centers", but if higher rates can lead to satisfaction- it's a win-win!! I can sorta see the logic too- the customer is thinking, "Wow, this guy seems slow, but he's the most expensive one we could afford, so I guess it must be how much consultants costs these days."

Then again, Seth Godin says we should be giving away stuff for free. Oh, maybe that's what this blog is! Even then, it's not worth the price.

The Backbone of SOA is Standardization?!?

Hurry out and buy this report on SOA from Input. I read the insightful reformatting of it on Washington Technology. It is frankly amazing how this publication can continue to peddle such inane drivel. Apparently, SOA is about standardization now:

SOA calls for agencies to standardize their technologies, which could create a significant shift in the market, said Deniece Peterson, senior analyst with Reston, Va.-based Input’s Executive Program.

“Standardization is the backbone of SOA,” she said. “For providers who usually supply proprietary solutions, this will force them to find other ways to be best-of-breed. If they don’t, they’ll just become very easily replaced.” On the other hand, providers who understand SOA and can market their offerings accordingly stand to gain.

Well, at least it's not about alignment anymore. I am glad this genius is spreading her BS straight to the executives. Do they notice when everyone says SOA is about something different?

At least I understand how it works now. Years ago, some developer decided that applications should have APIs (and the running instances of those APIs are services) so the data+application could be used in a loosely joined confederation of multiple systems. And lots of other software architect types noticed and gave it a name and said it was good, while slipping in their own favorite protocol or application in there. So, before anyone really agreed on what it meant, the word was out that SOA was good. Since basically all it means is good, any fool can say whatever jibberish they want is SOA as a shorthand for saying it's good in some technical way you don't understand.

Googling "backbone of soa" is instructive. I found these:
"Backbone of SOA is Service"
"Web services are the backbone of SOA"
"ESBs have become universally accepted at the backbone of SOA"
"ESB is the backbone of SOA"
"we suggest that you seriously consider an en-masse deployment of AMQP as the backbone of SOA"
"Messaging is the backbone of SOA"
"XML is the backbone of SOA"
"With processes serving as the backbone of SOA-based composite applications"
"distributed computing, which is the backbone of SOA"
"It's very appealing to mainframe customers, and the System z, which is the latest mainframe, is fully enabled to be the backbone of SOA"

With all of these backbones, maybe we should switch the software design metaphor from architect to anatomist. I am personally in favor of REST+JSON as the middle finger of SOA.

Frequent readers of my blog may recall that the appeal of the content-free term is simple: people in the business appreciate being served by IT, as opposed to the unfortunate situations that always seem to occur when IT departments are giving the job of enforcer of business rules via the replacement of efficient ad hoc systems with official electronic processes from which thou shall not deviate.

Are you being served?