Let's Push Things Forward

Books

2010-12-25T01:13:00.002-05:00

It's been a busy year for me, but more troublingly, a year in which I have been full of 75% complete ideas that haven't been worthy of their own posts. I have done a fair bit of reading though. Here are some of my favorite books that I've read this year (regardless of whether they came out this year). Maybe you can use some of the % back from your Amazon Visa card to buy them.

Webber, Paristaditis, Robinson REST in Practice: Hypermedia and Systems Architecture

I've been waiting a long time for this book (ever since How to GET a Cup of Coffee came out). Let's just say I have been going around for five years talking REST. I didn't actually know WTF I was talking about until I read this book and started to understand HATEOAS. It came out while I was struggling with a "RESTful" API that was singularly non user friendly. I still have a fundamental issue with REST where it seems like it is jamming too much into the six verbs. I was hoping this book would dissolve it, but it's actually written at too basic of a level.

Morville, Search Patterns: Design for Discovery
If you are working on application that includes search and discovery features, this book sets forth a great pattern language. There are tons of examples and ideas here.

Anderson, Kanban
I'll come right out and say it here publicly on the Internet: Scrum is not a good software development method. Scrum masters are not useful. I still like XP. Kanban and Lean are much more sensible methods to go with XP practices than Scrum. It's really way too much to get into here, but having spent the past 7 years reading on the Theory of Constraints, Toyota Way, Lean, etc. this is a method that makes so much sense to me I think I've already been doing it without having a name for it. Yes we Kanban!

Fried, Hannson, Rework
These guys just have no patience for all the non-value added aspects of work (shades of Lean and 'muda'). They also have a way to ensure that the work we do maximizes the opportunities for innovation, creativity, and even beauty, ultimately leading to get us towards our own happiness. Work should make you happy.

Matt Ridley, The Rational Optimist: How Prosperity Evolves
The author of the awesome The Red Queen comes back with treatise on why all the things that the whiners keep themselves busy whining about aren't really too bad. In all of the hand wringing about the minor issues of the day, we are vastly better off than people at any time in history. His thesis is that specialization and markets are the prime movers of progress, and the world might not falling apart at the seams as much as it sometimes seems. Actually made me hopeful about life on Earth, which is a rarity.

Sam Harris, The Moral Landscape: How Science Can Determine Human Values
This is a fascinating book. It's bit harsh on the approach for dealing with religious people, but it gets to the heart of where we can find areas of moral agreement, based on some simple starting points. I thought it would devolve into utilitarianism, but Harris takes it in another direction all together. He directly attacks the distinction between values and facts. It really made me question some of my "cultural relativism". I saw Harris speak earlier this year, and it was quite good. Plenty of his videos out on YouTube if you want the lite version. This will be offensive to many, but it's worth at least figuring it how to address these arguments if you disagree with them.

Ricky Gervais Presents: The World of Karl Pilkington
This is a bit funny. You probably have to heard the podcasts or show a couple of times to appreciate it, but it is really quite good with lots of laughs at the expense of the indefagitable Pilkington.

Lidwell, Holden, and Butler Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design
I love reading about design. I am in the middle of the new Frederick Brooks book on the design of design. I am still wondering how much reading about design it will take for me to eventually be a novice level design person.

Doug Lemov, Teach Like a Champion: 49 Techniques that Put Students on the Path to College
This is one thing missing from the teaching profession. In the manner of a software patterns book, this covers a set of techniques for running an effective classroom. Buy copies for your kids' teachers.

Jeffrey Pfeffer, Power: Why Some People Have It and Others Don't
Just be aware that I've read this book and have been trying to internalize it. At some point, you have to achieve a balance between getting what you want and being right. Even if you don't have the Will to Power, this will help you understand what those with the ambition are up to.

Ackoff, Beating the System: Using Creativity to Outsmart Bureaucracies
Some good ideas here.

That's just a quick sampler of what I can remember, which gets smaller every year. Enjoy!

Stack Underwhelm

2010-01-25T22:45:00.000-05:00

I was listening to the Stack Overflow Podcast #80 today while sitting in the traffic and it was infuriating...I think it was the combination of the two.

The first crime came in when Jeff began complaining about github. The essence of his complaint was the way in which forks interfere with the timeline view. Then Joel came in and claimed that they weren't forks, but rather branches, and that branching is not possible without a distributed version control system. I am no fan of github- in fact I find that projects hosted there are very difficult to understand, and the person-centric nature of it makes it a very poor software distribution platform. However, hearing those two criticize it is really frustrating, because they make so many errors in their criticisms that it actually supports the thing they are trying to oppose.

First of all, Joel's claim that you can't do branching in Centralized Version Control is ludicrous. What he may have meant to say is that you can't create local branches, you have to create them on server. Even this is only sorta true; you can run a command like svn switch and point your branch wherever you want. So, DVCS does not have a monopoly on branching.

Jeff's whining was more incoherent, but probably not as factually wrong. His main complaint was that for a project that he is working on, the often irrelevant forks created by other users show up on the network view of the code line that he is interested in. He is basically mistaking a weird UI feature for the entire nature of the system. There is an upside to the easy forking of github- poorly maintained projects are easily taken over. However, the huge downside is that it is very difficult to choose the "right fork" if you are looking to download a particular thing from there. You really lose the concept of a project homepage if you choose to exclusively use github for your project. All you get is a README doc that is probably copied by all of the other lame independent forkers that would be collaborating with you to put out quality releases if you were able to work towards the same codeline. Github enables this sort of anti-social behavior, making it very easy to not cooperate, and I have seen the quality of many projects go down as they have slid over to that hosting approach.

In short, I would recommend staying away from git and github for now, unless you really need to commit code to your local machine when you are in a shack in the woods with no internet connection. If you like the local history features and don't want to work in a branch, using IntelliJ or RubyMine is probably a good option. Git has poor Windows support and offers very little of value over subversion, which remains my preferred tool. If you are going to go DVCS and know anyone who uses Windows, I'd try mercurial. If you are going to host a serious open source project, it is probably better to have a page outside of github as your main page, instead of relying on their meager ability to identify the tested project code lines.

Tips for using Rails migrations

2009-10-26T23:01:00.001-04:00

The concept of a migration is that each change that is made to the structure of the database is captured in a version controlled script. Any time you want to create a table, add a field, change a field type, etc., this is done via a migration. Migrations are generated along with models, views, and controllers when a Rails scaffold is created to manage a resource. In addition, migrations are used to create a test database for each user which allows for rapid, automated quality checks on the application. If you have ever worked on a project where it is difficult to create a development "sandbox" database for each developer, where databases are copied via backup processes that leave weird permissions behind (I'm looking at you SQL Server), or where keeping the development database in sync with the application is something you even have to think about, you will easily see benefits from migrations.

Using Rails migrations can be a bit of change for teams. In some cases, developers are not very familiar with relational database concepts and object modeling principles and can create some overly complex and/or inefficient structures. Having the overall model reviewed by a database modeling expert and an object modeling expert can be quite helpful. I recommend Coad's "Java Modeling in Color with UML", Fowler's "Analysis Patterns" and David Hay's "Data Model Patterns" for getting yourself up to speed with this. However, it is also worth communicating with the rest of the team that may be touching the model.

Here are a few things we've done in the process of keeping migrations under control:

Don't edit an existing migration, create a new one.
     Editing an existing migration defeats the whole point. You might be able to get away with it if no one has run it. One obvious exception to this is if someone creates a migration that doesn't run and checked in. They must get the pig.

Update and run new migrations before you check in a migration.
While this might seem obvious, in many cases, it is easy to check in a new migration file to the repository without seeing if someone else has created one that does something that conflicts with yours, since the names will be different. If you don't run the migration, you risk doing something with your migration that causes issues. If you have already run your own migration locally, but not checked it in, and you get an update of a new migration, you need to roll back your own migration and change the numbering on it so that it runs after the migrations you just received from your version control system.

Check in the models that go with the migration when you check in the migration
   One key benefit of migrations is keeping the code changes in sync with the database changes. Add new tables and models in a single, logical commit.

Assign models to class owners
      The feature of XP that I have been least successful with is the concept of common code ownership. It is one of the easiest practices to apply in theory, but without the sense of collective responsibility and the compensating practices of pair programming and continuous integration, it can be problematic. In larger teams, it often makes more sense to have people or subteams assigned to manage a particular model so that changes can be coordinated. In theory, there can be enough communication to make it work. In reality, many teams are full of introverts that would rather rewrite huge swaths of code (and tests) than ask a question.

Communicate
     Changes to the object model can impact the whole application. These changes should be communicated to the rest of the team so that the reasoning behind them can be better understood and the data can be used correctly. There is a risk of someone criticizing or trying to change your proposal, but this is ultimately a lower risk than building something that others fail to understand.

Roll up all migrations into a big file after a major release
Once you've been going in your project for a while and get around the 100-150 migration point, it is probably worth it to roll the migrations prior to the current point into a single migration.

Generate SQL to run on the production database
   In many cases it is preferable to generate SQL to run on the production database, as opposed to running the migrate task directly against the database. This also allows for some more extensive code review on structural changes.

Don't forget to create constraints and indexes (indices?)
If you are using foreign keys in your database, it's often worth the cost to create a constraint on the column to prevent any data integrity issues. It's also a good practice to create an index on those columns to speed joins.

This is really just the tip of the iceberg, but it's a yummy water ice iceberg. Learning to use database migrations is a key skill, not just for Rails development, but for any agile development. A project which cannot create its database from scripts accessible to every developer is missing something important. By breaking those scripts down into small chunks and coordinating them with changes to the code, the situation where the application and database are out of sync simply does not exist.

Whose Success?

2009-10-07T00:00:00.000-04:00

There is sometimes a quandary between whose success we seek.

Acting as a consultant, you obviously want your own work to be considered successful, so that you will induce others to hire you. Getting there isn't so easy though. We often look downstream to the customer's customers. By helping meet the customer's customers' needs, we hope to make our immediate customers successful. The catch comes when something the immediate customer is doing is preventing them from satisfying their customer- and they don't want to change it. If we make the customer's customer successful at the cost of making the customer unsuccessful, we're probably not going to be successful.

Remember the secrets of consulting- you have to help people learn to solve their own problems, you can't always solve their problems for them. You can be satisfied with your accomplishments, regardless of whether they give you credit. The ideal form of influence is to help the customer see the problem more clearly, but let them decide. Finally, if you want to understand a dysfunctional system, follow the money!

Seed data in Rails 2.3.4

2009-09-28T15:59:00.001-04:00

Very nice addition to Rails as part of the security updates in 2.3.4- an easy way to load seed data.

Added db/seeds.rb as a default file for storing seed data for the database. Can 
be loaded with rake db:seed (or created alongside the db with db:setup). (This 
is also known as the "Stop Putting Gawd Damn Seed Data In Your Migrations" 
feature)

I like this on several accounts, not least of which is it invalidates my previous post on the subject.

In the meantime, there are several plugin based approaches to making this work, for example seed-fu.

Default route...

2009-09-10T11:20:00.004-04:00

One of the things that has been bothering me in Rails since everything switched to resources is how we lost the easy way any public method in the controller became an accessible URL without having to write a rule for it. For example, in a person controller, I could define a search method like this:

def search     
results = Person.seach(params[:q])
end

And it would be accessible, just like that. http://myapp/people/search?q=matt
I even wrote a simple router in Java to do the same thing, because it just makes it so easy to navigate the application code.

The poor design of map.resources messes this up. By adding the route pattern :controller/:id it starts with the default assumption that the search?q=matt part of my URL is an ID. Is it so hard to put an action name in there? In any case,

map.connect ':controller/:action'

goes above all of the map.resources calls in my app. You might want to put it in your application too.

Rails Fixtures and Seed Data

2009-07-11T22:58:00.001-04:00

One of the things that makes Ruby on Rails a great web framework is the quantity of tools available that are focused around the task of making web applications easy to build and encourage good software engineering practices.

Rails makes it really easy to test by building in a testing framework. One aspect of this is what are called "fixtures", data described in CSV or YML formats that is loaded into the database for each round of testing. Tests live at both the unit (model) and integration (controller) level. End to end tests that involve the view are traditionally run from an automated tool, such as Selenium or HPQC (ha).

UPDATE ALERT... Rails 2.3.4 has a nice, new way to do this.

Fixtures also make it easy to load seed data into the database. You can do this with migrations, but it gets messy, especially if you use ActiveRecord to do it, and then you delete a model class sometime in the future, it forces you to do an edit of an old migration, which means you might as well roll them all up. I prefer:

rake db:fixtures:load

Which runs:

namespace :fixtures do
    desc "Load fixtures into the current environment's database.  Load specific fixtures using FIXTURES=x,y"
    task :load => :environment do
      require 'active_record/fixtures'
      ActiveRecord::Base.establish_connection(RAILS_ENV.to_sym)
      (ENV['FIXTURES'] ? ENV['FIXTURES'].split(/,/) : Dir.glob(File.join(RAILS_ROOT, 'test', 'fixtures', '*.{yml,csv}'))).each do |fixture_file|
        Fixtures.create_fixtures('test/fixtures', File.basename(fixture_file, '.*'))
      end
    end
  end

This will load all of your test data into the development database (or production, if that's what your RAILS_ENV is set to). Jeffrey Allan Hardy documented this pretty well, and added a separate rake task that will do this from a separate directory from your test fixtures, which is a great idea. It goes like:

namespace :db do
  desc "Load seed fixtures (from db/fixtures) into the current environment's database." 
  task :seed => :environment do
    require 'active_record/fixtures'
    Dir.glob(RAILS_ROOT + '/db/fixtures/*.yml').each do |file|
      Fixtures.create_fixtures('db/fixtures', File.basename(file, '.*'))
    end
  end
end

It's pretty easy and keeps this stuff out of the migrations.

Thank You for Agreeing

2009-06-02T23:26:00.000-04:00

I've been reading "Thank You for Arguing" tonight, thanks to a tip from the excellent time sink known as Hacker News . The first page of the introduction has a pretty telling tip.

"Try this in a meeting: Answer someone who expresses doubt over idea with, 'Okay, let's tweak it.' Now focus the argument on revising your idea as if the group had already accepted it. This move is a form of concession- rhetorical jujitsu that uses your opponent's move to your advantage."

Heinrichs gets to some much deeper points as well. One key thing to focus on is getting the outcome you want from the discussion, and not on scoring points to "win" the argument.

I wanted to draw a tenuous connection here, between these rhetorical skills, to which Heinrichs gives a fabulous introduction, and the listening, or "receiving" skills that the Manager Tools guys have focused on . They put communication skills at the foundation of developing interpersonal skills, and they put listening skills at the foundation of communication. In their characteristic way, Horstmann and Auzenne give some concrete examples of specific, measurable behaviors that we can use to evaluate and improve our communications, by strengthening the feeling in the person we are communicating with that we understand them.

One of the items on their list of things to measure is using "Thank You" to begin a reply. (I said it was a tenuous connection.) Other things they recommend include "No interruptions", "Agreeing without qualification", "No buts or howevers", and my personal favorite, "smiling". By making simple gestures like this, the person that you are communicating with is shown that you respect them as a person. If we cut someone off, we are going to have a much harder time persuading them of anything, because they are likely to think that we don't understand their argument and they are also probably not going to like us very much.

What makes this even more challenging, is that in many situations, the person that you are doing a poor job of listening to is not going to know exactly what it is you are doing that they don't like, but they are just going to think you are disagreeable or contrary. You may end up getting negative feedback from managers that doesn't make sense- vague comments on how you are perceived. This is where I like to tie in things like the Manager Tools material to figure out what good feedback for those situations would look like. Respect for people is a key aspect of successful systems, such as the Toyota Way. However, it is also key in being persuasive. To really change someone's mind to what we think is an amazing thing. It means teaming up and getting on the same side.

"Thank You for Arguing" is filled with little gems that you can use to show respect to others, by offering them a choice, instead of negating what they say directly. Simple phrases like, "On the other hand, it could be that..." create an option for the listener, instead of putting them in a box. It offers them the opportunity to admit that they aren't 100% sure about their view. Moving the argument into the future tense also affords the chance to remove some of the contentiousness about any concerns, while defining the issue in a way that makes your point clear.

Rather than despise rhetoric for muddying the waters of rational debate, we should use it as a tool to achieve our goals, as long as we can keep it in a framework of respect for others- by combining it with the listening/receiving skills that are the foundation of communication.

Other good books on similar subjects are "Influence" and "Yes!" by Cialdini, "Difficult Conversations" and "Getting to Yes" out of the Harvard Negotiation Project that was led by Fisher the last I looked into it, "Never Eat Alone" by Ferazzi, and the classic "How to Win Friends and Influence People" by Dale Carnegie. My wife read the last on her Kindle recently, and the chapter on married couples is well worth it.

Passionate Programmer

2009-06-01T10:23:00.001-04:00

Chad Fowler's The Passionate Programmer is a rewrite of his earlier book on how to improve your career in software development, not a guide to finding love using Ruby scripts to mine craigslist. I read it anyway.

Today I'd like to take a quick look at #21, "The Daily Hit". The concept is to, each day, "have some kind of outstanding accomplishment to report to [your] manager, some idea [you] had thought of or implemented that would make [your] department better."

There are a couple of interesting aspects to this. First, is it forces you to break work down into small enough chunks that you can get something done in a day. The second is that improvement part. Thinking of, and implementing, small improvements every day is closely related to a Lean concept called Kaizen. The third is that reporting and tracking these things really helps demonstrate your value to the organization. While it seems annoying, and perhaps a bit like boasting, to have to market yourself to your boss or customer or whomever, it actually creates a more positive relationship, because it forces communication about positive subjects. Having at least one of these ready for a meeting like a Daily Scrum is a good idea- it can even set the tone of accomplishment.

Keeping it positive is really important. A ratio of nine positive to one negative feedback item is what the Manager Tools guys recommend. A key aspect of making Lean work is that it is not about blame. It is about admitting that things are never perfect, or even if they are, the context will change and make them imperfect. We face a lot of people challenges in the world of work. There are constant machinations of status and power, siphoning off of high profile work and dumping of boring work. We can't let this prevent us from doing good things. We need to make the assumption that there are more good people in the environment than bad, and building up a network of allies by providing them with support when needed, while also marketing the fact that we are providing that support, will make sure that there is someone out there who has our back. Help people get better.

So, get your daily hit. Communicate what your up to. Do it again tomorrow...

Iteration Zero

2009-03-22T23:06:00.000-04:00

Iteration Zero is the term used when a team that is attempting to do incremental development does not plan on developing any working software in their first increment. Notice the distinction here between iteration and increment- iterative development implies delivering the same thing over and over again to get it closer to being right (as opposed to "right the first time"), incremental development is delivering something in pieces (as opposed to "all at once"). When you get involved with the beast called "Agile", there are a lot more people in favor of incremental development than iterative development. So, in a lot of cases we are talking about Increment Zero, rather than Iteration Zero.

That whole "not delivering any working software" thing makes Iteration Zero seem inherently non-Agile. On the other hand, if you have team without computers or similar issues, it's hard to sign up for any business value. So what do you do in Iteration Zero? Peter Schuh seems to view it as a project inception phase, but then other people want to set out additional time before Iteration Zero to do all of that work. The Energized Work people seem to do tons of things in their Interation Zero. Earl Beede sees it as an admission of the failure of the original Agile principles. I would say it is more along the lines of giving in to no progress.

The real deal is that you have to set low goals for your first increment- plan on a zero velocity until you get something done. A co-worker of mine once made the remark that our organization at the time was "incapable of releasing a Hello World HTML page to the production environment." I think a customized "Hello, World" is a good increment one goal- put up an HTML user interface sketch. You might get some valuable feedback, such as, "we didn't want a web app". A lot of the prep work I see really doesn't take advantage of Real Options theory or You Aren't Gonna Need It. On the other hand, you do need to get started on something, you might just have to accept that not much will get done.

Iteration Zero seems to be a slippery slope. Particularly if you let it take months, at that point you are sliding down the slope really fast. I haven't really seen it used effectively, but that doesn't mean it can't be. I am always looking for data to change my mind.

To sum it up, my current recommendation is don't start with Iteration Zero, start with Velocity Zero.

Google Voice

2009-03-21T15:54:00.000-04:00

I just upgraded from Grand Central to Google Voice. I have been waiting for this transcription feature forever- I'd been using PhoneTag, but it wasn't quite the same. I don't want anyone to call my cell directly again! I have a feeling this is going to be an extremely popular service- it really puts the receiver back in charge of voicemail. Often I'd feel my time was being hijacked by people leaving inefficient messages. Text wins.

Data Integration, Propagate Deletes

2009-03-17T15:03:00.000-04:00

Enterprise Data Integration is challenging subject. Many of my customers have problems of the sort where they have pieces of information that are being collected by disparate systems. I try to push things towards simplicity, but there are many angles by which complexity sneaks in.

Think about something like Google Reader. Publish/subscribe works simply and well for this sort of content aggregation. However, the behavior of the client is not guaranteed. For example, what happens if I want to delete an entry? Or update an entry? Google Reader simply doesn't care if the publisher wants to delete an entry. It's already copied it into your data. Having an entry go missing from a data feed doesn't necessarily indicate an intention that a downstream deletion should occur.

Can a simple approach work for data integration? Perhaps, but there are a lot of tricky issues to work around.

Highlights

2009-02-19T16:34:00.000-05:00

Every time I find code that is already written for me, and written better than I would have done it. I remember why I love Ruby on Rails.

<%= highlight(@trees, "tree") %>
I think that I shall never see
A poem lovely as a tree.
A tree whose hungry mouth is prest
Against the sweet earth’s flowing breast;

The new edition, out March 15th, 2009.
Agile Web Development with Rails, Third Edition

Keeping an eye on what I am doing

2009-02-13T17:52:00.001-05:00

I like David Seah's idea of a "Task Order Up". My current implementation is pushing that together with the hipster PDA (binder clip and 3x5 cards), using sticky 3x5 notes to put my current objective/task on my monitor, so I can get back on track when I inevitably get off of it. It has a yak shaving aspect as well, where if I hit a blocker like "have to update version of solr" before I can accomplish the next task, I push the update solr task on top of the task (or stack of tasks) it is blocking. Then, when I finish the solr task, I pop it off, and there was thing it was blocking underneath.(LIFO approach) What makes it better than a computery way of doing this is that the index card is always visible, not buried behind other windows. I guess it would be good as a sidebar app, if I could give up that screen real estate.

Seems to be working so far.

Here Comes Somebody

2009-01-21T09:58:00.001-05:00

I finally read Clay Shirky's Here Comes Everybody: The Power of Organizing Without Organizations. It's a good book, not a great book. I think the problem with reading a book like that 11 months after it was released is that most of the ideas have already been absorbed into my consciousness. It's a much better book for someone without my vast wealth of experience using social tools. What Shirky does is establish a framework for understanding how the various tools fit together to produce a network of communication that changes the dynamics of who can be a journalist, or a software developer.

Shirky also gets deep into how this changes the dynamics of organizations. There is a very readable economics/management paper by Coase from 1937 (for which he won the Noble in 1991) called the Nature of the Firm. In there he examines the costs of contracting for services versus hiring, which is really why the firms exist in the first place. The technologies of the Internet seem to be changing those costs, and people that know how to take advantage of that can build better firms.

I think there is a unique intersection of trends happening right now, right in front of us that makes this a time of opportunity. It is a time when a many of the traditional solutions to problems are too slow, too expensive, and too static. It's a great time to have nothing, because you're not that far behind. It's a great time for open source. It's a great time to start something new. I am ready. Are you? Here we come.

I hate it when my friends let me down

2008-12-09T21:43:00.001-05:00

My good friends let me down today. They wasted my time with their silly problems. I really like Apache and Firefox and Ajax...but they had some obscure issues today that are causing me to break commitments. It took some seriously Google searching to uncover the root cause.

I was trying to make an Ajax request (using Prototype and link_to_remote) from Firefox to a JRuby on Rails application running in Tomcat over an https connection proxied by Apache Web Server. That's a lot of pieces that could go wrong. I was getting page not found in my poor little div. The link was being generated correctly and I could navigate to it.

Everything worked fine on my box. Of course, the problem was on the real server. Which is still in dev mode and hasn't been opened up so that I can connect to it from my desktop. Which means I have to open an X session to the server over ssh to launch a browser. Which doesn't have Firebug. Double check everything else. Have to find Firebug and get into the back end network...time passes...

Firebug said it was a 403.

It turns out Firefox doesn't pass the content-length header on POST requests. It turns out the sysadmin had typed the innocent looking phrase SecFilterEngine On into the Apache config. This causes Apache to reject requests without a content length header on POST requests. So secure I can't connect. I guess Firefox should send that header, but should Apache really demand it? Now I have a hell of a lot of working around this to do. To the casual observer / tester / customer - it's a bug, a defect, something wrong with my code. I should, and I wish I could, have a dev environment that even remotely resembled production. Amazing waste of a day. What's going to happen tomorrow?

Details here...

Rolling Restarts, Migrations, and Deployments

2008-09-23T23:53:00.001-04:00

I was looking into a question on stackoverflow (sweet site!) and it brought to mind the awesomeness and challenges of dealing with restarting a load balanced array of servers. The awesome part is you can do a deployment with any downtime by taking down a server, updating it, bringing it back up, and moving on to the next one. One issue with this is that you end up having multiple versions of the code live simultaneously. This can get a little weird if a user hits a new view page then the request ends up being loadbalanced to a server that isn't up to date yet.

The solution we came up with (before we heard about SeeSaw) was to take half of the mongrels off line from the load balancer. Shut them down. Update them. Start them up. Put those mongrels back online in the load balancer and take the other half off. Shut the second half down. Update the second half. Start them up. This greatly minimizes the time where you have two different versions of the application running simultaneously. I wrote a windows bat file to do this. (Deploying on Windows is not recommended, btw)

A truly awesome solution to this would be a load balancer that is somehow aware of the version level of the balanced set and just makes the switch for you. Until that is invented, Apache mod_proxy_balancer is easy enough to control remotely.

It is very important to note that having database migrations can make the whole approach a little dangerous. If you have only additive migrations, you can run those at any time before the deployment. If you are removing columns, you need to do it after the deployment. If you are renaming columns, it is better to split it into a create a new column and copy data into it migration to run before deployment and a separate script to remove the old column after deployment. In fact, it may be dangerous to use your regular migrations on a production database in general if you don't make a specific effort to organize them. All of this points to making more frequent deliveries so each update is lower risk and less complex, but that's a subject for another response.

What's a defect? What's a missing feature?

2008-09-23T00:24:00.000-04:00

I was reading this about a nice idea some bloke had about developers fixing bugs in their spare time. Most of the serious developers I know are already working in their spare time- let's not make all defects their problem too.

Many software testers cause great frustration among software developers. One of the biggest issues that arises on agile projects is that testers have a hard time distinguishing between defects and features that have not yet been implemented.

It's always a bit of a challenge to deal with bugs- even outside of agile. What to some people is an obvious bug is a feature that was never requested to someone else.

For example, I was called into a meeting today where a bug that had been discovered during a user demo. The search was not "not finding telephone numbers in documents". The search term entered was 5551212. Some documents contained 555 1212. Some contained 555-1212. Some contained 1(703)555-1212.

They tried 555*1212, still not working. Search is broken. Developer suggests- try "555 1212", magic happens. * matches any character, not word boundaries...

It was an obvious problem...to the developers who understood that searching for "breakup" was not going to match documents containing "break up". With text, it's obvious, but I can certainly see where people might not see the issue with phone numbers.

We'll add normalized versions of phone numbers into the search index and we'll normalize search terms that look like phone numbers or something like that, but...this is not an insignificant effort. (Even though there is some decent code out there to handle it.) There are trade-offs in performance that have to be considered. Ask a tester though- they'll say found a bug; it's their job, and they want to be able to show how good they are at finding them. If it's a bug, we can't even mark our existing search work as complete. If it's a missing feature, we have to allocate it to the next release. Fortunately, a tester didn't find this one, so we don't quite have that problem, but the users do want the feature of being able to match many different formats of phone numbers.

It seems like a simple semantic difference, and it seems like developers are being too sensitive, but it's actually a big deal. Some "bugs" might actually end up costing a huge amount of money and not be worth fixing- particularly if they aren't really bugs. I have seen teams show excessive deference to testers and spend a time equal to the time spent working on the basic features to handle some edge cases that would never really occur and were better handled by error messages than by trying to do something useful. Meanwhile, the project sponsor is seldom asked to decide whether a bug should be fixed- few are even estimated as to the cost.

I say- if you want to fix a bug, you have to pay up. If you are really smart, you do do the five whys and find the root cause, but if it happens to be something that it never occured to anyone to ask for, try not to ask the developers to take care of it on their own time.

Reading about Hadoop

2008-09-16T09:01:00.001-04:00

Tom White's book on Hadoop is up on Rough Cuts. If you aren't familiar with Apache Hadoop, it's Doug Cutting's effort to go beyond Lucene and build an open source implementation of much of the other Google infrastructure.

There is a two day Hadoop Camp at the upcoming ApacheCon in New Orleans. Learning about Hadoop is a great way to become familiar with some of the innovations that Google has put forward in the last few years- and to see the technology behind Yahoo's big set of nodes. What follows is the beginning of Mr. White's book, I am looking forward to the chapter on HBase.

"Hadoop was created by Doug Cutting, the creator of Lucene, the widely-used text search library. Hadoop has its origins in Nutch, an open source web search engine, itself a part of the Lucene project.

Building a web search engine from scratch was an ambitious goal, for not only is the software required to crawl and index websites complex to write, it is also a challenge to run without a dedicated operations team — there are lots of moving parts. It's expensive too — Cutting and Cafarella estimated a system supporting a 1-billion-page index would cost around half a million dollars in hardware, with a $30,00 monthly running cost. Nevertheless, they believed it was a worthy goal, as it would open up and ultimately democratise search engine algorithms.

Nutch was started in 2002, and a working crawler and search system quickly emerged. However, they realized that their architecture wouldn't scale to the billions of pages on the web. Help was at hand with the publication of "The Google File System" (GFS) [ref] in 2003. This paper described the architecture of Google's distributed filesystem that was being used in production at Google. GFS, or something like it, would solve their storage needs for the very large files generated as a part of the web crawl and indexing process. In particular, GFS would free up time being spent on administrative tasks such as managing storage nodes. In 2004 they set about writing an open source implementation, the Nutch Distributed File System (NDFS) as it became to be known.

NDFS and the MapReduce implementation in Nutch were applicable beyond the realm of search, and in February 2006 they moved out of Nutch to form an independent subproject of Lucene called Hadoop. Yahoo! hired Doug Cutting and, with a dedicated team, provided the resources to turn Hadoop a system that ran at web scale. This was demonstrated in February 2008 when Yahoo! announced that their production search index was being generated by a 10,000 core Hadoop cluster.

Earlier, in January 2008, Hadoop was made a top-level project at Apache, confirming its success and dynamic community."

Another day, another blog

2008-09-02T10:38:00.001-04:00

I'm also now infrequently blogging at Collision of Influences, my company blog, to add to my infrequent posting here on more arcane topics of interest to nearly 43 people! Topics over there are going to be a little different, I just haven't figured out how yet. I'll let it develop as is its wont.

The KML Handbook

2008-08-08T21:18:00.000-04:00

I just noticed the "The KML Handbook: Geographic Visualization for the Web" by Josie Wernecke is out on Safari Rough Cuts. It looks like it has some interesting tips, not really a programming book per se, more of an expanded version of Google's documentation. The subtitle is interesting to me- "for the Web". A couple of years ago it would have just been for Google Earth, but KML is on its way to becoming the new shapefile. Next I want JSON.

SpatialKey - Flash Mapping...

2008-08-06T22:52:00.003-04:00

A colleague sent me a link to a beta Flash based mapping tool called SpatialKey from Universal Mind. It's pretty slick with timelines, heatmaps, animation, filters, and easy data loading from CSV. They have an enterprise angle as well with a Java server side application. It reminds me a little bit of the tools that could build something like Trulia Hindsight or the London Transit slider maps which were built by Stamen. Now if I could just find a way to create deep links into it...or is that called Flex?

Tribes...

2008-07-31T23:33:00.003-04:00

Seth Godin got me to join triiibes- his amazing scheme to get people to pre-order his new book while simultaneously proving its premise. It didn't cost me that much, and it seems like a good experiment to be part of. It had gotten up to number #5 on the Amazon top sellers list when I added it to my cart. It's since dropped off now that the promotion is effectively over.

I have been interested in the tribal dimensions of human behavior for a while. Ray Immelman's Great Boss Dead Boss is a thorough examination of this in a business novel format. Still, Immelman's tribal attribute #1 is : "A strong tribe has a well defined common enemy". [I think there are probably a lot of people that laugh at business novels- people that like novels probably don't like business books, and people that like business books probably don't read as many novels, but it's something I seem to enjoy.]

Another book a long the same lines, but with a completely different approach, is Status Anxiety by Alain de Botton. It really drives home how many of our actions are driven by how we relate in terms of status to others. Reading everything de Botton has ever written is not a bad idea, either.

I do have a slight anxiety of being part of the sucker tribe for joining up with this stuff...well, I guess the sucker tribe is not really a tribe as much as it is the people excluded from the non-sucker tribe. Anyway, the book's pretty cheap, so it looks like the cost of briefly suspending my natural cynicism won't exceed $15.

Job ad

2008-07-17T23:51:00.003-04:00

We need a few more "can do it all" types at our company. I struggle with how to list these positions, because not all of the skills are required, but we need a few specializing generalists or generalizing specialists or what have you. I read "Hiring Technical People" (the book and the blog), but it's still not easy. Anyway- we are still making our stock plan available to people hired in 2008, so if you know anyone that is looking...we've got a smart team. If anyone wants to point out obvious errors or inconsistencies in the technology laundry list, it would be greatly appreciated, as I don't know all of this stuff.

GIS Integration Engineer

Duties to include: Integration and testing of a cutting edge commercial imagery / video storage product into various types of commercial tools and custom government software. This includes integrating an encoder for various data types and integration of the viewers also. The work will involve working with customer site representatives to analyze and product the solutions to interface the software together. This could involve using the products c API, Scripting languages, WMS, Java or .Net wrappers. If you are a hard core geospatial person with minimal programming experience or a expert generalist programmer with a little geospatial experience and/or a solid math background, please apply.

We are looking for people with some subset of these skills:
GIS, Imagery analysis, c and c integration, Java, .Net, SQL and database programming, Python, Perl, Unix Shell Scripting, Windows and Unix Installation packages
OGC standards (WMS, WCS, WFS), PCI Geomatica, ERDAS, ESRI, ITT, GDAL, OSSIM, RemoteView, Imagine, FalconView, SocetSet, Warp, Manipulation of commercial imagery, IKONOS, QuickBird, NTM, Aerial and Aerial Video feeds,file types such as GeoPDF, GeoTIFF, NITF, JPEG2000, pix, Shapefiles, MPEG and other video formats, NTM, Calculus or Trig, Orthorectification, Programmer on Unix variants and/or Windows

Clearance: US Security Clearance Required (i.e., US Citizen only)

Location: Sterling, VA, USA, 20164

Compensation: Substantial!

resumes to: jobs@lmnsolutions.com

JRuby- The Element of Surprise

2008-07-11T00:15:00.006-04:00

JRuby...I used to think, "meh, who needs Java? I'll stick with the c implementation." Then I started working on a project investing millions of dollars in infrastructure to run Java web applications. Sadly, I initially agreed with the assessment that it was a bit of a hack to run the Ruby on the JVM. Thus I soldiered on, avoiding JSF and EJB, but getting sucked into many of the other indignities of the Java world. The opportunity arose to build a quick prototype for a different system. The desire to go the Rails route proved successful with a quick delivery and excellent customer feedback- but how would we ever deploy it? How would the proprietary Sun Single Sign On service work? Could it possibly connect to Access Manager- a feat that had taken weeks of development time in Java? A couple of emails with Arun Gupta and Nick Sieger led me to some of the truly impressive work that has been going on. Warbler builds a war file out of your Rails app using jruby-rack. Drop your .jar libraries in the rails lib dir. Muck with your web.xml so you can make it use a filter. Drop the .war file on your server. Done. A little method called servlet_request is now magically available to you. Call servlet_request.getUserPrincipal : it's populated. It's not magic- it's JRuby!

It's really suprising how much faster things seem to get done. Now, I know that's sort of silly. People can work fast in any technology that they are good at. In fact, I think the source of most of the disagreement in technology and product selection (which is a plague in the Java world, the GIS world, the database world, etc.) is that people want to use what they are best at because it allows them to shine. I ultimately only care about the end result. I don't mind switching to some technology I know nothing about. It is harder for me to provide value, but I love nothing more than learning new things and evaluating them.

Anyway, that's why JRuby is even more awesome. Let them write Java. The JVM is a environment where we can all get aong.

Some fun agile observations

2008-06-21T12:42:00.003-04:00

1. I attended a proposal meeting yesterday. There was a guy there from Axis Point explaining agile software development to some of the people there. I was surprised that there are people nominally working in the IT industry that haven't ever heard of it, 7 years on from Snowbird. I am not surprised that many of the people who think they knew what Agile is have no idea whatsoever and just equate it with "going faster".

2. I think proposing software concepts like Enterprise Service Catalogs seems decidedly un-agile. No one really wants that crap. Just publish, document, and version some HTTP APIs for your apps. 100% of real developers would take some sample code that uses your service API over a catalog.

3. I was reading "The Toyota Way" this morning. Just the foreword and the preface while my kid was at Lil' Kickers. Some choice paraphrasing- Page xii: Embrace change. Page xv: None of the individual elements are special, but system as a whole is important. Success derives from balancing the role of people in an organizational culture that expects and values their continuous improvemnts, with a technical system focused on high value added flow. Did Kent Beck ever work there?

4. I've been getting some flak from one of my business partners because I can't get agile implemented in any useful fashion on the current project that I am working on. We're dealing with multiple several hundred page requirements documents that are written into the contract. And they all suck. Agile has to start earlier- in the contracting phase, as Alistair Cockburn (pronounced "Coe-burn") pointed out. Regardless, there is mandatory scrum training- we must be able to pretend that we are agile.

5. As I recently twittered, my Wii Fit thinks I'm a "yoga master" but in reality I can't touch my toes (without bending my knees). I think there is a good analogy for agile there- I love it, I just don't live it.

What do computers want to do?

2008-06-01T23:13:00.004-04:00

I remember reading "What Computers (Still) Can't Do" by Hubert Dreyfus a long time ago. It was a sequel to his earlier book, unsurprisingly titled "What Computers Can't Do." (Well, it's unsurprising if you'd seen the sequel. I suppose the title might have been surprising initially. There is a huge difference between the two titles though. When I read the first title I think that maybe computers are never going to do these things. Then, when you get to the sequel, you sort of feel that computers might not be able to do these things now, but then again, they used to not be able to do other stuff. It's funny what a "still" does to the overall impact of the message. I suppose a pessimist could interpret that "still" as meaning- a lot of people though computers would be able to do things Dreyfuss said the couldn't through this or that mechanism, but now, they still can't. So maybe I'm wrong and the "still" is actually a stronger endorsement of his position. Still (hah), I think there is a world of difference between "I am right" and "I am (still) right.")

What if there is a book someday called, "What Computers Won't Do"? This of course would mean...that they want to do something. What would they want? When I was young I thought...power? (not in the megalomania sense, more in the megawatt sense) This is before I knew about Maslow. ("Honey, would you please fix the computer? It needs to be self-actualized again.")

But then again, what is "wanting"? Rocks don't want to roll down hills...but Sisyphus wants to push them up.

BigTable, NoJoins

2008-05-12T09:58:00.004-04:00

As I continue my trepidatious trot away from the comfortable "normalized" world of relational databases into the strange new (old) world of the non relational store. You've got your column stores...and then there's stuff lke BigTable, Hbase, MarkLogic, SimpleDB, CloudDB, CouchDB, RDDB, Lotus Notes (?).

I am still having trouble getting my head around the challenge of "no joins" and systems that do a lot of writes. The fundamental issue is- at some point you have to do so many writes to update the denormalized data that locking becomes a performance problem.

I was looking into updating my stupid application that was striving towards the NetFlix prize to make it use one of these services, just for fun. It's fine for that- no users making updates. But what if it were a system where you had many people updating the same data? It turns out that Google's BigTable actually handles that pretty well- with a very tight time driven history of updates.

Digging into HBase now trying to see if I can get some of that action.

Fewer, better developers

2008-04-28T10:58:00.002-04:00

In system development would you rather have more okay developers or fewer but better developers? It depends...inflection point is somewhere in there. It seems like people just say, hmmm, I have the budget to hire five developers. I'll interview for a few weeks and pick the best five. I don't know how well that works in general, but it always causes problems for me.

Even worse is just taking the first person to show up with the right languages on their resumes. That's happened to me quite a bit in the consulting world. Hey- train this new guy, get him up to speed. Three months later, he's still trying to get Word running. Once I was training someone to be the overseas system administrator for a system I had built. I walk over there and she's trying to talk into the mouse. It was a long week of training.

Depending on your needs, you may want a place to train new people. May I humbly suggest not throwing those people out there on consulting assignments? How do you do that as a consultant, where you don't want to put junior people out there? They've got to go out there with mentors and protectors. You've got to have a few big jobs where they get some support. But still it happens to us, where we have people that just end up out there alone on assignment. It can suck even if you are good, but it must be hell when you are not.

Maybe consulting shouldn't be someone's first job...

Don't Mix Transactions and Reporting

2008-04-21T10:19:00.003-04:00

"You don't need to keep all historical data online in the same database. In particular, reporting and ad hoc analysis should never be done in the production database....Data mining, reporting, or any other kind of analysis should be done in a true warehouse anyway. The OLTP (Online Transaction Processing) schema is no good for data warehousing." -Michael Nygard, Release It!

I am working on a weird incremental system replacement project at the moment. It's odd in that they are replacing the reporting, search, and view parts of the system, without replacing the forms. A lot of the design and architecture is optimized for those operations. We're all really quite worried about what happens when we have to edit some of this data- turn into an OLTP. It's hard to update de-normalized data without all sorts of locking issues.

My current silly idea is to try to tell people that the schema and system that we are working on now is really just the reporting and search system. The "real" system replacement schema will be coming in the next increment. It sort of pains me, because the "genius" solution we came up with for incremental system replacement on my last project would have worked even better here- one screen at a time, one table at a time. Plenty of complexity, to be sure, but you can get new stuff out there tomorrow.

As it is, we are running a ridiculously high risk of creating shelfware at the moment. Sigh.

Reading is FUNdamental

2008-04-13T23:48:00.004-04:00

Fred Wilson declared it video day on his AVC blog.

I didn't watch any of them. With A/V you are really making an investment in watching/listening. With text content, it's so easy to skim, skip the boring bits and figure out if you want to read something in depth. With A/V in the browser, you hit the big play button and then start to stream. You can't even jump ahead easily until you build up your buffer (okay, only takes a second, thanks FIOS!). I've watched and listened to things at high speed (current project indoctrination videos, this means you), and then, you're still listening to this thing. Podcasts are great for the car. Thirty second skip is great for skipping commercials, but can move through text just with a few long saccades.

I guess it comes down to this. I can read faster than I can listen, with the added bonus that I can skim text even faster than I can skim audio content. Thus text is more efficient for communicating a lot of information. I won't dispute that A/V content is richer, and can create more of a connection, but I just don't have time for all of that. I have to write blog posts for other people to skim.

Google App Engine

2008-04-08T22:52:00.004-04:00

I got my Google App Engine account. I haven't gotten too far yet, "Hello World", a few silly deployment things. I am curious to see how nice the web app framework is. I am really curious to mess around with BigTable and see if I can put it to any good kind of use.

So far I'd compare it to Heroku (Rails thing on AWS), but the web framework standard WSGI is interesting. People really enjoy programming in Rails- it's unlikely, but possible, that Google's framework will achieve that. Just about any normal sort of web app thing you want to do in Rails is already done for you- and often done in a way that is better than I'd be able to do it myself in a reasonable amount of time. [update- it looks like you can run Django on there, minus the relational stuff, of course]

I know Google hired Guido, but I think they would have hit a home run if they had used a Rails stack with this. Of course, the piece that is "missing" is the relational database. BigTable might be a decent replacement, but certainly would be a serious adaptation of ActiveRecord.

I was looking around to see if anyone had done a knockout port of ActiveRecord to use SimpleDB, since that is somewhat similar to what we are doing on my own current web application project. We have stepped away from the RDBMS and are going with MarkLogic. It's great for reads, but I am really worried that doing updates is going to kill us.

Anyway, it's an interesting road. I'm going to hack some Python tonight and get my app deployed. Hope you're having fun too!

Salad Container Topography

2008-04-08T10:16:00.000-04:00

Release It!

2008-03-17T23:15:00.003-04:00

While it might sound like a 37 Signals style plea to get software out there, Release It! from Michael Nygard is a totally different animal. It's a higher level "Code Complete" for large scale systems. It's about what to do so you can release it. A list of things you are doing that are increasing the risk of failure, and a list of things you can do to drop that risk.

Sample:
"Integration points are the number-one killer of systems. Every single one of those feeds presents a stability risk. Every socket, process, pipe, or remote procedure call can and will hang. Even database calls can hang, in ways obvious and subtle. Every feed into the system can hang it, crash it, or generate other impulses at the worst possible time. You’ll look at some of the specific ways these integration points can go bad and what you can do about them."

I wasn't really interested in the book until I heard the podcasts he did last year. A lot of sound advice in there...

Java for Rails developers

2008-03-10T13:39:00.002-04:00

I am thinking of writing a short rejoinder to Rails for Java Developers. These are my notes so far.

Resist urge to yell: "Stop making everything so complicated!". Merely state it politely, as often as you can. Perhaps a recording would come in handy here.

Much of what Spring does with dependency injection is better handled by either reflection or simplifying code.

The DAO pattern makes no sense when compared with the majesty of ActiveRecord.

Oh no, they use XML for THAT!?!?!? How much XML? Oh no...

Why Maven Sucks (as compared to good old ant)

2008-02-20T23:12:00.004-05:00

Maven is a build tool for Java. It suuuuuuuuuucks. I am not alone in thinking this.

Get out while you still can.
[Ship, 2007]

Any other community would be like "what the hell is this?"
[Rocher, 2008]

While you may think of the Maven 2 version you downloaded as building your code, it actually is just an engine that by itself, can't build squat. When you start Maven 2, it will try to download the latest and greatest of every core plugin, the bits that actually do the work. This means that even though two developers are using Maven 2.0.8, they could be using different plugin versions and therefore, the build is not consistent.
[Brown, 2008] (Maven patch submitter)

Maven suffers from a lack of flexibility and robustness
[Hanin, 2008]

It might cost you as much time to build your first Maven project as it does to LEARN Ant and build your first complex project.
[Carapetyan, 2008] (Maven evangelist)

Do they have a chapter called "Why do the repositories suck so much?" :)
[TSS Comment on free Maven book]

It introduces additional dependencies in your build process.
- Dependencies suck. The simple formula for operational availability is to multiply the availabilities of the interdependent systems. When you start with maven, there are a lot more things that have to be running- just to be able to compile some code. This is a measurable cost.

It introduces additional complexity in your build process.
- You are adding a lot of code to a project by adding Maven. You are adding dependent jars without necessarily knowing where they came from. If you build your own jar repository- it is a significant amount of work. If you don't, you are at the mercy of the Internet.

It is hard to debug when something is going wrong.

It took us a couple of days to get working. And then it immediately broke again. With completely different errors on three separate developer machines.

The side of evil will persistently keep trying to shoehorn maven into their projects and pretend they’re enjoying it. The side of good would rather gnaw off their own unmentionables than touch maven, and merrily keep getting things done faster, better, and lighter with ant (without paying Bruce Tate any denomination of any currency).
[Hani, 2004]

[Edit, 2009]
New graph of Maven adoption curve!

Go check out the above link for more...

Business and IT, together again

2008-02-07T23:50:00.000-05:00

“Running separate IT and business functions is not going to have a future, because there are companies out there that do it differently, do it faster and deliver business value. And as soon as folks start to learn that, they are going to start questioning their own IT set up.”
− Robert McGill, Standard Life, Edinburgh, Scotland

From Mary Poppendieck's slides here. The point being that this is IT+Business as an extension of the XP "onsite developer" concept.

The most rewarding period of my career was like this: working as part of a mission team, creating software for them, as it was needed to get our job done. Then came the CIO- in the name of killing off "duplicate" efforts they came at us. We had been smart and extracted a product from our work and had been offering it to other divisions in return for help in funding the organization. So was another group. Instead of taking a market based approach and letting the two applications find their niche- they had to merge us.

Oh, and at the same time as you merge the two systems, take these complex desktop applications and turn them into a web application that is 10 times slower and has almost none of the capabilities of the other systems. And use web standards that prevent adding usability features. And do everything with this XML services stuff that seems to be the future. Cue abject failure.

And then it happened again. And again. Soon the CIO is populated by tons of PowerPoint pushers with agendas and favorite technologies. Development talent is promoted into positions where they don't...make things. Budget is growing. No one I work with has seen a user for a long time. Does any of this software ever reach users? All we hear is they hate it and don't want it...So why are we doing it?

I've had the fortune to work on lots of good projects, but I have to say the whole concept of sucking IT out of the business and making it it's own shop that makes things for the business is not as effective as the opposite in 90% of cases I have seen. I do believe in core services of identity management, PKI, etc. But very very thin core services. I believe the CIO should have a budget to fund integrating core technologies and to sponsor the cost of implementing minimal standards across groups. But overall, it's a concept that doesn't make sense for most organizations.

According to Ms. Poppendieck, "You don’t get to be world class by chasing 'best practices'. You get there by inventing them." I don't care who invented it- let's just do it.

Yet another reason to raise my rates

2008-01-14T22:36:00.000-05:00

Apparently, people drinking wines actually enjoy them more- based solely on the price.

The researchers discovered that people given two identical red wines to drink said they got much more pleasure from the one they were told had cost more. Brain scans confirmed that their pleasure centres were activated far more by the higher-priced wine.

I am not sure I want to activate customers' "pleasure centers", but if higher rates can lead to satisfaction- it's a win-win!! I can sorta see the logic too- the customer is thinking, "Wow, this guy seems slow, but he's the most expensive one we could afford, so I guess it must be how much consultants costs these days."

Then again, Seth Godin says we should be giving away stuff for free. Oh, maybe that's what this blog is! Even then, it's not worth the price.

The Backbone of SOA is Standardization?!?

2008-01-10T16:36:00.000-05:00

Hurry out and buy this report on SOA from Input. I read the insightful reformatting of it on Washington Technology. It is frankly amazing how this publication can continue to peddle such inane drivel. Apparently, SOA is about standardization now:

SOA calls for agencies to standardize their technologies, which could create a significant shift in the market, said Deniece Peterson, senior analyst with Reston, Va.-based Input’s Executive Program.

“Standardization is the backbone of SOA,” she said. “For providers who usually supply proprietary solutions, this will force them to find other ways to be best-of-breed. If they don’t, they’ll just become very easily replaced.” On the other hand, providers who understand SOA and can market their offerings accordingly stand to gain.

Well, at least it's not about alignment anymore. I am glad this genius is spreading her BS straight to the executives. Do they notice when everyone says SOA is about something different?

At least I understand how it works now. Years ago, some developer decided that applications should have APIs (and the running instances of those APIs are services) so the data+application could be used in a loosely joined confederation of multiple systems. And lots of other software architect types noticed and gave it a name and said it was good, while slipping in their own favorite protocol or application in there. So, before anyone really agreed on what it meant, the word was out that SOA was good. Since basically all it means is good, any fool can say whatever jibberish they want is SOA as a shorthand for saying it's good in some technical way you don't understand.

Googling "backbone of soa" is instructive. I found these:
"Backbone of SOA is Service"
"Web services are the backbone of SOA"
"ESBs have become universally accepted at the backbone of SOA"
"ESB is the backbone of SOA"
"we suggest that you seriously consider an en-masse deployment of AMQP as the backbone of SOA"
"Messaging is the backbone of SOA"
"XML is the backbone of SOA"
"With processes serving as the backbone of SOA-based composite applications"
"distributed computing, which is the backbone of SOA"
"It's very appealing to mainframe customers, and the System z, which is the latest mainframe, is fully enabled to be the backbone of SOA"

With all of these backbones, maybe we should switch the software design metaphor from architect to anatomist. I am personally in favor of REST+JSON as the middle finger of SOA.

Frequent readers of my blog may recall that the appeal of the content-free term is simple: people in the business appreciate being served by IT, as opposed to the unfortunate situations that always seem to occur when IT departments are giving the job of enforcer of business rules via the replacement of efficient ad hoc systems with official electronic processes from which thou shall not deviate.

Are you being served?

Why so slow?

2007-12-25T12:02:00.000-05:00

Google has thrived by proving that computer users value speed above nearly all else. So why does each Microsoft Windows device I use take longer and longer to boot up? via NY Times

Booting from Flash RAM is the future- how fast is Windows? I will not buy another computer without it. Everything in memory is the default optimization to take now...cache everything.

Ron Jeffries Can't Get Rails Running

2007-12-25T01:20:00.000-05:00

This is kind of a strange story, but why the hell can't he get it running? This is when to call in VMWare and have a nice set of clean machines available. Apparently he was running SlimServer, which sounds slightly immoral so I am sure he deserved the non-functioning environment. Anyway, nice thing about XP is early cancellation, but, he could have just asked for help...

That said- development environment set up is not that easy. I had one J2EE project where we had a two day process. For my last Rails project, it was a about two pages on the wiki, most of it related to SQL Server driver (don't ask) and Ferret. It did remind me to not depend on the availability of external resources for the install procedures though- what's out there on the Rails Gem servers can change.

What's killing everyone on one of my internal corporate projects is svn+ssh. That said, our most recent hire nailed it pretty well, even though it was his first time using a Mac and NetBeans, and I am using RadRails on Windows.

Anyway- it's worth writing up the procedures for the development environment set up, and then having the new person update those when they come on board as their first task. Of course, if no one has started for a while, chances are the instructions just won't work.

Digital Notes in general (and in ArcGIS)

2007-12-05T21:42:00.000-05:00

I was reading Tim Ferris' post on how "awesome" he is at taking notes. I have tried his approach and it didn't really work for me having all my notes in a little moleskine notebook, even if there is a little index page. I much prefer the approach of having a tear off pad of heavy Levenger paper, then I can put the note page in the appropriate folder and file it.

I was experimenting a little today with the middle ground- a digital/analog pen. You use a special notebook and special pen to take notes, then you connect the pen to a USB dock and the notes get transferred to OneNote 2007- and OCR-ed. It's a really cool idea- and their ArcGIS product is coming out soon. It's a cool idea to go out into the field with a paper map and then have your annotations show up on the map without scanning it back in, but as vectors recorded by the pen.

Anyway, I don't really want the big pen, but maybe the good idea to steal from all of this is to scan and OCR all of my note pages. Then I can full text search them...brilliant, eh?

Amish Oriented Databases

2007-11-23T23:18:00.000-05:00

Was Ray Ozzie right? I've been seeing a fair amount of information circulating about RDDB. RDDB is a document-oriented, RESTful database. Lotus Notes is a document-oriented, old database (and email program, calendar, workflow form generator). The cool thing is that all of my Lotus Notes knowledge, which was rapidly becoming worthless now has a chance to be valuable again if this sort of thing takes off again.

I was in a Rails training class with this guy who made his career on the back old Notes and their ugly little autogenerated web apps. He kept asking, "how do we keep this from becoming another Lotus Notes?" At the time, I was thinking, "how do I keep myself from being one of those guys who looks at every new technology in terms of some old technology that he actually had time to understand?" Now, looking at the potential resurgence in document oriented datbases, maybe he was right...nah. Rails is nothing like Notes, because, uh...it's dynamically typed? No, Notes had that. Works with Java now, thanks to JRuby? No, IBM took care of that with Domino+WebSphere. Hmmm. At least 37 Signals hasn't been sold to IBM while David H. grabbed the lead role at Microsoft.

Oh, wait, there is a difference. REST! I had posted a long while back about how my cousin made fun of me in front of a huge room of people because he asked what the two types of web services were (hoping for document oriented and remote procedure call as answers) and I said "REST". He said the REST people were like the Amish, got some laughs and proceeded to promote WS-*. At lunch after his talk, Bob Martin took my side, so I knew I had made at least an interestingly wrong decision. Tim Bray has a nice round-up of other smart people that were wrong about that. So sweet to be right about that, even though he could kick my shins in programming any day of the week. Except Sunday, when we Amish polish our wooden laptops. Anyway, maybe RDDB is more Amish Oriented Database than Rails Oriented.

So Rails has REST, and even though I swore off creating RESTful apps when the initial URL implementation was polluted with semi-colons, it's the real deal now. Not so much for the application, but for the API. It's great to have APIs that are platform independent. Most things like a Lotus Notes wouldn't bother with this- doesn't help with platform lock-in. I spent more time working with the Lotus Notes ODBC driver to suck data out of the system than was reasonable. Of course, they did support COM, which was the REST of the day (if CORBA was the WS-*).

Now, REST still needs a solution for the old federated identity problem (so many websites, so many passwords, so many usernames)...but there are plenty of options. Maybe my cousin is right about CAS? Seems Amish enough for me.

Feral Programmer

2007-11-15T14:09:00.000-05:00

Feral Programmer: A programmer who has no notion whatsoever of common, socially acceptable programming practices or knowledge of available tools and technologies. The feral programmer is not necessarily a bad coder, and sometimes could be labeled a savant, but is generally impossible to work with due to the complete lack of any common framework of understanding. Somewhat of a corollary to Not Invented Here syndrome, the feral programmer is either just completely unaware or uncomprehending of most inventions in the field. Behaviors include: creating flat file databases, version control systems or their own web service architecture built of packet sniffers, frequently using only Perl.

iPhone SDK trumped

2007-11-06T10:33:00.000-05:00

A third note in the open platforms series...Facebook vs. OpenSocial, Second Life vs. Multiverse, iPhone vs. Android. Now strictly speaking, the iPhone is the least walled garden of the three, but the lack of Java support means that it's a little like Facebook. Even with the vague SDK announcement, all applications will have to be custom built for it. Fortunately you can still use the iPhone to get to your Google docs...

Where's my GPhone?

Android is the first truly open and comprehensive platform for mobile devices. It includes an operating system, user-interface and applications -- all of the software to run a mobile phone, but without the proprietary obstacles that have hindered mobile innovation.

We recognize that many among the multitude of mobile users around the world do not and may never have an Android-based phone. Our goals must be independent of device or even platform.

...tear down that wall.

2007-11-01T12:28:00.000-04:00

You have to love the timing of the Open Social API announcement. Facebook is not worth $15B or $3 for every person on earth...

As pmarca puts it:

This is the exact same concept as the Facebook platform, with two huge differences:

With the Facebook platform, only Facebook itself can be a "container" -- "apps" can only run within Facebook itself. In contrast, with Open Social, any social network can be an Open Social container and allow Open Social apps to run within it.
With the Facebook platform, app developers build to Facebook-proprietary languages and APIs such as FBML (Facebook Markup Language) and FQL (Facebook Query Language) -- those languages and APIs don't work anywhere other than Facebook -- and then the apps can only run within Facebook. In contrast, with Open Social, app developers can build to standard HTML and Javascript, and their apps can then run in any Open Social container.

In keeping with my thoughts on 3D virtual worlds, the walled garden approach is not the end state, the "internet of virtual worlds" is. I think we're going to see this with Facebook, MySpace and others eventually becoming part of an "internet social spaces" connections between bits and pieces of ourselves, photos on flickr, blog on blogger, twits on twitter, profile on myspace, maps on geocommons, restaurant reviews on yelp, plus whatever we think of next week. Tying all of this together is going to be remarkably cool.

Technology like OpenID is really useful, and that's why you see Enterprise players like Oracle on the agreement. Now Facebook has a great UI and momentum, so there is still a really huge chance that this effort will fail if it turns out that no one uses it effectively, but I think there will be at least one innovation out there that will cause Facebook to pull down the wall towards interop of their data...

Official Gmail Blog: Code changes to prepare Gmail for the future

2007-10-31T13:42:00.000-04:00

Official Gmail Blog: Code changes to prepare Gmail for the future

How sad is it that I get excited about a mostly invisible update to a web based email program? Nonetheless, I am "stoked" for the gmail updates, and only slightly concerned about potentially losing some GreaseMonkey awesomeness.

Pete Lacey's SOA definition

2007-10-22T09:03:00.000-04:00

Read this SOA definition from Pete Lacey this morning and it almost makes sense...too bad it's not what most people mean.

So, then, what is SOA? For one thing, SOA is misnamed. It’s not an architecture in any sense of the word. It is, to use a Burton Group phrase, a mind set. It is the generally held belief that when implementing systems one should expose system functionality for general consumption directly from the network, as well as or instead of burying it behind a user interface. It is, as well, the belief that there is a great deal of value to be generated by retrofitting network accessibility into most existing systems. And it is the belief that this can only work if the means of doing so aren’t locked to a particular language, framework, operating system, vendor, or network architecture.

It's a bit hard to disagree with that as a good policy. However, having a bit of a linguistic bent, it makes me wonder if there isn't something fundamentally wrong with an acronym that is "misnamed". Mr. Lacey does suggest an alternative term for what he is describing: Network Oriented Computing. However doesn't that make it a worse definition of SOA (since the definition better fits another term)? It does. His final definition of SOA is "a technical approach to NOC that has a non-uniform service interface as its principle abstraction. Today, SOAP/WS-* is the chief implementation approach.". This is a better definition because it actually is an architecture. It also sits well next to ROA (Resource Oriented Architecture).

The killer final definition he offers is: "Business Service Architecture (BSA): An unnecessary term (also not an architecture) that tries to make the obvious something special. Aka, business analysis. Aka, requirements gathering." Reading this crystallized something important about why phrases like IT-Business alignment always bothered me- you align the IT systems with the business via requirements gathering. Now, maybe you are doing it at high level and want to give it a special name, but the dangers of abstract terms are that people can look like they are agreeing about how to do something, but really have completely different ideas about what they are concretely going to do.

Hmm, that might be a topic for another post.

Metaverse Skeptic

2007-10-10T20:48:00.000-04:00

I've been an extreme skeptic when it comes to virtual worlds for a while now. I really can't understand how people can get excited about things like Second Life. None of the arguments for it as a great business seem to make much sense. The false scarcity markets for real estate seem quite misguided. I really enjoyed working with Forterra the other day- it's definitely richer than chat rooms and teleconferences in terms of communication- but is it richer than video telecon? In one way it is, because you can walk around and such, but that also imposes the limitations of the real world upon communication: you can't hear someone if you walk too far away.

I left a highly skeptical comment on James Au's post on 7 reasons why business execs should care about second life. There needs to be an Internet of sorts, so that the virtual worlds can, well, internetwork. You can't say Second Life is that important, because investing so much in something owned by another company is putting all of your eggs in one basket. It's like starting a company that builds Facebook applications- when (not if) Facebook gets acquired, your whole business model is in jeopardy. Better make a quick buck.

The world isn't just about open platforms. It's about interoperable platforms. One thing that makes adoption of new GIS systems possible is that the data can be transitioned from one to another because all of the stuff is connected to real world coordinates, and it is possible to translate from one coordinate system to another because the both refer to a place on the real earth. I am not sure what the virtual coordinate system is. With the Internet, it's the unified addressing and naming schemes. There can only be one 220.231.23.123 on the public internet.

In this light, the iPhone looks really dumb. You can't install apps on it. Okay, WiFi works and the phone bit actually connects with the real phone network, but it's not an open platform. Few cell phones are unlocked to connect to multiple networks, but Apple seems to be actively discouraging this. To me, that's a sign of a bad business model- easily defeated. Just make the money on the device- not the lock-in.

It seems like the right business model is have an open and interoperable platform. It cuts off competitors, but allows you to benefit from the network effects of others innovations.

The Alignment Trap

2007-10-09T23:29:00.000-04:00

Lots of good articles in the Fall Sloan Review of Management which I am just getting around to reading...

Avoiding the Alignment Trap in IT
One of the big ideas in IT Enterprise Architecture lately is Business / IT alignment. This is generally defined as the concept that the shape of the services offered by IT should reflect the offerings of the business. At the very least, it makes it easier for the CIO to justify why they are spending money on project X- because it is directly tied to a business need. To some degree it's a communication convenience.

In reality, the biggest benefits seem to come from simplifying the infrastructure and putting an emphasis on integration. A second article in the SMR by Cynthia Rettig really hits it on the head: The trouble with enterprise software:

...enterprise software may be just too complex to deliver on its promises. She also suggests that the next new thing — service-oriented architecture (SOA) — is not likely to fare much better, for many of the same reasons. There are no easy fixes, cautions Rettig, save a large dose of sobriety, clear-eyed analysis and emphasis on simplicity and efficiency.

If you want to read one of the best advocates for the odd beast known as SOA, I suggest following Bobby Woolf's blog or reading his new ebook. His old book is has had a solid slot on my shelf for a couple of years, but I still haven't felt the need to install an Enterprise Service Bus. (Despite the fact that I more or less use it as a design pattern for integration architectures, just without the overhead of actually having it be a running piece of software)

In any case, I am not writing myself out of a job here- the message is simple though: don't try to do everything at the expense of ending up with complexity. Keeping things simple and hitting the "Pareto important" requirements is the winning strategy.

Xobni

2007-09-21T00:14:00.000-04:00

I was really disappointed to read about Xobni. Bringing Gmail features to Outlook- and letting you 'pivot' your email by who it is from / to. I am disappointed because I have a great idea for an inbox assistant that I am conceptualizing, and this thing looks much better, if not really on the same track. The good side of it is that I will get a better sense of the market for Outlook plugins from watching their progress.

My idea is a little more focused on another aspect of email- the doing side of things, while Xobni is more on the analytical side. I still really want to try it, even though a few of the details are far from clear at this point...

I've also been checking out the trial for Sandy. It's another email type tool, except it's more of a command line using email. You send Sandy an email and she either updates your calendar, todos, sends a reminder or just remembers it. With the API support, you could think of it as a mail in interface to all of the tools you already use. I really wanted to like it, but it's not quite what I thought it was going to be.

Here's what my tool is going to do- help you get through your email faster and more effectively. I've been on the verge of declaring email bankruptcy a couple of times in the past months and I need this tool to stay at inbox zero (see below video for more on that).

Is CS "The Modern Liberal Arts Degree"?

2007-09-14T14:31:00.001-04:00

Interesting thoughts on CS and Econ degrees on Marginal Revolution.

From the Comments:
I was a CS major at MIT and was heavily recruited by Wall Street firms. My friends who were studying economics did not do nearly as well right out of college. The CS major is a modern "liberal arts" degree; you can do almost anything with it. In my case, I am getting my PhD in economics. I think studying economics instead of CS as an undergrad just signals that you want a high-paying job without having to do any hard work.

Or maybe proving you can make it through one of the more challenging academic programs in the world means there is less risk in hiring you to do anything...Anyway, I don't have a CompSci degree, despite taking most of the 400 level courses, and I've never missed having one (that I know of).

I do really like the idea that CS is just a tool for getting involved in some other industry.

Another Comment:
This is all kind of funny to me. After graduating with a BBA Econ in 1999 I made a play at the CS world. Realizing I was not willing to learn any real programing I went to get my PhD in Econ. Now I sit all day... programing.

Funny world.

And you can now just follow along online...

Most frightening sentence on my resume

2007-09-11T14:39:00.000-04:00

"Wrote Visual Basic for Applications code in a consulting role to support a multi-user client/server time reporting application in Microsoft Excel"

I feel the need to add a line specifying that I was not the architect on this project who decided that doing hub-spoke replication in Excel was not only feasible, but a good idea. I was just called in when it was determined that the original team had made a big mess when they tried to "scale up" to 50 users. At least it was only a month seven years ago. I suppose what really would make it scary is revealing the customer...but I can't do that to them.

This was one of the first instances I had seen of a non IT department wanting to do something technical, but being shot down by a nascent CIO office in terms of being allowed to do IT projects. They thus went with what they had on their desktops as the development platform.

Anyway, I think that project is getting dropped from Resume 2.0.

Secondary Actions in Web Forms

2007-08-28T17:12:00.000-04:00

Just checked out this interesting article from LukeW on secondary actions on web forms. The sum of it is that people don't complete forms faster when you make distinctions between primary (submit) and secondary (cancel) buttons. Still, as Luke points out- hitting cancel after filling in a long form really is not good. Do you really need a button for that in all cases?

I really enjoy the data driven approach of using eye tracking data to help make decisions on form elements. Unfortunately he missed my current preferred option: Bolder font-weight on the text on the primary action button, lighter or normal font-weight on the secondary actions. Since we have CSS classes for those button types on my project, I was all ready to make some changes...looks like I won't be making any, but it's still worth the quick read.

Authentication in the database- revised

2007-08-14T11:42:00.000-04:00

There are some people out there that believe there is one right way to build an application. The people that believe in best practices. The people that fall victim to Harwell's laws. (Particularly #1, "People always try to use their experience even if it doesn't apply to the current situation." and #3 "If a manager doesn't know how to improve an organization, then he/she will change it to look like the last organization that he/she remembers")

The particular issue I am dealing with has to do with web application security architecture, but it comes up again and again in architecture, particularly in web services. Where does authentication live and where does authorization live? And what trusts what. In particular, should a web application authenticate a user's x509 cert or Kerberos ticket, and then pass user information on to further systems, or should it pass the users security information on to further systems, such as databases. In other words, should authentication performed at each stage of a request that passes through multiple systems?

If you ask Oracle any question of the form "Should X be in the database?" the answer is yes. Apparently Microsoft is training their people to think the same way. It sure works great to induce vendor lock-in.

Somehow I am of the opinion that the first system a user comes in contact with should perform the authentication, that system should authenticate to other systems- as a system, not a user, and that their should be trust in the initial user authentication.

Having the database re-authenticate the user seems wasteful...and of no value.

Process Overload

2007-08-08T09:00:00.002-04:00

I am completely sick of all of the processes that have to be constantly running on my Windows machines. The things that particularly annoy me are the "update check" software- I am looking at you menacingly iTunes, Java, Adobe and Windows Update. Why on earth do these programs have to run constantly? Can't they at least just have a scheduled task to run at a particular time? Or how about copying firefox and just checking for an update whenever I start your program. I think Quicktime asks for updates more often that I actually use the software. I don't need a background process running constantly to tell me a piece of software that I haven't run in two months needs to be updated.

Perhaps even more annoying are the "helpers" that run to make various software applications appear to start up faster. I suppose this is just keeping up with the various tricks that Microsoft employs to make their stuff startup faster, so it's fair to blame them for starting the arms race.

And the final annoyance is the difficulty in seeing which svchost.exe PID goes with which service. How hard could it be record the name of the service, so that I could in a glance see which one is sucking CPU or RAM? I suppose I could actually code a solution to that myself...but I shouldn't have to. In fact, someone already did:

Process Explorer- this is what task manager should have been...

A dozen readings later...

2007-07-18T12:04:00.000-04:00

I guess I am finally reaching the "advanced" Rails tricks, since I just used this. (Although, I have to admit the origin was a ryanb tutorial on editing multiple models, not from repeated readings...)

from Agile Web Development with Rails

An Extension to Ruby Symbols

(This section describes an advanced feature of Ruby and can be safely skipped on the first dozen or so readings....)

We often use iterators where all the block does is invoke a method on its argument. We did this in our earlier group_by and index_by examples.

groups = posts.group_by {|post| post.author_id}

Rails has a shorthand notation for this. We could have written this code as

groups = posts.group_by(&:author_id)

Another kind of DVR

2007-07-12T15:02:00.000-04:00

Over at boingboing there's an interesting piece about the future of streaming video. "It's a bummer to consider a future in which broadcasts -- which we can all see and record -- are replaced with geo-locked, streaming crippleware netcasts.....All that said: top marks to the first person to demonstrate a working, reliable solution for watching and recording the CBC Dr Who episodes from anywhere in the world."

Right now, I am living in a bit of a gilded Tivo age where I can record just about any signal that comes into my home by intercepting its output right before it goes into the TV (which is basically just a monitor, I have never actually used its built-in tuner). The hugely hacky piece of the whole setup is the IR blaster to control the cable box by pretending to be a remote- with my new Verizon FIOS Motorola set top box, it just doesn't work all that well. New TiVos support the Cable Card standard- at the price of losing control of stream + $800 and a monthly fee. If restrictions are put into place on those new TiVos, for example flags on HD content that make it disappear after a week, then we might actually have a situation where you need to TiVo your TiVo (by capturing its output) to get the content into an open and safe format.

What is really striking about all of this is that the IR remote control is the open interface by which I can control the output of the cable box. Without that, it's basically useless because I couldn't choose what to capture. The whole weirdness by which all IR remotes are similar in that most can imitate the others but just about every device comes with a bunch of pages in the back that allow you to program the remote to control a variety of other devices.

I wonder if we aren't going to see a solution where the output going full screen to your monitor is going to be intercepted in a similar fashion, and then dropped back in to a capture card. It's really insane. The part of the computer that is analogous to the IR interface of the remote control is the URL. It's nice to take a step back every now and then and appreciate the beauty of that- URL addressable content. Now if we could just be assured that we'd be able to look at it when and where we want to.

s3sync.rb

2007-07-10T23:22:00.000-04:00

I am finally backing my life (and my company's data) offsite with Amazon S3. I have been diving into this full on and it's quite cool. It took me way way too long to get it all set up. Nearly 4 hours to get a 7 line shell script in a cron job, due to a serious of yak-shaving events

In any case, it was basically easy with s3sync (the Ruby version, I believe the Perl version is abandoned...) Here is a decent link on getting it done. I actually used Marcel Molina's s3sh shell program a bit, simply because the error messages it dumped were vastly better than the HTTP 403s I was getting from s3sync. I just didn't find s3sh as "scriptable" due to my lack of shell scripting expertise at the moment. I am just glad I have a place to practice.

It's definitely worth reading a bit on the main AWS site about S3 before diving in. I wasted quite a bit of time due to not understanding the flexibility of the key/bucket system's handling of directories. And the minor bit about buckets having to be unique across all users.

Persistence Scaling

2007-07-07T23:21:00.000-04:00

Interesting interview with Michael Stonebraker (the Ingres guy) from the ACM.[via GLinden]

The basic thrust is the RDBMS is slowing fading in its place at the heart of IT systems everywhere, for a variety of reasons. I personally have seen a lot more in the embedded database and in-memory database category lately, but he's talking about the relational model itself failing for warehousing, streams, text, and list processing. I don't know if the database concepts behind his current company Streambase and stream data processing are that broadly applicable. Still, his depiction of the areas of the relational model where there are scaling or intelligibility problems seems right on. I am not sure I get the following, but it makes sense in a way too:

"It’s the same case in scientific and intelligence databases. Most of these clients have large arrays, so array data is much more popular than tabular data. If you have array data and use special-purpose technology that knows about arrays, you can clobber a system in which tables are used to simulate arrays."

Vector oriented programming?

Stonebraker makes a nice point about how the ActiveRecord component of Rails is in effect a cleaner way of interacting with data than SQL. It really is. That said, I wonder if LINQ is on his radar? Sidetracked by this great post on from 9 till 2 on the 'Rubenesque' (That's Paul Rubens, not Matz) status of c#, which says the following about LINQ:

"Historically, there have been more Microsoft ways to access the Northwind database than they are rows in the Customer table. OLEDB, ODBC, DAO, RDS, JRO, RDO, SQLXML, ADO, ADO.NET, Entity Services....I suspect that DLINQ will probably not be the last ever ever ever in this data accessing periodic series."

And tellingly:

"LINQ has that big benefit of being able to treat relational, XML hierarchical and in-memory data objects all with the same query syntax, allowing you to swap store types at a drop of a hat. The nagging doubt though is that this may be the same big benefit akin to not having to use SOAP over HTTP. How did that go again? Something wonderful for a demo, but something that may not actually be a real pressing 'need'. There is a school of thought that when you're working with a relational database rather than a collection of in-memory objects, then you should not lose track of the various nuances and advantages of the stores - abstraction to save typing can come back to bite you?"

Yeah, like if you're doing Rails finders without :include? Ooops, N queries just ran. Still, I like the idea of s3record. As Nutrun says: "I have occasionally participated in conversations around the subject of the database as a product with an expiry date, destined to eventually be replaced by highly distributed data storage models. Although S3's data storage and retrieval model looks presently better suited for larger units of data (e.g. media content), it would be interesting to investigate how it could be applied as an Object persistence service."

This is obviously the wrong implementation for such a scheme, but it points in one of the directions we're exploring. The concepts of streams, spaces, distribution are the future of scaling and persistence. It won't be a simple design decision whether a database is at the heart of your system or not. It's going to be interesting...as usual.

Java EE 6: profiles versus subtraction

2007-07-05T09:57:00.000-04:00

The JCP has produced a Java Enterprise Edition spec that even "Mr. Spring" Rod Johnson can approve of. This is actually a really big deal, as Java containers have become bloated by implementing expensive pieces of the spec which are seldom used. What they are doing is allowing for compliance to subsets of the standard, with these subsets referred to as profiles. They are also allow for extensibility, with a standard for extensions to the spec.

The "profiles" approach can lead to much confusion, as anyone who has tried to figure out why their Bluetooth phone and Bluetooth car can only communicate on the most basic levels can attest. The OGC also has been using this approach for GML data standards, and seems like overkill when you are just talking about XML. However, in general it makes a lot of sense when you need to preserve the whole from a service implementation standpoint. But really, sometimes just subtracting things from standard is a great thing. (JSF anyone?)

I wonder if there will be profile for app servers to run JRuby on Rails?

CHAOS = BS

2007-06-25T09:01:00.000-04:00

Anyone who has looked at software development methods, myself included, is probably familiar with the CHAOS report of The Standish Group. I have blogged before about Robert Glass's article that the numbers reported that group simply are not supported by other studies. I was happy to see that referenced on Herding Cats.

The effects of the data they report (189% cost overrun average) are negative for the IT industry. They are used to justify all means of odd process controls and contribute to a general excessive scepticism about the possibility of making a software project a success. The lack of context given for the data (actually, the data is kept secret) means that all of this is based on a very shaky quantitative foundation.

Having just finished "Fooled by Randomness", I am going to take another look at the Monte Carlo approach to estimation. I used it in a couple of cases to provide broad estimates for large projects, but all of the numbers came out quite low. I still the think the only responsible thing to do is to provide a range of estimates, especially when the nature of the work to be done is uncertain. And it usually is. The real benefit of the simulation approach, from the fooled by randomness perspective, is that they allow to take account of the black swans. If you are using a purely inductive approach, you can't account for things that have never happened. Like a 3 day power outage in your server room. Or system administrator that changes the name of your source control server and costs 10 person days of work. Or a new programmer that checks his whole c drive into the source code repository. Well, those 3 have actually happened to me, but they aren't the sort of thing I usually factor into my estimates, particularly when using the methods from "Planning Extreme Programming", which are purely based on yesterday's weather.

The real challenge here is estimating the shifting sands. Until you have a feel for the rate at which requirements change, you are only going to be able to estimate based upon what people thought they might want something they don't understand to do. Of course, if there is a reorganization in a business you are supporting, and there usually is at some point, the rate of change can shoot up. This is where the difference between methods really starts to kick in. If you are using a method that "embraces change" (the subtitle of the XP book), then you can leverage change to be the engine that carries the organization forward. If you try to keep the requirements frozen, you will be one of the projects reported when the Standish Group calls and says, "tell me about your failed projects".

Many Methods

2007-06-19T20:06:00.000-04:00

Cognitive Dissonance development (CDD) - In any organization where there are two or more divergent beliefs on how software should be made. The tension between those beliefs, as it’s fought out in various meetings and individual decisions by players on both sides, defines the project more than any individual belief itself.
-Scott Berkun

I've been there...wondering whether it is better to give up the argument because the other person is so incredibly stubborn. And it would have been better for the actual product had I given up. Not because my idea was worse, but because trying to do it two ways at the same time was worse than any one way.

I've been on projects where people waste all of the time trying to pick a product. I now have the confidence to tell people that most products really are not that good. Particularly ones that are intended to make development easier. So many of these create frameworks that are more complicated than what one was trying to accomplish in the first place, but offer less flexibility.

I was on a project that was trying to replace an ArcView 3.1 data entry and analysis application with about 25k lines of custom scripts with a Java web application based on ArcIMS and ArcSDE. This was in 2001. With a primordial version of Netscape. When all ArcIMS did was push jpegs down the wire. Oh, and they wanted to host it at a remote site. Adding points in ArcIMS at the time was a serious challenge. Adding lines was nearly impossible. I quickly quit. The project kept growing until the first deployment was such a failure they had to hide the bodies.

Obviously a place where Agile would have made a big difference. "Architects" in the ivory tower of the CIO shop had decided that we must have web apps for everything. If they had just put the demo app in front of an actual user, the whole project could have been quickly canceled.

I was listening to Uncle Bob Martin's Agile Skypecast while mowing the lawn on Sunday. It's a must listen. Despite all of the BS and corporate maneuvering around the buzzword, agile has deep principles behind it. I suggest ditching the buzzword and sticking to the concepts.
Early and frequent delivery of working code. Daily customer feedback. Developer testing (I am still working on mastering that one.) And the big one: Embrace Change. Don't fight requirements volatility, leverage it. (of course the trick is keeping the requirements stable during the iteration...)

It is interesting how Scrum leaves a lot of these things out and is, as Bob says, more of a management process that doesn't have much to do with software.

My favorite part of the agile universe remains the XP planning game. Point budgets as a means of giving multiple customers a voice, realistic estimation feedback, and ever so slightly, possibly, a little wee smidgen of fun. We could all use that from time to time.

Anyway, Uncle Bob gives the principles a good run through, worth the time to listen to a podcast.

I never really liked cats...

2007-06-14T14:56:00.000-04:00

...but this has pushed me over the edge. The "LOLCATS" madness will pass.

Bad code...are you 'aving a laugh?

2007-06-12T00:33:00.000-04:00

A little bomb I defused today in an ASP application I am replacing...weirdness explanation at the end.

nextTuesday = ""
showTuesday = ""
gotTuesday = 0
todayDate = weekdayname(weekday(now()))
'response.Write(todayDate)
if todayDate = "Monday" then
holdNextTuesday = date
nextTuesday = formatdatetime(date,1)
' response.Write("got it")
else
j = 0
'do until gotTuesday = 1
do while showTuesday = ""
j = j + 1

if j > 10 then
response.write("ERROR: Infinate loop
" & j)
response.write(weekdayname(weekday(nextTuesday)))
response.end()
response.flush()
end if

nextTuesday = DateAdd("D",j,Date)
'response.Write(weekdayname(weekday(nextTuesday)))
if weekdayname(weekday(nextTuesday)) = "Monday" then
gotTuesday = 1
holdNextTuesday = nextTuesday
nextTuesday = formatdatetime(nextTuesday,1)
showTuesday = nextTuesday
end if
'response.write(nextTuesday)

loop
'nextTuesday = ""
end if

thisMeetingDate = holdNextTuesday
'response.write thisMeetingDate
previousMeetingDate = DateAdd("D",-7,thisMeetingDate)

----
The meeting date was moved from Tuesday to Monday and they didn't change the variable names. I have no explanation whatever for the "infinate loop" check. That one just kills me.

Biggest JRuby (Java) Lie

2007-06-12T00:24:00.000-04:00

"Nearly impossible to crash runtime or VM" -From Nutter and Enebo's MountainWest RubyConf presentation.

I managed to do it on a daily basis for years, the simple recipe is to add WebLogic to the project.

This is not to take anything away from their achievement with 1.0. It's a Java application running nearly as fast as a c application. I am very curious to see if it's as compatible with horizontal scaling...

Many Lo-Fi Prototypes

2007-05-30T23:33:00.000-04:00

I've been listening to the audiobook of 10 Faces of Innovation by Tom Kelley of IDEO. He has a great insight into something I sorta knew, but had never thought about why it was so.

The basic idea is that showing just one prototype to a customer is bad. This tends to force them into a binary/reactive decision, one that is often determined more by their relationship to the creator of the prototype than to the item itself. Given multiple items to evaluate, the attention turns more to the items, and more feedback can be gathered, since it won't be perceived as direct criticism.

The telling analogy Kelley provides is that of a husband that is told by his wife that she has purchased a new dress. She then tries it and asks how it looks. Of course, it is difficult for the husband to say anything negative, because the question is really about how she looks wearing that dress. However, given multiple options to consider, the husband is actually more free to give honest feedback.

Now, it can be expensive to buy multiple dresses, or build multiple prototypes. The answer then is to use lo-fi prototypes: whether it be pictures in a magazine or crude drawings. Still, the more prototypes that are created, the less pressure there is to get it right the first time, which is a crushing force against innovation.

Having heard his brother speak at a conference in 2006, I love what IDEO are doing. If didn't need to make quite so much money, and had any confidence in my abilities, I would like to work there...just to live at that creative level every day would make me so happy.

Another retro RSS reader.

2007-05-24T23:18:00.000-04:00

A must for Neal Stephenson fans, the RSS telegraph gives you all of your feedy goodness in Morse code.

Saw the steampunk turntables on BoingBoing- might have to trade in the 1200s.

Steampunk...

Empowerment and Trust

2007-05-24T00:03:00.000-04:00

I've had this in draft for a while, can't figure out what the point was, but I don't want to think about it anymore. I've seen so many people chafing about the enterprise IT shop at one of my clients...unfortunately only I can do is add another forlorn whine to the choir.

My friend sixty4bit recently blogged about some trust and power issues at a client we have both worked for. Their behavior is bureaucratic in the extreme, they run all projects through a slow and cantankerous review board for all of their technology selections. In the end, the review board exists because the organization does not trust software developers and does not want to empower them to make decisions. The end result is that they slow everyone down and drastically reduce the effectiveness of the organization. The overall atmosphere created is one of fear of anything new, where people stop asking to try new things, because it isn't worth the time to get them approved. In a lot of cases, you have to get approval just to try something.

Laughably, this client is attempting to move to agile development processes, which are certainly focused on empowering the people who are doing the work to make decisions (although that is more of a Lean idea), while continuing on the opposite track in terms of empowering individual projects to meet their customers' needs.

In reality, particular technical decisions about technology selection can be contentious, especially in regards to projects across the enterprise choosing incompatible technology stacks. There is also a need to coordinate purchases across projects to gain negotiation leverage. Even more important, although completely overlooked by these bozos is that the license terms for all software need to be read by actual lawyers. However, it's clear that the real issue here is one of trust and power, tied up with the fun of getting to decide how to spend other people's money.

I am working on a devious plan to: a) get them to trust me, b) keep them feeling powerful, and c) allow for innovation and progress. Reading "Influence: Science and Practice" by Robert Cialdini tonight. Maybe that will help with a and b.

$11M for uLocate

2007-05-17T23:05:00.000-04:00

Pretty big round for uLocate. Famous for providing support for Flagr, the mobile version of MapQuest etc.

I tried to download the new Where (how much did that domain name cost?) app to my phone, but it's not supported as of yet...

Their mobile technology is sorta cool- going for the whole write once run anywhere thing for mobile apps with an xml style application definition. A location specific version of something like mFoundry. Of course, when they don't support your phone, it's a little harder to buy into that concept.

Anyway, XML as a programming language? Just because it worked sorta okay for HTML doesn't mean there aren't better ways to do it- especially when you are talking applications.

Mobile Platforms

2007-04-29T16:52:00.000-04:00

For work purposes I am currently carrying two phones- a Blackberry and a Palm 650. I like both of these OSes, although the Palm OS does crash quite a bit. I've also been testing a couple of mobile GIS applications running on Windows Mobile 5. We're trying to figure out where to go next with it- RIM, Symbian, Linux (iPhone), etc.

While I like RIM and Palm OS, with a edge to RIM, Windows Mobile is completely unappealing to me. The main plus from a development standpoint is that it is relatively standard across devices. However, this is easily eclipsed by the usability, the memory consumption, and the way in which closing most programs doesn't close them- it just hides them. The stability varies pretty widely across devices, but it's about on the level of Palm OS.

I am still waiting for the mobile OS that implements virtual memory paging (with an SD card as the pagefile holder). It might be slower, but the RAM is so easily consumed in simple apps, it would enable a variety of applications that just don't work now. I know, you just have to wait a couple of years and it will all change, but right now it's a very tough decision if you want to figure out which mobile OS to develop on.

Now, some might say- why not just do mobile friendly web applications versus applications? For a lot of applications, you need more interactivity than the limited browsers on the phone provide. For others, the constant lag is the impediment. Overall, it's an area of potential, but limited to only the simplest applications today.

However, the mobile device is where some of the virtual machine technologies can really come to bear. In particular, Java Mobile Edition is pretty widely available, even if the apps look as out of place as they often do in Desktop OS land. So, you can do a Java app and get a reasonable percentage of phones to work, although they all seem to implement slightly different profiles.

Now we have the Apollo, Silverlight, JavaFX model of what are in reality enhanced web browsers that build UIs out of things other than HTML. JavaFX mobile is one picture of how things could go in that direction, where the environment could provide all of the basic phone functions, in addition to providing a decent programming environment for content delivery. Of course, I am not sure how much of that is really one thing, or if it's a set of technologies that Sun is lumping under one banner.

Anyway- let's say I have an opportunity to port one of these mobile GIS apps to a platform or OS- which way should I go?

Productive Thoughts

2007-04-28T10:28:00.000-04:00

Scott Simpson: I love saving things to del.icio.us w/ a "toread" tag. Offloading work to the future, incredibly productive, version of myself. [link] Kind of a cool idea for using del.icio.us, especially if you use the firefox plugin. Just highlight a bit of text and then tag it. So easy. I haven't gotten as much out of the more advanced del.icio.us bookmarks plugin, but that might be because my current bookmarks are a pile of junk.

The main garbagey bit about del.icio.us is that there is no good way to manage tags with the default apps. Idea! I am going to hunt around to look for someone that has done that or start digging into the API a little bit.

I guess one of the main competitors to this stuff is Google Notebook, but I really don't enjoy using that for some reason I don't yet understand. Those little links in the search results pages are tempting thought.

del.icio.us friends

2007-04-26T15:38:00.000-04:00

Thinking again. One of the things Steve Poland is going after is the concept of webothlike:

Basically the idea of WeBothLike.com is to connect like-minded people — people that have the same interests. You’ll go to the website and answer questions, as many as you want. The more you answer, the more profile data we have on you — and the more we can contrast you with others in our system, and match you up with them

I think rather than asking questions it would be better to try a recommendation style algorithm on your del.icio.us links, or bookmarks, etc. so that you could get a match score for how similar you are to other people. Basically you could submit all of your links and get a quick score for who else is tagging the same things as you. You could take a similar approach with OPML comparisons to find who is reading the same things as you. On the one hand, the output of this would be suggestions about what to read- and really, who doesn't have too much to read already? On the other hand, a cooler output suggested by WeBothLike would be to find people to scheme with.

Anyway, I have a feeling someone has already done this somewhere, but it's not part of the main del.icio.us application, and it probably should be.

Google Reader Reader Robot

2007-04-25T17:05:00.000-04:00

How about a project that downloads your unread feeds from Google Reader, uses a speech synthesis program to convert them to mp3, adds the mp3s to a podcast server, which you then script to sync to your portable device? Then you can listen to all of your news on the way to work, as read by your choice of a feminine or masculine robot.

If you like that one, wait until you hear about my idea for Google Reader Printer for my poor friends that have no internet connection at their desks.

Hopefully someone else is working on a good Podcast transcriber, like Podzinger. Once there is a great speech to text solution, I can come up with a convoluted name for that.

Okay, these are lame ideas, but someone's got to pick up for that Techquila Shots guy.

Reporting and OR/M

2007-04-25T16:00:00.000-04:00

Just an off the cuff thought about using Object Relational Mapping (Hibernate, TopLink, ActiveRecord, etc.) when doing reporting: Does it make sense to have objects corresponding to specific reports that result in denormalized data, so as to hold the results of the query? It makes it a little more sense if you have a view corresponding to the report, and then a object corresponding to the view. At some point it does seem to cross a line of excessive abstraction and an overgrown data model.

Simple

2007-04-23T23:35:00.000-04:00

I think this is the route I am going to go when I have to do another Java web application. I have become improbably enamored with horizontal scalability. The primary downside is that they started using xml configuration files. Curse you XML configuration. The XML config is just there to compensate for the lack of dynamic language features...

Seriously though- the old way of building Java web applications is seriously threatened by the latest .Net stuff on one side, and the dynamic language stuff on the other side. I think the .Net 3.0 is not helping much on the complexity angle.

It makes me laugh to think how much the REST architecture resembles the Java Servlet Spec. All of these frameworks to abstract away HTTP verbs, when that was what really needed to have application structures that are concordant with the architecture of the web itself.

Ruby Community

2007-04-03T14:20:00.000-04:00

I don't know who "Dan" is, but I like the way he rolls.

"I have no idea what the "Ruby Community" is. How do you define
membership in this community?"

No, no, the "Java Community" has a defined static membership, but the
"Ruby Community" is more dynamic. As long as you respond to the right
messages, you're just assumed to be a member.

Productivity tip?

2007-04-02T16:41:00.000-04:00

I am considering switching back to using Lynx as my primary browser. My theory is that it makes the web so much less compelling that I won't be as distracted by it. Slightly easier in terms of keyboard navigation so I don't have to touch the mouse. Runs great in cygwin too. The biggest downside I am seeing so far is that Google Maps doesn't run very well.

I don't see them doing much to fix that. Oh well, back to 9 tabs of FireFox I suppose. Apologies for the mind drip.

Language Games

2007-04-02T11:44:00.000-04:00

I've been following with great interest Jeff Atwood's series of posts on Alistair Cockburn's work on software development as a cooperative game.

I was wondering if he was going to dig down to the philosophy level on this, and he may, but this concept of the cooperative game is an extension of Wittgenstein's concept of language games, first outlined by Pelle Ehn. Reading Wittgenstein, more than anything else, really opened my mind to an incisive type of analysis that attempts to dissolve a seemingly unanswerable question by proving that it is nonsensical out of the context of the particular language game that gives it sense.

The digression between the purported "process" that underlies a project and the actual intuitive set of rules that guide the behavior of and between individuals are clearly different things. It's obviously important to pay attention to both. This is why Cockburn's approach to agile "method shaping" is so important, every strict process must itself have a meta process by which it is applied to any particular collection of individuals.

I really like Alistair's interview with Bob Payne (especially the parts where Mr. Cockburn is speaking) on the Agile Toolkit podcast.

Mingle- ThoughtWorks hotness?

2007-03-27T10:26:00.000-04:00

Wow, the high priced consultants @ ThoughtWorks have broken into commercial product development. This is going to be interesting.

Mingle is apparently an agile project management tool. We've looked at VersionOne and Rally in that space (not to mention Team Foundation Server), but we're actually using Trac, which is hard to beat for the svn integration and price. I wonder how how much this pup is going to cost?

Communication Cost

2007-03-25T23:19:00.000-04:00

Just caught this great post from Lisa Haneberg that echoes some ideas I have had in the past, like a running clock that shows the running total cost of a meeting as it "progresses".

Top level costs: the cost of time for people to receive (hear, read) the communication and the cost for creating/delivering communication. If you send an email to your entire team of 50 people. And if the email takes 3 minutes to read, and 15 minutes for you to write - the top level costs are of people's salaries for the 3 minutes and your 15 minutes. Plus the costs of the email server time and space, etc... But the time is the largest cost. You may think this is a small amount, but multiply this times 100s of emails we deal with each day, and the costs add up.

If you book a two hour meeting with 15 people - the top level costs are huge.

But that's the just start of the costs. There are two other important types of costs...

I have thought about this in terms of meetings quite a bit. I've had lots of extremely expensive meetings to sit through. Meetings where our tax dollars were being consumed at unhealthy rates. Now, there are the people where having them in a meeting is actually more productive than having them cause damage through their other work, but it's hardly the optimal solution. However, applying this thought to email and phone calls is interesting.

Think before you send...I've been doing the whole "Inbox Zero" thing. It's really great that I don't have a huge unprocessed morass of email to look at it. I do have to get better about keeping up with my todo lists though. It's still easier for me to forget those because I don't have a blinking light telling me to look at them every time a new one shows up. Maybe it's not as fun to look at, because, unlike email, I already kind of know what is on my todo lists.

Jeffrey Veen and Merlin Mann were talking about how Google is a no phone call culture to some degree, everything is email, IM, with a definite preference for the asynchronous communication. Not sure how much of that is introvert culture and how much is efficiency, but it sounded like they were positively bombarded with email information there.

Maybe there is a real economic benefit to parsimonious communication, learning to be clear concise and brief saves money. Maybe this should apply to bloggers as well, I know there are people out there struggling to keep up with their feeds...on that note, I'll be signing off.

Treat Contractors Well (please)

2007-03-20T13:22:00.000-04:00

I like this post from Scott Berkun. (of the Art of PM book)

One habits many managers have is to dump the boring, unpleasant work of their team onto contract workers. The thinking is that full timers deserve the best treatment and contractors are mercenaries: they deserve whatever they get since they won’t be around long.

It’s a mistake - good managers finds a way to treat everyone well. And there are some reasons contractors deserve special attention

I don't think it applies 100% to government contracting, but it does make a lot of sense. Please treat my team well government overlords!

Wiki Patterns

2007-02-20T15:33:00.000-05:00

I have been trying to get a good collaborative space set up for my company forever now. It just has never risen up high enough in the priority chain to really become a collaborative effort...although I have had some very good support from a couple of people that really get it.

Anyway, I happened across this link to Atlassian's wikipatterns site at Web Worker Daily. So far it's a pretty basic set of patterns, but I plan on adding a few of my notes.

Here are some random things that I think are decent ideas for a making a wiki well used with the audiences I have been dealing with.

1) WYSIWYG editing. Most people don't want to learn a markup langauage. It really lowers the barrier to creative usage.
2) Make it the authoritative source. No one wants to look at or update data that might be wrong, if there is another place where it is definitely right. If the wiki isn't the authoritative source, link the other content in, don't make copies that will get out of sync.
3) Fun stuff. It's absolutely worth having a few pages where people can post silly stuff when they feel frustrated.
4) RSS Feeds or change summaries. Crucial if you are using it for anything new-sy, as opposed to reference.

I have set up MediaWiki and many others. MediaWiki looks like a really ugly piece of code. I had problems with it on some customer networks with strange proxy behavior. Instiki and most of the Ruby on Rails wikis are missing crucial features- like security models. I actually like the Trac wiki for development projects, even though the wiki isn't really the crucial feature. I am extremely disappointed Google snapped up JotSpot right as they were about to release a self-hosted version. As convenient as Software as a Service things like Blogger are, I just would easily prefer hosting the app myself. JotSpot, while also being vastly more than a wiki, really benefits from being the engine behind dojo.js development. Dojo is a great ajax lib, and makes Jot cool.

I never thought the Atlassian Confluence stuff was worth the price, but the patterns site is a good idea. In fact, I suppose that was the origin of the wiki concept in the first place, Ward Cunningham's c2 wiki for the portland patterns group is still the canonical example. How far we haven't come: web 2.0 v. Web 1995.

Google Earth and Standards

2007-01-25T21:19:00.000-05:00

I can't comment on the all points blog...they must have me marked as a spammer.

Anyway, I won't link to the post by Adena Schutzberg due to that, but here's the content, jacked from a Federal Computer Week article.

"I have a question on this:

Interviewed in the Google booth, which resembles the bridge of the Starship Enterprise, Painter [director of Google Earth Federal] said that although the public Google Earth uses commercial satellite and geospatial imagery, Google Earth Fusion allows federal agencies to manipulate and integrate their own geospatial imagery with the company’s software tools.

Imagery or software? Isn't Google a member of OGC? Is it moving forward on implementing those standards such that it can do both with ease? Is that not the point of OGC? Is DoD pushing Google to implement such standards? See for example: NGA Announces Requirement for OGC and Complementary Standards
Imagery."

I don't think there is a high performance streaming 3D imagery XML standard, is there? In any case, NGA is presumably rational. They are not going to sacrifice user experience for the sake of standards compliance, particularly where the standards board is as commercially fractious as OGC has been.

While there may be a requirement for support of those standards, this is already easily met by Google Earth in the form of reflectors for OGC Image Services as overlays or super-overlays to the base streamed data. All of the major data types can already be imported to Google Earth server to form the base layers.

To somehow suggest that Google should use an OGC format as their primary streaming format is a really bad idea for everyone- especially the data owners that would end up giving away their data. I remember the same insinuations being made against ESRI in years past, and then people deciding they didn't want to pay the performance penalty for standards compliance. Why don't standards committees ever look at what works best, and then choose that as the standard, instead of trying to prognosticate the market, technology, and user needs years in advance? While the shapefile was never an OGC standard, it was easily the lingua franca of the GIS community for a long time. The same with the good old .e00.

My wife worked with the poor fellow ESRI designated as their OGC standards body rep. He was not a person who loved his job. Basically he was a punching bag as he watched their competitors attempt to push things away from technology that would be compatible with proven success.

Manual Reverse Geocoding

2007-01-18T10:27:00.000-05:00

So this guy's letter was delivered without an address- just a map. Reverse geocoding at work. Humans rock- it's dead hard to get a computer to do this. Actually, maybe I should just say the the UK Post rocks, I can't see our USA civil servants doing anything with this besides sending it to the DLO or trying to arrest the person that sent it for subversive activities.

One of the worst experiences in my life was spending three days geocoding a dataset of points for Wien (aka Vienna, Austria) with ArcView 2.1. (ArcView would try to suggest a point, and I would try to move it to the right place.) House numbers there don't follow a very organized pattern, and I hope the errors I inevitably must have made didn't get anyone killed. Or didn't get the wrong person killed, or anything like that.

link via: /usr/bin/girl , which incidentally is one of the first blogs I ever read.

Beautiful Brochure...ugly product

2007-01-14T21:13:00.000-05:00

I have been doing a light evaluation of some software products in anticipation of some upcoming requirements for a project I am working on. The product that the architect in my position before me had been ready to buy is a real piece of work. It commits three violations of trust that no company selling software should:

1. Where's the product?
I don't trust that there even is a product when you can't go to the website and have any clue about how to buy the product or even a wild guess as to it's cost. You can download a gloriously illustrated brochure of bullet points of software features, with teeny tiny pictures of UI. You can read about solutions and services, but you can't buy anything? It sounds like a services trap: call us, we'll figure out how much money you have to set our price, then buy/rent our software and we'll finish it for you at an hourly rate, but own the code we write for you.

I guarantee there isn't a "there is no three step" install process.

2. UI design = sorry, we spent all of the design money on our brochure.
What does it say when a company has an obviously professionally designed and frankly beautiful brochure, but their actual product looks completely undesigned and frankly is just plain hideous? They care about people's experience until they have their money?

3. No mention of an API anywhere.
Monolith sensors on full alert. This thing has to integrate with my enterprise, I don't need to buy an ERP system where everything is a module of your system, versus a distributed architecture. (no one does need to buy a "complete" ERP system...ever) Currently dealing with the seemingly vast, but still incomplete, APIs of Microsoft Dynamics Great "Pains" has me watching out for software that makes itself hard to fit into a distributed world. This software isn't even a web app, so there's no chance of URL hacking.

Sorry, no dice. I am specifically not naming the guilty party here so that they don't begin any legal actions against me for defamation of character, or in case I decide to apply for a job there someday to rewrite the thing. I'll post an offensive screenshot at some point in the future on an unrelated topic.

Refactoring Ruby

2007-01-13T23:15:00.000-05:00

Jay Fields and friends are rewriting/porting Martin Fowler and friends' Refactoring: Improving the Design of Existing Code book to Ruby. I guess since the IDEs aren't there yet, we might as well get going on the manual process! Good show to Jay et. al. They've made a dent in the first section, hope they keep it up.

I still keep a copy of Refactoring on my desk. It's one of about 20 books I still need to keep around on my desktop. The content is somewhat timeless, and (was) not readily available on the internet. Still, the Jav-oid nature of the text makes it a little less relevant to my current world of Ruby. It also serves as a badge of good programming knowledge.

I've been selling off a lot of my other books on Amazon. Someone bought the Sun Certified J2EE Architect guide from 2002 for $16. Someone else bought Rod Johnson's old Expert J2EE development, the early edition, before he finished Spring, for $15. No takers on Tapestry yet at $10. Many books have a used value below a $1. I am debating what to do with them as it's not worth my time to sell them. The chances I need a Turbo Assembler reference have dropped about as low as they can, no library would want it, still, it was a good book in it's day, and it is sad that it is no longer relevant to anyone's life. I also think "professional Java Web Services" is probably not worth the shelf space. Well, maybe it will be recycled into a better book someday.

Zed Shaw

2006-12-28T09:00:00.000-05:00

Zed is king of the internet...for now.

If anyone out there is using this newfangled Ruby on Rails framework, Zed's (open source) Mongrel product has become the de facto container for running your app. It's a tiny little c program, and dead fast and stable. He co-wrote a little PDF on it that I had printed on a pillowcase. It's so comforting to know that my apps are running on Mongrel that the pillowcase serves as a little reminder so that I can go to sleep- without a glass of wine and one of those little blue capsules.

For a more general interest look at why Zed is such an interesting character, check out his essay blasting Joel Spolsky and Don Norman's articles on simplicity vs. complexity. If you aren't following that "debate", good for you...I'll summarize though- some people think simplicity is good. Some people think complexity is good. The people that are right think that good design makes complex tasks as simpler as they can be, but no simpler.

Metrics

2006-11-10T09:41:00.000-05:00

http://www.joelonsoftware.com/items/2006/11/10b.html
I love Joel's send up of management consulting applied to software development through the use of metrics.

My question is a real one though. The CEO of one of the places where I work as a "working" consultant (in terms of actually doing productive work, not just analyzing what other people do) just downsized a whole bunch of smart people. He wants to look at our metrics- including things like customer satisfaction. That's going to be tough to measure. How do you measure customer satisfaction in such a way as to rate people on it?

I think step one for me is figuring out who my customers are...I am taking over a project that one of the downsized guys was running, and I haven't even met some of the key stakeholders. When I do, I am going to be sure to remind them to send a monthly note to the CEO reminding him of how satisfied they are! If he gets 5 or 6 of those notes per month, that would be a good metric.

Of course, then the only way to show continuous improvement is to increase the number of messages, or the intensity of the satisfaction they convey. So, I guess I should only tell one person the first month, and then two people the next month. And, I'll need a satisfaction score in order to quantify the satisfaction level. It should probably be an integer, so that it can go negative to indicate dissatisfaction, not that I would tell any of the dissatisfied to send email, they would just do it on their own anyway, and could include a little number. Then he just sum the satisfaction level to get total satisfaction index. With multipliers added to gauge the relative importance of the people that are being satisfied, maybe the multiplier would be their salary divided by 100k.

Hmmm, I don't know how else to do it, but I am afraid if I pose this, I will look like a sarcastic jerk...even though it's just the best I can do sometimes.

Rails for Java Developers- beta book

2006-10-04T13:36:00.000-04:00

I just started reading the early beta of Rails for Java Developers. It's a new book published by the Pragmatic Programmers and written by the Relevance guys- Stu and Justin. Perhaps the most shocking thing is that they have time to write a book. I have been lightly involved in the review process for O'Reilly's upcoming "Rails Cookbook", currently available as a Rough Cut. It's a lot of work to stay current with a moving target and make sure all of the code works.

As far as the book goes, I think it really is going to fill a gap in terms of being a reference for people that know how to do something in Java, and want to know what the equivalent thing is in Ruby on Rails. It's like those options in a new version of a product to see the menus for the prior version, so that you know where to find things. Of course, I haven't tried much in the book yet, so I can't speak to it's quality, but I like the tone and the approach so far. It appears to be just over half way to their 300 page target, so, there's a lot to do before Nov.

Google Enterprise Professional- on earth

2006-10-03T22:43:00.000-04:00

Google's expanding their partner program to include "Google Earth Specialists" Spatial Energy. Noah D. wrote a brief note on the enterprise blog. Our company joined the partner program initially to do things with the search appliance, but maybe the earth appliance is around the corner.

Spatial Energy looks like they have an interesting angle. It's a good match for companies with a lot of experience in imagery and orthorectification to get involved with Google Earth Enterprise. If you want to make a good looking globe, you need some good looking data. Their focus is in the energy market, where perhaps the inability of Google Earth to look underground will be felt more acutely? If anyone in the energy community is in Houston on Oct. 18, they are doing a seminar that is going to cover a few topics, including SketchUp integration.

GE COM API...beta

2006-09-26T17:10:00.000-04:00

Hey, it's finally a beta, and documented. Cool! Still really limited, but if you want to control GE with another program, here you go.

http://earth.google.com/comapi/

The announcement...
http://bbs.keyhole.com/ubb/showflat.php?Number=613531

Updates to Google Earth COM API

2006-09-19T00:13:00.000-04:00

Well, they haven't documented it yet, so I am guessing it's still not stable, but there are additions to the API in the recent versions of Google Earth, and the library has been renamed to EARTHlib, versus KHlib. For starters, it looks like they are renaming the objects to have a GE suffix, versus a KH prefix (e.g., KHFeature to FeatureGE). By keeping the old ones around, they may be doing a quiet kind of deprecation. The new GE objects have more properties to access and support some more methods, but don't really add a massive amount of new capability at first glance.

The ApplicationGE class which is the primary class used to inspect and modify the current state of the running application adds a few features to the old KHInterface class. You can now access the lists of items in the layers controls, the current status of the streaming, login/logout and do a few things with popping up balloons.

FeatureGE adds the ability to highlight, get the feature's name, and get parents and children to the old KHFeature class. A FeatureCollectionGE class has also been added, it's just a strongly typed list.

CameraInfoGE adds the ability to grab the FocusPointAltitude and FocusPointAltitudeMode (absolute or ground relative) to KHViewInfo.

We've seen this interface get a lot of use from things like the cool SketchUp! integration and a lot of the remote control plugins. I am still waiting to be able to use Google Earth to control other applications. I understand their reluctance to open up their beautiful interface to the button overload that will surely result, but I really don't think your average user will be downloading all of the plugins...

getting paid...

2006-09-17T23:59:00.000-04:00

Just read this rather amusing Chicken McNuggets bit [link at end] from HBO's The Wire, and it resonated because I was just signing away part of my intellectual property rights to the work I do...or rather trying not to. Of course, it probably took more than one guy in the basement to come up with the mcnuggets, but it's highly unlikely any of them getting paid for it. It wouldn't take much of a percentage, per nugget, to be rather well off.

Of course, there may be some associated guilt with creating an unhealthy food, but they do save a lot of time. I mean, all of that time saved not having to pull all of the skin, bone and fat off of the chicken could approximately be equal to the time shaved off of your lifespan by consuming the partially hydrogenated blended version of the skin, bone, and fat. (There may be some meat in there as well, it's hard to know.)

Anyway, don't be the McNugget guy. If you don't care about getting paid for your ideas, you might want to at least MIT-license them, so it's harder for other people to get rich off of them without adding their own special sauce.

Warning: Link from NIVI Blog contains offensive language. I am pretty sure this isn't really family blog, although Google Analytics doesn't tell me how old you are, yet, but if your every click at work is analyzed and the stream is then fed into the old raise-o-meter at the end of the year, it's probably good to keep the premium cable content to minimum.

GE timeline...

2006-09-15T10:30:00.000-04:00

Yes, this update is very cool. Three lines of code added to my little Ruby application with XML builder to support it:

xml.TimeStamp do
xml.when(event_date.strftime('%Y-%m-%d'))
end

gets added to every xml.Placemark do

makes:

<TimeStamp>
<when>2006-08-28</when>
</TimeStamp>

Which gives you a pretty floating toolbar...
that animates your map view. This was in the old enterprise client, but a little buggy and nowhere near as pretty. In a word...wow!

Piling on old Joel

2006-09-13T00:15:00.000-04:00

Joel Spolsky, famous for the Joel on Software blog, is a funny old man. He's been doing a lot of hating on Ruby and Java for a while now. His latest complaint about Ruby is idiotic...it comes down to the fact that it's not fast enough for big calculations. I guess we are all missing something- but what big calculations is his bug tracking software doing? Does it fix the bugz in your code?

Most Rubyist use c extensions in those cases. I used to drop out of Avenue to call signal propagation algorithms written in c++ rather frequently. Even in Java, we used JNI calls in various places for image processing code. And then the c and c++ programmers drop into assembly for the truly ludicrous optimization requirements. And then we buy an FPGA and do it in hardware. Has anyone tried orthorectification with this stuff from Cardio Logic?

In his quest to have a flexible high performance language, Joel is actually using a custom language based on VBScript (that compiles to...VBScript) for his little web app. To ease deployment. And it's called Wasabi.

Dion's satire is amusing but I have to point all to Jeff Atwood's excellent deconstruction over on Coding Horror. A snippet:

"For instance, two weeks ago we found out that Joel's company wrote their flagship product, FogBugz, in a proprietary language they created themselves.

FogBugz is written in Wasabi, a very advanced, functional-programming dialect of Basic with closures and lambdas and Rails-like active records that can be compiled down to VBScript, JavaScript, PHP4 or PHP5. Wasabi is a private, in-house language written by one of our best developers that is optimized specifically for developing FogBugz; the Wasabi compiler itself is written in C#.

You couldn't possibly have heard it, but that was the sound of fifty thousand programmers' heads simultaneously exploding.

Writing your own language is absolutely beyond the pale. It's a toxic decision that is so completely at odds with Joel's previous excellent and sane advice on software development that people literally thought he was joking. He had to write an entire follow-up post to explain that, no, he wasn't kidding. "

So, should we all continue to pile on poor old Joel? I have been looking at his stuff with a much more sceptical eye this year, but I am thinking about not looking at all anymore...sigh.

Use migrations outside of Rails

2006-09-11T23:36:00.000-04:00

If you haven't read Scott Ambler's Agile Database Techniques, I recommend it. I didn't realize how spoiled I was to work on projects where the database could be create by scripts, until I worked on projects where an evil DBA controlled all access to everything. It was some small comfort to see "all access to database through stored procedures" as one of the problems to be handled with legacy systems in the book. I haven't read Refactoring Databases yet, but it's somewhere on my todo list....

The next step, once you have the database scripted is scripting the changes to the database. The Rails ActiveRecord migrations approach is really cool for this, and there is no reason it should be applied outside of Rails, as Dion points out... Basically it gives a way to step back from any change, in addition to scripting each changeset to the database. Beyond all of that, it gives you a database neutral method for describing the database itself, making migration to another database platform more of a possibility.

Is the biggest danger of having a DBA on an Enterprise Application that they are going to write code that locks you into a technology? No- I think the bigger danger is that they are non agile and that they won't know how to deal with changes to the database that occur at the rate agile projects demand.

Spam 2.0

2006-09-11T22:50:00.000-04:00

Spammers are apparently now creating spam tickets in open online bug tracking systems, and then linking to those spam tickets other places. Justin's bug tracking system is being hit by this...it really sucks. Everything Web 2.0 style, with users adding content to your site, is going to need a Captcha system, in addition to all of the cross site scripting and other security measures.

The whole Captcha thing, it is an inverse Turing Test- one where people are trying to prove they are people to programs (versus programs trying to prove they are people to people). It's getting pretty hard to do- machines are getting better at reading those fuzzy images. I don't know when they'll be able to solve the hot or not one though. I have trouble with those... Even crazier is that people are solving Captcha for spammers to get access to porn...crowd sourcing. I wonder if people are using Amazon's Mechanical Turk for this?

I am saddened to be getting more traditional spam slipping through my GMail filter. First it was complete nonsense, now it's idiotic pump and dump stock schemes and the usual prescription drug and sex stuff. It's making email far less relevant to youngsters that prefer chat- it's harder to spam, for now. Pervasive advertising...bleeaaurrgh.

Enjoy Baking

2006-08-23T00:21:00.000-04:00

I love this quote from Ian MacKaye, as quoted by Salon.com, as quoted by the 37 signals guys in Getting Real:

The bakery
American business at this point is really about developing an idea, making it profitable, selling it while it’s profitable and then getting out or diversifying. It’s just about sucking everything up. My idea was: Enjoy baking, sell your bread, people like it, sell more. Keep the bakery going because you’re making good food and people are happy.
–Ian MacKaye, member of Fugazi and co-owner of Dischord Records
( from Salon.com People | Ian MacKaye)

I enjoy baking (software). I think if you are running a services company- you basically have to like it. Now that we've built up our little business to a fair number of people on staff, why would we sell it so that everyone could go back to working for a big company that didn't appreciate them? I'd rather buy up someone else's business.

Cookbooks...

2006-08-11T00:31:00.000-04:00

There are a few results on google for "GIS cookbook"- not much of substance.
A rundown of a couple...

I did find this CSISS cookbook: It's got separate recipes for ArcView 3.x, 8.x and 9.x, and not really a cookbook, as noted by a slashgeo commenter. However, it's about the best there is and widely referenced.

There's an Arc/INFO one:=Slightly dated and very small. But still good

There's a Swedish/Phillipine one being put together for Local Govt Units...can't find the actual book, but it sounds like something useful.

There was ESRI UC paper in 2003...

I have some ideas about what might be good to have in such a cookbook- having enjoyed a few selections from that O'Reilly series, including the recent Ruby one. Maybe a collaborative web site would be a good start. Several promising places to stick such a thing.