Rails Fixtures and Seed Data

One of the things that makes Ruby on Rails a great web framework is the quantity of tools available that are focused around the task of making web applications easy to build and encourage good software engineering practices.

Rails makes it really easy to test by building in a testing framework. One aspect of this is what are called "fixtures", data described in CSV or YML formats that is loaded into the database for each round of testing.  Tests live at both the unit (model) and integration (controller) level. End to end tests that involve the view are traditionally run from an automated tool, such as Selenium or HPQC (ha).

Fixtures also make it easy to load seed data into the database.  You can do this with migrations, but it gets messy, especially if you use ActiveRecord to do it, and then you delete a model class sometime in the future, it forces you to do an edit of an old migration, which means you might as well roll them all up. I prefer:

rake db:fixtures:load

Which runs:

namespace :fixtures do
    desc "Load fixtures into the current environment's database.  Load specific fixtures using FIXTURES=x,y"
    task :load => :environment do
      require 'active_record/fixtures'
      ActiveRecord::Base.establish_connection(RAILS_ENV.to_sym)
      (ENV['FIXTURES'] ? ENV['FIXTURES'].split(/,/) : Dir.glob(File.join(RAILS_ROOT, 'test', 'fixtures', '*.{yml,csv}'))).each do |fixture_file|
        Fixtures.create_fixtures('test/fixtures', File.basename(fixture_file, '.*'))
      end
    end
  end

This will load all of your test data into the development database (or production, if that's what your RAILS_ENV is set to).  Jeffrey Allan Hardy documented this pretty well, and added a separate rake task that will do this from a separate directory from your test fixtures, which is a great idea. It goes like:

namespace :db do
  desc "Load seed fixtures (from db/fixtures) into the current environment's database." 
  task :seed => :environment do
    require 'active_record/fixtures'
    Dir.glob(RAILS_ROOT + '/db/fixtures/*.yml').each do |file|
      Fixtures.create_fixtures('db/fixtures', File.basename(file, '.*'))
    end
  end
end

It's pretty easy and keeps this stuff out of the migrations.

Thank You for Agreeing

I've been reading "Thank You for Arguing" tonight, thanks to a tip from the excellent time sink known as Hacker News .  The first page of the introduction has a pretty telling tip.

"Try this in a meeting: Answer someone who expresses doubt over idea with, 'Okay, let's tweak it.' Now focus the argument on revising your idea as if the group had already accepted it. This move is a form of concession- rhetorical jujitsu that uses your opponent's move to your advantage."

Heinrichs gets to some much deeper points as well. One key thing to focus on is getting the outcome you want from the discussion, and not on scoring points to "win" the argument.

I wanted to draw a tenuous connection here, between these rhetorical skills, to which Heinrichs gives a fabulous introduction, and the listening, or "receiving" skills that the Manager Tools guys have focused on .  They put communication skills at the foundation of developing interpersonal skills, and they put listening skills at the foundation of communication.  In their characteristic way, Horstmann and Auzenne give some concrete examples of specific, measurable behaviors that we can use to evaluate and improve our communications, by strengthening the feeling in the person we are communicating with that we understand them.

One of the items on their list of things to measure is using "Thank You" to begin a reply.  (I said it was a tenuous connection.) Other things they recommend include "No interruptions", "Agreeing without qualification", "No buts or howevers", and my personal favorite, "smiling". By making simple gestures like this, the person that you are communicating with is shown that you respect them as a person.  If we cut someone off, we are going to have a much harder time persuading them of anything, because they are likely to think that we don't understand their argument and they are also probably not going to like us very much.

What makes this even more challenging, is that in many situations, the person that you are doing a poor job of listening to is not going to know exactly what it is you are doing that they don't like, but they are just going to think you are disagreeable or contrary.  You may end up getting negative feedback from managers that doesn't make sense- vague comments on how you are perceived.  This is where I like to tie in things like the Manager Tools material to figure out what good feedback for those situations would look like. Respect for people is a key aspect of successful systems, such as the Toyota Way.  However, it is also key in being persuasive. To really change someone's mind to what we think is an amazing thing. It means teaming up and getting on the same side.

"Thank You for Arguing" is filled with little gems that you can use to show respect to others, by offering them a choice, instead of negating what they say directly. Simple phrases like, "On the other hand, it could be that..." create an option for the listener, instead of putting them in a box. It offers them the opportunity to admit that they aren't 100% sure about their view.  Moving the argument into the future tense also affords the chance to remove some of the contentiousness about any concerns, while defining the issue in a way that makes your point clear.

Rather than despise rhetoric for muddying the waters of rational debate, we should use it as a tool to achieve our goals, as long as we can keep it in a framework of respect for others- by combining it with the listening/receiving skills that are the foundation of communication.

Other good books on similar subjects are "Influence" and "Yes!" by Cialdini, "Difficult Conversations" and "Getting to Yes" out of the Harvard Negotiation Project that was led by Fisher the last I looked into it, "Never Eat Alone" by Ferazzi, and the classic "How to Win Friends and Influence People" by Dale Carnegie. My wife read the last on her Kindle recently, and the chapter on married couples is well worth it.

Passionate Programmer

Chad Fowler's The Passionate Programmer is a rewrite of his earlier book on how to improve your career in software development, not a guide to finding love using Ruby scripts to mine craigslist. I read it anyway.

Today I'd like to take a quick look at #21, "The Daily Hit". The concept is to, each day, "have some kind of outstanding accomplishment to report to [your] manager, some idea [you] had thought of or implemented that would make [your] department better."

There are a couple of interesting aspects to this. First, is it forces you to break work down into small enough chunks that you can get something done in a day.  The second is that improvement part. Thinking of, and implementing, small improvements every day is closely related to a Lean concept called Kaizen. The third is that reporting and tracking these things really helps demonstrate your value to the organization.  While it seems annoying, and perhaps a bit like boasting, to have to market yourself to your boss or customer or whomever, it actually creates a more positive relationship, because it forces communication about positive subjects.  Having at least one of these ready for a meeting like a Daily Scrum is a good idea- it can even set the tone of accomplishment.

Keeping it positive is really important. A ratio of nine positive to one negative feedback item is what the Manager Tools guys recommend. A key aspect of making Lean work is that it is not about blame. It is about admitting that things are never perfect, or even if they are, the context will change and make them imperfect.  We face a lot of people challenges in the world of work. There are constant machinations of status and power, siphoning off of high profile work and dumping of boring work.  We can't let this prevent us from doing good things. We need to make the assumption that there are more good people in the environment than bad, and building up a network of allies by providing them with support when needed, while also marketing the fact that we are providing that support, will make sure that there is someone out there who has our back. Help people get better.

So, get your daily hit. Communicate what your up to. Do it again tomorrow...

Iteration Zero

Iteration Zero is the term used when a team that is attempting to do incremental development does not plan on developing any working software in their first increment.  Notice the distinction here between iteration and increment- iterative development implies delivering the same thing over and over again to get it closer to being right (as opposed to "right the first time"), incremental development is delivering something in pieces (as opposed to "all at once").  When you get involved with the beast called "Agile", there are a lot more people in favor of incremental development than iterative development. So, in a lot of cases we are talking about Increment Zero, rather than Iteration Zero.

That whole "not delivering any working software" thing makes Iteration Zero seem inherently non-Agile. On the other hand, if you have team without computers or similar issues, it's hard to sign up for any business value. So what do you do in Iteration Zero? Peter Schuh seems to view it as a project inception phase, but then other people want to set out additional time before Iteration Zero to do all of that work.  The Energized Work people seem to do tons of things in their Interation Zero.  Earl Beede sees it as an admission of the failure of the original Agile principles.  I would say it is more along the lines of giving in to no progress.

The real deal is that you have to set low goals for your first increment- plan on a zero velocity until you get something done. A co-worker of mine once made the remark that our organization at the time was "incapable of releasing a Hello World HTML page to the production environment."  I think a customized "Hello, World" is a good increment one goal- put up an HTML user interface sketch. You might get some valuable feedback, such as, "we didn't want a web app".  A lot of the prep work I see really doesn't take advantage of Real Options theory or You Aren't Gonna Need It. On the other hand, you do need to get started on something, you might just have to accept that not much will get done.

Iteration Zero seems to be a slippery slope. Particularly if you let it take months, at that point you are sliding down the slope really fast.  I haven't really seen it used effectively, but that doesn't mean it can't be. I am always looking for data to change my mind.

To sum it up, my current recommendation is don't start with Iteration Zero, start with Velocity Zero.

Google Voice

I just upgraded from Grand Central to Google Voice.  I have been waiting for this transcription feature forever- I'd been using PhoneTag, but it wasn't quite the same. I don't want anyone to call my cell directly again!  I have a feeling this is going to be an extremely popular service- it really puts the receiver back in charge of voicemail.  Often I'd feel my time was being hijacked by people leaving inefficient messages. Text wins.

Data Integration, Propagate Deletes

Enterprise Data Integration is challenging subject. Many of my customers have problems of the sort where they have pieces of information that are being collected by disparate systems. I try to push things towards simplicity, but there are many angles by which complexity sneaks in.

Think about something like Google Reader. Publish/subscribe works simply and well for this sort of content aggregation. However, the behavior of the client is not guaranteed. For example, what happens if I want to delete an entry? Or update an entry? Google Reader simply doesn't care if the publisher wants to delete an entry. It's already copied it into your data. Having an entry go missing from a data feed doesn't necessarily indicate an intention that a downstream deletion should occur. 

Can a simple approach work for data integration? Perhaps, but there are a lot of tricky issues to work around. 

Highlights

Every time I find code that is already written for me, and written better than I would have done it. I remember why I love Ruby on Rails.


<%= highlight(@trees, "tree") %>
I think that I shall never see
A poem lovely as a tree.
A tree whose hungry mouth is prest
Against the sweet earth’s flowing breast;

The new edition, out March 15th, 2009.
Agile Web Development with Rails, Third Edition

Keeping an eye on what I am doing

I like David Seah's idea of a "Task Order Up". My current implementation is pushing that together with the hipster PDA (binder clip and 3x5 cards), using sticky 3x5 notes to put my current objective/task on my monitor, so I can get back on track when I inevitably get off of it. It has a yak shaving aspect as well, where if I hit a blocker like "have to update version of solr" before I can accomplish the next task, I push the update solr task on top of the task (or stack of tasks) it is blocking. Then, when I finish the solr task, I pop it off, and there was thing it was blocking underneath.(LIFO approach)  What makes it better than a computery way of doing this is that the index card is always visible, not buried behind other windows. I guess it would be good as a sidebar app, if I could give up that screen real estate.

Seems to be working so far.

Here Comes Somebody

I finally read Clay Shirky's Here Comes Everybody: The Power of Organizing Without Organizations. It's a good book, not a great book. I think the problem with reading a book like that 11 months after it was released is that most of the ideas have already been absorbed into my consciousness. It's a much better book for someone without my vast wealth of experience using social tools. What Shirky does is establish a framework for understanding how the various tools fit together to produce a network of communication that changes the dynamics of who can be a journalist, or a software developer.

Shirky also gets deep into how this changes the dynamics of organizations.  There is a very readable economics/management paper by Coase from 1937 (for which he won the Noble in 1991) called the Nature of the Firm. In there he examines the costs of contracting for services versus hiring, which is really why the firms exist in the first place.  The technologies of the Internet seem to be changing those costs, and people that know how to take advantage of that can build better firms.

I think there is a unique intersection of trends happening right now, right in front of us that makes this a time of opportunity. It is a time when a many of the traditional solutions to problems are too slow, too expensive, and too static. It's a great time to have nothing, because you're not that far behind. It's a great time for open source. It's a great time to start something new. I am ready. Are you? Here we come.

I hate it when my friends let me down

My good friends let me down today. They wasted my time with their silly problems. I really like Apache and Firefox and Ajax...but they had some obscure issues today that are causing me to break commitments. It took some seriously Google searching to uncover the root cause.

I was trying to make an Ajax request (using Prototype and link_to_remote) from Firefox to a JRuby on Rails application running in Tomcat over an https connection proxied by Apache Web Server. That's a lot of pieces that could go wrong. I was getting page not found in my poor little div. The link was being generated correctly and I could navigate to it.

Everything worked fine on my box. Of course, the problem was on the real server. Which is still in dev mode and hasn't been opened up so that I can connect to it from my desktop. Which means I have to open an X session to the server over ssh to launch a browser. Which doesn't have Firebug. Double check everything else. Have to find Firebug and get into the back end network...time passes...

Firebug said it was a 403.

It turns out Firefox doesn't pass the content-length header on POST requests. It turns out the sysadmin had typed the innocent looking phrase SecFilterEngine On into the Apache config. This causes Apache to reject requests without a content length header on POST requests. So secure I can't connect. I guess Firefox should send that header, but should Apache really demand it? Now I have a hell of a lot of working around this to do. To the casual observer / tester / customer - it's a bug, a defect, something wrong with my code. I should, and I wish I could, have a dev environment that even remotely resembled production. Amazing waste of a day. What's going to happen tomorrow?

Details here...