Tips for using Rails migrations

 The concept of a migration is that each change that is made to the structure of the database is captured in a version controlled script.  Any time you want to create a table, add a field, change a field type, etc., this is done via a migration.  Migrations are generated along with models, views, and controllers when a Rails scaffold is created to manage a resource. In addition, migrations are used to create a test database for each user which allows for rapid, automated quality checks on the application. If you have ever worked on a project where it is difficult to create a development "sandbox" database for each developer, where databases are copied via backup processes that leave weird permissions behind (I'm looking at you SQL Server), or where keeping the development database in sync with the application is something you even have to think about, you will easily see benefits from migrations.

Using Rails migrations can be a bit of change for teams. In some cases, developers are not very familiar with relational database concepts and object modeling principles and can create some overly complex and/or inefficient structures. Having the overall model reviewed by a database modeling expert and an object modeling expert can be quite helpful.  I recommend Coad's "Java Modeling in Color with UML", Fowler's "Analysis Patterns" and David Hay's "Data Model Patterns"  for getting yourself up to speed with this. However, it is also worth communicating with the rest of the team that may be touching the model. 

Here are a few things we've done in the process of keeping migrations under control:

Don't edit an existing migration, create a new one.
     Editing an existing migration defeats the whole point.  You might be able to get away with it if no one has run it.  One obvious exception to this is if someone creates a migration that doesn't run and checked in. They must get the pig.


Update and run new migrations before you check in a migration.
      While this might seem obvious, in many cases, it is easy to check in a new migration file to the repository without seeing if someone else has created one that does something that conflicts with yours, since the names will be different.  If you don't run the migration, you risk doing something with your migration that causes issues. If you have already run your own migration locally, but not checked it in, and you get an update of a new migration, you need to roll back your own migration and change the numbering on it so that it runs after the migrations you just received from your version control system.

Check in the models that go with the migration when you check in the migration
     One key benefit of migrations is keeping the code changes in sync with the database changes.  Add new tables and models in a single, logical commit.


Assign models to class owners
      The feature of XP that I have been least successful with is the concept of common code ownership.  It is one of the easiest practices to apply in theory, but without the sense of collective responsibility and the compensating practices of pair programming and continuous integration, it can be problematic.  In larger teams, it often makes more sense to have people or subteams assigned to manage a particular model so that changes can be coordinated.  In theory, there can be enough communication to make it work. In reality, many teams are full of introverts that would rather rewrite huge swaths of code (and tests) than ask a question.

Communicate
     Changes to the object model can impact the whole application. These changes should be communicated to the rest of the team so that the reasoning behind them can be better understood and the data can be used correctly.  There is a risk of someone criticizing or trying to change your proposal, but this is ultimately a lower risk than building something that others fail to understand.

Roll up all migrations into a big file after a major release
    Once you've been going in your project for a while and get around the 100-150 migration point, it is probably worth it to roll the migrations prior to the current point into a single migration.

Generate SQL to run on the production database
   In many cases it is preferable to generate SQL to run on the production database, as opposed to running the migrate task directly against the database. This also allows for some more extensive code review on structural changes.

Don't forget to create constraints and indexes (indices?)
  If you are using foreign keys in your database, it's often worth the cost to create a constraint on the column to prevent any data integrity issues.  It's also a good practice to create an index on those columns to speed joins.

This is really just the tip of the iceberg, but it's a yummy water ice iceberg. Learning to use database migrations is a key skill, not just for Rails development, but for any agile development. A project which cannot create its database from scripts accessible to every developer is missing something important. By breaking those scripts down into small chunks and coordinating them with changes to the code, the situation where the application and database are out of sync simply does not exist.

Whose Success?

There is sometimes a quandary between whose success we seek.

Acting as a consultant, you obviously want your own work to be considered successful, so that you will induce others to hire you. Getting there isn't so easy though. We often look downstream to the customer's customers. By helping meet the customer's customers' needs, we hope to make our immediate customers successful. The catch comes when something the immediate customer is doing is preventing them from satisfying their customer- and they don't want to change it. If we make the customer's customer successful at the cost of making the customer unsuccessful, we're probably not going to be successful.

Remember the secrets of consulting- you have to help people learn to solve their own problems, you can't always solve their problems for them. You can be satisfied with your accomplishments, regardless of whether they give you credit. The ideal form of influence is to help the customer see the problem more clearly, but let them decide. Finally, if you want to understand a dysfunctional system, follow the money!

Seed data in Rails 2.3.4

Very nice addition to Rails as part of the security updates in 2.3.4- an easy way to load seed data.

Added db/seeds.rb as a default file for storing seed data for the database. Can 
be loaded with rake db:seed (or created alongside the db with db:setup). (This 
is also known as the "Stop Putting Gawd Damn Seed Data In Your Migrations" 
feature)

I like this on several accounts, not least of which is it invalidates my previous post on the subject.

In the meantime, there are several plugin based approaches to making this work, for example seed-fu.

Default route...

One of the things that has been bothering me in Rails since everything switched to resources is how we lost the easy way any public method in the controller became an accessible URL without having to write a rule for it. For example, in a person controller, I could define a search method like this:

def search     
results = Person.seach(params[:q])
end

And it would be accessible, just like that. http://myapp/people/search?q=matt
I even wrote a simple router in Java to do the same thing, because it just makes it so easy to navigate the application code.

The poor design of map.resources messes this up. By adding the route pattern :controller/:id it starts with the default assumption that the search?q=matt part of my URL is an ID. Is it so hard to put an action name in there? In any case,

map.connect ':controller/:action'

goes above all of the map.resources calls in my app. You might want to put it in your application too.

Rails Fixtures and Seed Data

One of the things that makes Ruby on Rails a great web framework is the quantity of tools available that are focused around the task of making web applications easy to build and encourage good software engineering practices.

Rails makes it really easy to test by building in a testing framework. One aspect of this is what are called "fixtures", data described in CSV or YML formats that is loaded into the database for each round of testing.  Tests live at both the unit (model) and integration (controller) level. End to end tests that involve the view are traditionally run from an automated tool, such as Selenium or HPQC (ha).

UPDATE ALERT... Rails 2.3.4 has a nice, new way to do this.


Fixtures also make it easy to load seed data into the database.  You can do this with migrations, but it gets messy, especially if you use ActiveRecord to do it, and then you delete a model class sometime in the future, it forces you to do an edit of an old migration, which means you might as well roll them all up. I prefer:

rake db:fixtures:load

Which runs:

namespace :fixtures do
    desc "Load fixtures into the current environment's database.  Load specific fixtures using FIXTURES=x,y"
    task :load => :environment do
      require 'active_record/fixtures'
      ActiveRecord::Base.establish_connection(RAILS_ENV.to_sym)
      (ENV['FIXTURES'] ? ENV['FIXTURES'].split(/,/) : Dir.glob(File.join(RAILS_ROOT, 'test', 'fixtures', '*.{yml,csv}'))).each do |fixture_file|
        Fixtures.create_fixtures('test/fixtures', File.basename(fixture_file, '.*'))
      end
    end
  end

This will load all of your test data into the development database (or production, if that's what your RAILS_ENV is set to).  Jeffrey Allan Hardy documented this pretty well, and added a separate rake task that will do this from a separate directory from your test fixtures, which is a great idea. It goes like:

namespace :db do
  desc "Load seed fixtures (from db/fixtures) into the current environment's database." 
  task :seed => :environment do
    require 'active_record/fixtures'
    Dir.glob(RAILS_ROOT + '/db/fixtures/*.yml').each do |file|
      Fixtures.create_fixtures('db/fixtures', File.basename(file, '.*'))
    end
  end
end

It's pretty easy and keeps this stuff out of the migrations.

Thank You for Agreeing

I've been reading "Thank You for Arguing" tonight, thanks to a tip from the excellent time sink known as Hacker News .  The first page of the introduction has a pretty telling tip.

"Try this in a meeting: Answer someone who expresses doubt over idea with, 'Okay, let's tweak it.' Now focus the argument on revising your idea as if the group had already accepted it. This move is a form of concession- rhetorical jujitsu that uses your opponent's move to your advantage."

Heinrichs gets to some much deeper points as well. One key thing to focus on is getting the outcome you want from the discussion, and not on scoring points to "win" the argument.

I wanted to draw a tenuous connection here, between these rhetorical skills, to which Heinrichs gives a fabulous introduction, and the listening, or "receiving" skills that the Manager Tools guys have focused on .  They put communication skills at the foundation of developing interpersonal skills, and they put listening skills at the foundation of communication.  In their characteristic way, Horstmann and Auzenne give some concrete examples of specific, measurable behaviors that we can use to evaluate and improve our communications, by strengthening the feeling in the person we are communicating with that we understand them.

One of the items on their list of things to measure is using "Thank You" to begin a reply.  (I said it was a tenuous connection.) Other things they recommend include "No interruptions", "Agreeing without qualification", "No buts or howevers", and my personal favorite, "smiling". By making simple gestures like this, the person that you are communicating with is shown that you respect them as a person.  If we cut someone off, we are going to have a much harder time persuading them of anything, because they are likely to think that we don't understand their argument and they are also probably not going to like us very much.

What makes this even more challenging, is that in many situations, the person that you are doing a poor job of listening to is not going to know exactly what it is you are doing that they don't like, but they are just going to think you are disagreeable or contrary.  You may end up getting negative feedback from managers that doesn't make sense- vague comments on how you are perceived.  This is where I like to tie in things like the Manager Tools material to figure out what good feedback for those situations would look like. Respect for people is a key aspect of successful systems, such as the Toyota Way.  However, it is also key in being persuasive. To really change someone's mind to what we think is an amazing thing. It means teaming up and getting on the same side.

"Thank You for Arguing" is filled with little gems that you can use to show respect to others, by offering them a choice, instead of negating what they say directly. Simple phrases like, "On the other hand, it could be that..." create an option for the listener, instead of putting them in a box. It offers them the opportunity to admit that they aren't 100% sure about their view.  Moving the argument into the future tense also affords the chance to remove some of the contentiousness about any concerns, while defining the issue in a way that makes your point clear.

Rather than despise rhetoric for muddying the waters of rational debate, we should use it as a tool to achieve our goals, as long as we can keep it in a framework of respect for others- by combining it with the listening/receiving skills that are the foundation of communication.

Other good books on similar subjects are "Influence" and "Yes!" by Cialdini, "Difficult Conversations" and "Getting to Yes" out of the Harvard Negotiation Project that was led by Fisher the last I looked into it, "Never Eat Alone" by Ferazzi, and the classic "How to Win Friends and Influence People" by Dale Carnegie. My wife read the last on her Kindle recently, and the chapter on married couples is well worth it.

Passionate Programmer

Chad Fowler's The Passionate Programmer is a rewrite of his earlier book on how to improve your career in software development, not a guide to finding love using Ruby scripts to mine craigslist. I read it anyway.

Today I'd like to take a quick look at #21, "The Daily Hit". The concept is to, each day, "have some kind of outstanding accomplishment to report to [your] manager, some idea [you] had thought of or implemented that would make [your] department better."

There are a couple of interesting aspects to this. First, is it forces you to break work down into small enough chunks that you can get something done in a day.  The second is that improvement part. Thinking of, and implementing, small improvements every day is closely related to a Lean concept called Kaizen. The third is that reporting and tracking these things really helps demonstrate your value to the organization.  While it seems annoying, and perhaps a bit like boasting, to have to market yourself to your boss or customer or whomever, it actually creates a more positive relationship, because it forces communication about positive subjects.  Having at least one of these ready for a meeting like a Daily Scrum is a good idea- it can even set the tone of accomplishment.

Keeping it positive is really important. A ratio of nine positive to one negative feedback item is what the Manager Tools guys recommend. A key aspect of making Lean work is that it is not about blame. It is about admitting that things are never perfect, or even if they are, the context will change and make them imperfect.  We face a lot of people challenges in the world of work. There are constant machinations of status and power, siphoning off of high profile work and dumping of boring work.  We can't let this prevent us from doing good things. We need to make the assumption that there are more good people in the environment than bad, and building up a network of allies by providing them with support when needed, while also marketing the fact that we are providing that support, will make sure that there is someone out there who has our back. Help people get better.

So, get your daily hit. Communicate what your up to. Do it again tomorrow...

Iteration Zero

Iteration Zero is the term used when a team that is attempting to do incremental development does not plan on developing any working software in their first increment.  Notice the distinction here between iteration and increment- iterative development implies delivering the same thing over and over again to get it closer to being right (as opposed to "right the first time"), incremental development is delivering something in pieces (as opposed to "all at once").  When you get involved with the beast called "Agile", there are a lot more people in favor of incremental development than iterative development. So, in a lot of cases we are talking about Increment Zero, rather than Iteration Zero.

That whole "not delivering any working software" thing makes Iteration Zero seem inherently non-Agile. On the other hand, if you have team without computers or similar issues, it's hard to sign up for any business value. So what do you do in Iteration Zero? Peter Schuh seems to view it as a project inception phase, but then other people want to set out additional time before Iteration Zero to do all of that work.  The Energized Work people seem to do tons of things in their Interation Zero.  Earl Beede sees it as an admission of the failure of the original Agile principles.  I would say it is more along the lines of giving in to no progress.

The real deal is that you have to set low goals for your first increment- plan on a zero velocity until you get something done. A co-worker of mine once made the remark that our organization at the time was "incapable of releasing a Hello World HTML page to the production environment."  I think a customized "Hello, World" is a good increment one goal- put up an HTML user interface sketch. You might get some valuable feedback, such as, "we didn't want a web app".  A lot of the prep work I see really doesn't take advantage of Real Options theory or You Aren't Gonna Need It. On the other hand, you do need to get started on something, you might just have to accept that not much will get done.

Iteration Zero seems to be a slippery slope. Particularly if you let it take months, at that point you are sliding down the slope really fast.  I haven't really seen it used effectively, but that doesn't mean it can't be. I am always looking for data to change my mind.

To sum it up, my current recommendation is don't start with Iteration Zero, start with Velocity Zero.

Google Voice

I just upgraded from Grand Central to Google Voice.  I have been waiting for this transcription feature forever- I'd been using PhoneTag, but it wasn't quite the same. I don't want anyone to call my cell directly again!  I have a feeling this is going to be an extremely popular service- it really puts the receiver back in charge of voicemail.  Often I'd feel my time was being hijacked by people leaving inefficient messages. Text wins.

Data Integration, Propagate Deletes

Enterprise Data Integration is challenging subject. Many of my customers have problems of the sort where they have pieces of information that are being collected by disparate systems. I try to push things towards simplicity, but there are many angles by which complexity sneaks in.

Think about something like Google Reader. Publish/subscribe works simply and well for this sort of content aggregation. However, the behavior of the client is not guaranteed. For example, what happens if I want to delete an entry? Or update an entry? Google Reader simply doesn't care if the publisher wants to delete an entry. It's already copied it into your data. Having an entry go missing from a data feed doesn't necessarily indicate an intention that a downstream deletion should occur. 

Can a simple approach work for data integration? Perhaps, but there are a lot of tricky issues to work around. 

Highlights

Every time I find code that is already written for me, and written better than I would have done it. I remember why I love Ruby on Rails.


<%= highlight(@trees, "tree") %>
I think that I shall never see
A poem lovely as a tree.
A tree whose hungry mouth is prest
Against the sweet earth’s flowing breast;

The new edition, out March 15th, 2009.
Agile Web Development with Rails, Third Edition

Keeping an eye on what I am doing

I like David Seah's idea of a "Task Order Up". My current implementation is pushing that together with the hipster PDA (binder clip and 3x5 cards), using sticky 3x5 notes to put my current objective/task on my monitor, so I can get back on track when I inevitably get off of it. It has a yak shaving aspect as well, where if I hit a blocker like "have to update version of solr" before I can accomplish the next task, I push the update solr task on top of the task (or stack of tasks) it is blocking. Then, when I finish the solr task, I pop it off, and there was thing it was blocking underneath.(LIFO approach)  What makes it better than a computery way of doing this is that the index card is always visible, not buried behind other windows. I guess it would be good as a sidebar app, if I could give up that screen real estate.

Seems to be working so far.

Here Comes Somebody

I finally read Clay Shirky's Here Comes Everybody: The Power of Organizing Without Organizations. It's a good book, not a great book. I think the problem with reading a book like that 11 months after it was released is that most of the ideas have already been absorbed into my consciousness. It's a much better book for someone without my vast wealth of experience using social tools. What Shirky does is establish a framework for understanding how the various tools fit together to produce a network of communication that changes the dynamics of who can be a journalist, or a software developer.

Shirky also gets deep into how this changes the dynamics of organizations.  There is a very readable economics/management paper by Coase from 1937 (for which he won the Noble in 1991) called the Nature of the Firm. In there he examines the costs of contracting for services versus hiring, which is really why the firms exist in the first place.  The technologies of the Internet seem to be changing those costs, and people that know how to take advantage of that can build better firms.

I think there is a unique intersection of trends happening right now, right in front of us that makes this a time of opportunity. It is a time when a many of the traditional solutions to problems are too slow, too expensive, and too static. It's a great time to have nothing, because you're not that far behind. It's a great time for open source. It's a great time to start something new. I am ready. Are you? Here we come.