Every time I find code that is already written for me, and written better than I would have done it. I remember why I love Ruby on Rails.
<%= highlight(@trees, "tree") %>
I think that I shall never see
A poem lovely as a tree.
A tree whose hungry mouth is prest
Against the sweet earth’s flowing breast;
The new edition, out March 15th, 2009.
Agile Web Development with Rails, Third Edition
I like David Seah's idea of a "Task Order Up". My current implementation is pushing that together with the hipster PDA (binder clip and 3x5 cards), using sticky 3x5 notes to put my current objective/task on my monitor, so I can get back on track when I inevitably get off of it. It has a yak shaving aspect as well, where if I hit a blocker like "have to update version of solr" before I can accomplish the next task, I push the update solr task on top of the task (or stack of tasks) it is blocking. Then, when I finish the solr task, I pop it off, and there was thing it was blocking underneath.(LIFO approach) What makes it better than a computery way of doing this is that the index card is always visible, not buried behind other windows. I guess it would be good as a sidebar app, if I could give up that screen real estate.
Seems to be working so far.
I finally read Clay Shirky's Here Comes Everybody: The Power of Organizing Without Organizations
Shirky also gets deep into how this changes the dynamics of organizations. There is a very readable economics/management paper by Coase from 1937 (for which he won the Noble in 1991) called the Nature of the Firm. In there he examines the costs of contracting for services versus hiring, which is really why the firms exist in the first place. The technologies of the Internet seem to be changing those costs, and people that know how to take advantage of that can build better firms.
I think there is a unique intersection of trends happening right now, right in front of us that makes this a time of opportunity. It is a time when a many of the traditional solutions to problems are too slow, too expensive, and too static. It's a great time to have nothing, because you're not that far behind. It's a great time for open source. It's a great time to start something new. I am ready. Are you? Here we come.
My good friends let me down today. They wasted my time with their silly problems. I really like Apache and Firefox and Ajax...but they had some obscure issues today that are causing me to break commitments. It took some seriously Google searching to uncover the root cause.
I was trying to make an Ajax request (using Prototype and link_to_remote) from Firefox to a JRuby on Rails application running in Tomcat over an https connection proxied by Apache Web Server. That's a lot of pieces that could go wrong. I was getting page not found in my poor little div. The link was being generated correctly and I could navigate to it.
Everything worked fine on my box. Of course, the problem was on the real server. Which is still in dev mode and hasn't been opened up so that I can connect to it from my desktop. Which means I have to open an X session to the server over ssh to launch a browser. Which doesn't have Firebug. Double check everything else. Have to find Firebug and get into the back end network...time passes...
Firebug said it was a 403.
It turns out Firefox doesn't pass the content-length header on POST requests. It turns out the sysadmin had typed the innocent looking phrase SecFilterEngine On into the Apache config. This causes Apache to reject requests without a content length header on POST requests. So secure I can't connect. I guess Firefox should send that header, but should Apache really demand it? Now I have a hell of a lot of working around this to do. To the casual observer / tester / customer - it's a bug, a defect, something wrong with my code. I should, and I wish I could, have a dev environment that even remotely resembled production. Amazing waste of a day. What's going to happen tomorrow?
Details here...
I was reading this about a nice idea some bloke had about developers fixing bugs in their spare time. Most of the serious developers I know are already working in their spare time- let's not make all defects their problem too.
Many software testers cause great frustration among software developers. One of the biggest issues that arises on agile projects is that testers have a hard time distinguishing between defects and features that have not yet been implemented.
It's always a bit of a challenge to deal with bugs- even outside of agile. What to some people is an obvious bug is a feature that was never requested to someone else.
For example, I was called into a meeting today where a bug that had been discovered during a user demo. The search was not "not finding telephone numbers in documents". The search term entered was 5551212. Some documents contained 555 1212. Some contained 555-1212. Some contained 1(703)555-1212.
They tried 555*1212, still not working. Search is broken. Developer suggests- try "555 1212", magic happens. * matches any character, not word boundaries...
It was an obvious problem...to the developers who understood that searching for "breakup" was not going to match documents containing "break up". With text, it's obvious, but I can certainly see where people might not see the issue with phone numbers.
We'll add normalized versions of phone numbers into the search index and we'll normalize search terms that look like phone numbers or something like that, but...this is not an insignificant effort. (Even though there is some decent code out there to handle it.) There are trade-offs in performance that have to be considered. Ask a tester though- they'll say found a bug; it's their job, and they want to be able to show how good they are at finding them. If it's a bug, we can't even mark our existing search work as complete. If it's a missing feature, we have to allocate it to the next release. Fortunately, a tester didn't find this one, so we don't quite have that problem, but the users do want the feature of being able to match many different formats of phone numbers.
It seems like a simple semantic difference, and it seems like developers are being too sensitive, but it's actually a big deal. Some "bugs" might actually end up costing a huge amount of money and not be worth fixing- particularly if they aren't really bugs. I have seen teams show excessive deference to testers and spend a time equal to the time spent working on the basic features to handle some edge cases that would never really occur and were better handled by error messages than by trying to do something useful. Meanwhile, the project sponsor is seldom asked to decide whether a bug should be fixed- few are even estimated as to the cost.
I say- if you want to fix a bug, you have to pay up. If you are really smart, you do do the five whys and find the root cause, but if it happens to be something that it never occured to anyone to ask for, try not to ask the developers to take care of it on their own time.
Tom White's book on Hadoop is up on Rough Cuts. If you aren't familiar with Apache Hadoop, it's Doug Cutting's effort to go beyond Lucene and build an open source implementation of much of the other Google infrastructure.
There is a two day Hadoop Camp at the upcoming ApacheCon in New Orleans. Learning about Hadoop is a great way to become familiar with some of the innovations that Google has put forward in the last few years- and to see the technology behind Yahoo's big set of nodes. What follows is the beginning of Mr. White's book, I am looking forward to the chapter on HBase.
"Hadoop was created by Doug Cutting, the creator of Lucene, the widely-used text search library. Hadoop has its origins in Nutch, an open source web search engine, itself a part of the Lucene project.
Building a web search engine from scratch was an ambitious goal, for not only is the software required to crawl and index websites complex to write, it is also a challenge to run without a dedicated operations team — there are lots of moving parts. It's expensive too — Cutting and Cafarella estimated a system supporting a 1-billion-page index would cost around half a million dollars in hardware, with a $30,00 monthly running cost. Nevertheless, they believed it was a worthy goal, as it would open up and ultimately democratise search engine algorithms.
Nutch was started in 2002, and a working crawler and search system quickly emerged. However, they realized that their architecture wouldn't scale to the billions of pages on the web. Help was at hand with the publication of "The Google File System" (GFS) [ref] in 2003. This paper described the architecture of Google's distributed filesystem that was being used in production at Google. GFS, or something like it, would solve their storage needs for the very large files generated as a part of the web crawl and indexing process. In particular, GFS would free up time being spent on administrative tasks such as managing storage nodes. In 2004 they set about writing an open source implementation, the Nutch Distributed File System (NDFS) as it became to be known.
NDFS and the MapReduce implementation in Nutch were applicable beyond the realm of search, and in February 2006 they moved out of Nutch to form an independent subproject of Lucene called Hadoop. Yahoo! hired Doug Cutting and, with a dedicated team, provided the resources to turn Hadoop a system that ran at web scale. This was demonstrated in February 2008 when Yahoo! announced that their production search index was being generated by a 10,000 core Hadoop cluster.
Earlier, in January 2008, Hadoop was made a top-level project at Apache, confirming its success and dynamic community."
I'm also now infrequently blogging at Collision of Influences, my company blog, to add to my infrequent posting here on more arcane topics of interest to nearly 43 people! Topics over there are going to be a little different, I just haven't figured out how yet. I'll let it develop as is its wont.
I just noticed the "The KML Handbook: Geographic Visualization for the Web" by Josie Wernecke is out on Safari Rough Cuts. It looks like it has some interesting tips, not really a programming book per se, more of an expanded version of Google's documentation. The subtitle is interesting to me- "for the Web". A couple of years ago it would have just been for Google Earth, but KML is on its way to becoming the new shapefile. Next I want JSON.
