Wednesday, December 04, 2019


Thinking about decisions

Check out this product from Osparna that supports investment diligence and decision-making. The idea is that we should take a systematic approach to decision making around investments. Rather than only base them on our intuition, we decide what the basis for a good decision is, and then gather the information. There's a lot more to it...

Thursday, August 29, 2019

Black box

Unpopular quote from my image and video processing professor - “The only problem with machine learning is that the machine does the learning and you don’t.”

While I understand that is missing a lot of nuance, it has stuck with me over the past few years as I feel like I am missing out on the cool machine learning work going on out there.
There is a ton of learning about calculus, probability, and statistics when doing machine learning, but I can’t shake the fact that at the end of the day, the output is basically a black box. As you start toying with AI you realize that the only way to learn from your architecture and results is by tuning parameters and trial and error.
Of course there are many applications that only AI can solve, which is all good and well, but I’m curious to hear from some heavy machine learning practitioners - what is exciting to you about your work?
This is a serious inquiry because I want to know if it’s worth exploring again. In the past university AI classes I took, I just got bored writing tiny programs that leveraged AI libraries to classify images, do some simple predictions etc.

The complaint that a neural network, or deep learning model, is a "black box", where you can't see why the model is making a particular output seems quite weak to me. Not being interested in it because of that is worse, because it is like saying that we shouldn't study neuroscience because we don't understand how memories are stored in the nervous system.  The box is not black. It is just a very complicated box, with lots of weights and layers. You can look at the weights and inputs and layers and literally see exactly what simple math applied to the inputs led to the output. It might take you a while, but it is all there. We are building tools all of the time to make it easier to make this process more efficient, but, unlike a brain, no one is stopping you from reaching into the box and taking a look.

Friday, October 27, 2017


I've been working hard to keep up to date with recent progress in machine learning.  The progress in Deep Learning is exciting to me, especially because it builds on all of the things I learned about neural networks in 1995-99 in college in grad school (particularly some work with Dr. William Levy at UVA), but never really had the opportunity to put into practice directly, except in some simple classifier type situations.

In any case, last year I took a short course from Miner and Kasch taught by Florian Muellerklein that was quite good. You can check out the slides and code on github.  I'd previously been messing around with Torch, Caffe, TensorFlow, cuDNN and a variety of other libraries, while easier than the old days of finding eigenvalues in c++ or running out of MatLab, they required a lot of configuration and such.  In the course, we jumped right into using keras. Wow, so much easier. It's a bit like a Ruby on Rails for neural networks, giving you some sensible defaults and get going right away, minimizing common errors, but different in that it is just a simpler interface overlaying other libraries.  If anyone is diving in this stuff, the keras path is the best path that I've tried.

To help one out on this path, there are now a couple of books on keras. Deep Learning with Python is the one that I recommend. It is written by Francois Chollet, who is the creator and maintainer of keras, and is now working for Google. Respective of that, I think it does the best job of communicating how the library is intended to be used, and puts things in the right context of experimentation with hyper-parameters and other topics that can take up all of your time.

In any case, I was recently asked what I thought about the joint project Gluon from Amazon and Microsoft. I think it is trying to do the same thing as keras, and I am hoping that it doesn't mean more divergence in the stack.  Already, the deep learning stack is pushing in the hardware direction. There were already a level of matrix math operations that were implemented in hardware, which is why Nvidia GPUs were such a early accelerator for progress based on similar needs in graphics processing, and AMD and Intel are not far behind, with Intel's acquisition of Nervana systems a key recent purchase in designing chips built for purpose. Google's TensorFlow Processing Units (TPUs) take this a step further, and are designed to push more of the TensorFlow code into hardware.  Obviously Microsoft and Amazon don't want to be left behind, as they won't be buying TPUs for AWS or Azure. Even Tesla is looking at building their own chips for image analysis in cars.

That leaves those of us at software level trying to find the right API to code against. Right now, the answer is for me keras, but keeping an eye on the whole stack is necessary to see what choice is the right one to make at the top of it.

Sunday, July 23, 2017

Programming Languages

Why are some programming languages more popular than others? The number one reason is libraries. If you are going to be building something for the iPhone, you would have used Objective-C and now Swift because they are supported the tools on that platform. If you are doing Microsoft development, you will probably have a similar preference for C#, because that's where the libraries needed to connect to various applications live. Python is successful in data science, because it has a great set of libraries for numerical analysis which have formed the basis for more complex. Ruby became quite popular because of the Rails framework. Java has a number of open source projects that have kept it alive, things like Lucene and Hadoop, so we can blame Doug Cutting for its continued existence (in addition to a virtual machine that supports a number of platforms). And, of course, we are all stuck with JavaScript because it is the only thing that runs in the web browsers- despite the numerous and ever changing set of libraries that JS developers have been rapidly moving through over the past five years.

 You'll notice that what is largely missing here are any features of the language, whether a language is functional or object-oriented, dynamic or compiled. In the post-modern programming world where things are built from other blocks, these kinds of features of the language tend to be dwarfed in importance, because the average enterprise project includes more code written elsewhere than it does code written specifically for the project.

Saturday, March 01, 2014

Another post about nothing

This post by Jeff Atwood really gets at why I don't blog much any more. There is a certain kind of pressure when people are actually or might be actually reading your blog that you don't get when it is just sitting there. It's the pressure that what I am doing at the moment is not as interesting or useful to other people as the things they are doing so I don't want to waste people's time even skimming it to decide that themselves. I also ran into some incidents where idiots took everything I said in the way they wanted to, or used it to strategic advantage against me to supposedly reveal my hidden preferences or lack of commitment to their chosen path in The Neverending Battle over Which Technology to Use. And then there are the other things, the Twitters and Google+... anyway, maybe I'll find something useful to say again someday. Just reading through my old ideas, there aren't too many embarrassing ones that would prevent me from ever looking at this stuff again.

Saturday, November 17, 2012


Counterparties: The lessons of tuition inflation

Felix Salmon dismisses the effect of loan subsidies on rising tuition costs by pointing out that the loans get repaid with interest, so that means it is not a subsidy.  Besides being a very weak argument, whether or not it is labeled a subsidy is immaterial. It is credit which allows people to pay more. Without a cap, colleges wouldn't be able to charge as much, and hire so many useless administrators. The cost of college is out of control. Watching the online disruption is fun. Would love to find a way to get more involved....

Thursday, November 15, 2012

Money for Music

Reading this semi-widely shared piece by Damon Krukowski- I really think the analogy is wrong. Pandora is like having your song played on the radio, not like selling 7" of vinyl. When I buy a piece of vinyl, I can listen to it as many times as I want, or at least until the grooves run down. He tries to compare the royalty for 7,000 plays of a track on Pandora to 7,000 sales of a single?? It's more like a radio station with 7,000 listeners playing it once. How much is Galaxie 500 making in radio royalties? I haven't heard them on the radio for 15 years. They should be comparing their $1.21 to $0.00.

Friday, October 05, 2012

Domain Modeling

Should the domain model be separate from the serialization format? For example, if you are using something like Avro or Thrift as your serialization format, does it make sense to have all of the objects in your domain modeled as objects generated by these technologies or is it better to just model them as Plain Old Java Objects and then find a way to serialize those?

This question has recently come up, and I find it reminiscent of the older question of whether your XML data model should be your domain model or whether your database model should your domain model. The basic difference effect of using this pure data model approach is that you end up with classes that do not contain any behavior- basically structs. In the other case, you end up with objects that have both data and behaviors that depend on that data. The additional wrinkle is that generating classes from schema, be it thrift, Avro, ProtocolBuffers or XML adds the additional wrinkle of working with generated code. You typically don't modify generated code, so this can make adding methods, functions, etc. to this data a little challenging.

There is a great body of knowledge out there that supports not putting a dependency in your code on a particular technology implementation. In particular, modeling your domain effectively and independent of other components follows a lot of solid principles, and lets you pursue a domain driven design. The opposite, an Anemic Domain Model has its limitations- a procedural design, generally missing out on OO.

So The unsatisfying answer, of course, is "it depends". To make this answer a bit more satisfying, we need to look at what it depends on, and how those dependencies affect the answer.

The first "depends" I would consider is the programming paradigm. If you are working in a functional programming paradigm, not just a procedural one, for example in a language like Clojure, separating data model from behavior is a natural pattern. Many of the object oriented advantages to a rich domain model simply do not apply. If you are working in an object oriented paradigm, for example in a language like Java, it makes more sense to put the behavior with the data.

These are still generalities though. Generated code is less of a challenge in languages other than Java. Some languages, such as C#, allow partial classes, so you can combine generated code with code that you write in the same class. Dynamically typed languages like Ruby provide a lot of flexibility, so having model classes that are generated by a framework like ActiveRecord, or a pattern like Active Record)is not a problem, because the classes contain no code.

The question then comes up of things like the Hadoop MapReduce framework. While your code may be in Java, the programming paradigm of map reduce is functional. (Some would even say that it is giving us an opportunity to get away from the object oriented programming model- and that it is sad that we produce CompSci graduates that only know Java.) Thus the benefits of having behavior associated with your objects is greatly reduced.

However, in the particular case we are considering, we are not just working in MapReduce, we have a number of application layers that can use the domain object. We are working in Java, so we can't do a lot of cute things with partial classes or mixins. We do want to use Avro as a serialization format though, so there is some momentum for using that as the medium of our domain model. It forces a certain weakness into our domain model, and an awkwardness to our programming. It is not as noticeable when writing functional style Map Reduce code, but it is quite noticeable when we try to abstract behavior such as validation and mapping operations that naturally go on the model. We can get something close to the

Of course, if we go to a more pure, POJO domain model, it then becomes a little more tricky to go in reverse and make the serialization occur with Avro. Still thinking on this one...

Tuesday, August 21, 2012


The patent system, and intellectual property rights claims in general, are out of control. You may think I am referring to the current Apple v. Samsung case where the invention of the rectangle is in question. I am actually more concerned about the White v. Heinz case. We get no benefit from allowing these independent inventors to hold companies who came up with the same idea hostage...just as we get no benefit from more organized patent trolls. It's a game lawyers are playing to exploit the legal system to take money from people. I came home yesterday and my daughter had created a shop for Patented fashion designs, along with a licensing scheme if you actually wanted to make it. The sewing machine was sitting there unused. The designs were great though, but the inspiration was clear. I guess Everything is a Remix.

Saturday, December 25, 2010


It's been a busy year for me, but more troublingly, a year in which I have been full of 75% complete ideas that haven't been worthy of their own posts.  I have done a fair bit of reading though. Here are some of my favorite books that I've read this year (regardless of whether they came out this year).  Maybe you can use some of the % back from your Amazon Visa card to buy them.

I've been waiting a long time for this book (ever since How to GET a Cup of Coffee came out).  Let's just say I have been going around for five years talking REST. I didn't actually know WTF I was talking about until I read this book and started to understand HATEOAS.  It came out while I was struggling with a "RESTful" API that was singularly non user friendly. I still have a fundamental issue with REST where it seems like it is jamming too much into the six verbs. I was hoping this book would dissolve it, but it's actually written at too basic of a level. 

Morville, Search Patterns: Design for Discovery
If you are working on application that includes search and discovery features, this book sets forth a great pattern language. There are tons of examples and ideas here.

Anderson, Kanban
I'll come right out and say it here publicly on the Internet: Scrum is not a good software development method. Scrum masters are not useful. I still like XP. Kanban and Lean are much more sensible methods to go with XP practices than Scrum. It's really way too much to get into here, but having spent the past 7 years reading on the Theory of Constraints, Toyota Way, Lean, etc. this is a method that makes so much sense to me I think I've already been doing it without having a name for it. Yes we Kanban!

Fried, Hannson, Rework
These guys just have no patience for all the non-value added aspects of work (shades of Lean and 'muda'). They also have a way to ensure that the work we do maximizes the opportunities for innovation, creativity, and even beauty, ultimately leading to get us towards our own happiness. Work should make you happy.

Matt Ridley, The Rational Optimist: How Prosperity Evolves
The author of the awesome The Red Queen comes back with treatise on why all the things that the whiners keep themselves busy whining about aren't really too bad. In all of the hand wringing about the minor issues of the day, we are vastly better off than people at any time in history. His thesis is that specialization and markets are the prime movers of progress, and the world might not falling apart at the seams as much as it sometimes seems. Actually made me hopeful about life on Earth, which is a rarity.

Sam Harris, The Moral Landscape: How Science Can Determine Human Values
This is a fascinating book. It's bit harsh on the approach for dealing with religious people, but it gets to the heart of where we can find areas of moral agreement, based on some simple starting points. I thought it would devolve into utilitarianism, but Harris takes it in another direction all together. He directly attacks the distinction between values and facts. It really made me question some of my "cultural relativism". I saw Harris speak earlier this year, and it was quite good. Plenty of his videos out on YouTube if you want the lite version. This will be offensive to many, but it's worth at least figuring it how to address these arguments if you disagree with them.

Ricky Gervais Presents: The World of Karl Pilkington
This is a bit funny.  You probably have to heard the podcasts or show a couple of times to appreciate it, but it is really quite good with lots of laughs at the expense of the indefagitable Pilkington.

Lidwell, Holden, and Butler Universal Principles of Design, Revised and Updated: 125 Ways to Enhance Usability, Influence Perception, Increase Appeal, Make Better Design Decisions, and Teach through Design
I love reading about design. I am in the middle of the new Frederick Brooks book on the design of design.  I am still wondering how much reading about design it will take for me to eventually be a novice level design person.

Doug Lemov, Teach Like a Champion: 49 Techniques that Put Students on the Path to College
This is one thing missing from the teaching profession. In the manner of a software patterns book, this covers a set of techniques for running an effective classroom. Buy copies for your kids' teachers.

Jeffrey Pfeffer, Power: Why Some People Have It and Others Don't
Just be aware that I've read this book and have been trying to internalize it. At some point, you have to achieve a balance between getting what you want and being right. Even if you don't have the Will to Power, this will help you understand what those with the ambition are up to.

Ackoff, Beating the System: Using Creativity to Outsmart Bureaucracies
Some good ideas here.

That's just a quick sampler of what I can remember, which gets smaller every year. Enjoy!