Value of Software in an Era of Increased Supply

I made a post on this topic over on the Osparna blog. 

The bottom line up front here:

"The broader economic point though is that there is going to be a lot more software, that software will be built more quickly, and it will be able to rely on large models to provide novel capabilities. The baseline economic response to an increase in supply without concomitant increase in demand would be for the price to fall. Absent other factors, software will become cheaper.  From an investment standpoint, the software is now less interesting than it used to be. We now want to see what relationship the company has with the customers, how well they solve the problem, and what unique dataset or insight do they have that will allow them to maintain a competitive moat. It can’t just be the code."

AI slop

 How do I avoid writing something that is no different from AI slop?

Why don't AI systems come back to you with a clarifying question? 

Why is everything an answer, an overconfident blag.

Identity Field

The Identity Field has to be the most obvious pattern. As Martin Fowler says, it is "mind numbingly simple."

However, imagine a system that lacks it. Total nightmare to deal with, as you have to define complex identity rules (for example, is the name and location the same- how similar enough can be considered "same"). 

I submit that this is also a challenge from a UI perspective in web applications. If you don't have a list-detail UI pattern, it gets very hard to find things. This is frighteningly common in the single page applications, where to get a piece of data you have to go to Dashboard three, left column in mode orange, scroll down 3 items. It requires the user to remember the association between the entity type and which dashboard it is on. The UI becomes incredibly brittle if you want to enable even the most basic hypermedia link of jumping from some reference to the entity to the dashboard in a particular mode.

The detail page for an entity is the page driven by its Identity Field.  It is its homepage. It is easier to find things if they have their own home. The home for the entity type is the list page.  The navigation for the application can reference these list pages, which let you navigate to a particular entity. Again, this seems "mind numbingly simple." However, when you have system that lacks it, it can become mind numbingly complex to navigate. 

Accuracy vs. Explainability Tradeoff

This fascinating article attempting to explain machine learning for statisticians is very interesting. I don't know if I even properly understand it, and the tone is a bit negative. Here's the simplest way I can think of it.

If I have:
    (b1 * x1) + (b2 * x2) + (b3 + x3) + a = y
the statistician is trying to minimize error in the b values and the machine learning person is trying to minimize error in the y value?

If you get the stats "too correct", the ML guy will know you are overfitting, and model will do worse on new data that was not in the sample/training set.

Is that it?

The Twitter for blogs

 Thinking about whether there is a way to beat Twitter with open standards. The day Twitter shut off RSS (so that people couldn't build apps that avoided their ads) we lost a huge function of it as a part of an aggregation system. However, tools like NewsBlur show it can still be scraped.

Spotify is trying to create a walled garden around podcasts. Again, shutting off the aggregation function.

Google Reader was very popular. With some additional tools to create self-hosted content, likes, and some additional discovery tools, it could have been a distributed twitter.

We may be able to do this.

Lowering the bar

I have another post sitting in the dock, but I can't really release it because I need to think about it a bit more. I've been trying to lower the bar yet again so that I can produce some shorter things, but I still don't want to put out things that are wrong!

Fred Wilson at AVC https://avc.com/2020/07/short-and-sweet/ has posted about how he has been getting shorter with his blog.




If shorter is good enough for Fred, it's good enough for me.

Short content seems like Twitter. I just am not finding Twitter rewarding lately. It seems like the Cultural Revolution on there, just people spitting at each other. Not pleasant in any way. I was trying to keep a list of just silly funny stuff in my approach to using lists as the way to consume Twitter, and only following people I know personally and am thus less likely to be sickened by. I just can't stick to it properly and keep following people.

Anyway, things are so ugly on Twitter I don't even feel like linking up these posts there.

Thoughts on David Patterson and Lex Fridman

I was watching the fascinating Lex Fridman discussion with David Patterson and found it very stimulating. It really made me think about the nature of computer science as a whole in a way that I haven't for a while. David Patterson is just so clear when he speaks about things that it makes you consider the basic nature of the endeavor. It reminded me of so many things I used to think about more frequently.

https://www.youtube.com/watch?v=naed4C4hfAg

Some random notes or ideas that occured or recurred to me. Not necessarily in the discussion, but where my mind went...

• The concept of layers of abstraction in computer science is simply fascinating. It is really possible to keep diving down deeper. It reminded me of when I was in my first electronics class and I was asking about how AND and OR logic gates worked. That wasn't really the point of the class, it was more about figuring out how to assemble those parts to make an adder or do something useful. The teacher didn't even really know or understand my question (or why I was slowing down the class by asking it). Some other student volunteered that were must be some kind of thing inside of the gate that would close or open based on the total voltage. It took me a while to realize this word I had heard so many times, transistor, was really how these were made.
• I was then drifting to thinking about the intersection between philosophy and computer science at that most basic level. Not only are the logic gates abstractions for the transistors used to create them, the gates themselves are assembled into things like adders, that compose arithmetic operations from boolean logic operations. This ties to (analytic) philosophy a bit, when you think about how Frege was talking about Peano's axioms, and Russell and Whitehead trying to derive mathematics from logic. It took me back to a semester when I studying Wittgenstein and Computer Architecture (two separate classes) and kept sliding to how Frege's types and type systems in programming languages are similar. I wonder what kind of programming language Wittgenstein would have created- maybe it would have a been a game...
• Of course, compilers themselves are a form of translation from one language to another. That's always interesting to think about. I really didn't quite grasp how Intel x86 architecture translates from CISC to RISC in the chip, just in time.
• The idea of compilers then got me thinking about the idea of interfaces as mediating between layers of abstraction. The layer of abstraction can be thought of as an interface. It defines how to interact with that layer, but provides this additional opaque box capability of being able to swap out the implementation without changing the other layers. 
• Some more prosaic thoughts on the idea of the best technology not always being the market winner. How much I hated Elasticsearch from a pure attitude and marketing perspective, their deceptive ways  How I felt vindicated when they went away from open source. How much I hate their current market position and hope AWS crushes them. 
• Just the discussion of how ARM has so effectively pursued RISC, and how Apple is making the software stack integrated with the hardware so effectively, I wonder when there will be a true open architecture phone with mass adoption that can compete. Amazing that Apple is moving to this for their desktop operating system. It seems crazy, but the possibilities are huge.
• Finally, the chat on mlperf was so interesting. I didn't realize Nervana had refused to release their mlperf scores prior to the acquisition by Intel and their tech has been abandoned. Meanwhile Intel acquired Habana Labs, partially because they did have good mlperf scores. Benchmarking is so important. It made me want to benchmark things I am doing!

Okay, that's about it so far, but I am going to try to start recording my impressions like this more often. Probably won't be able to keep it up. But who knows?

Transitory technologies

Most of the JavaScript frameworks seem to be transitory technologies. They exist to fulfill some perceived gap in the web browsers' capabilities. Once those gaps.are filled, their reason for existing fades.

Probably the two strongest pieces are Web assembly and Web components. We also have webgl. It seems hard to imagine that a virtual DOM and another event model on top of the event model that already exists, are the right way to deal with these technologies.

The only question is how long it takes.

Text as Design

I am not opposed to Design. I do think there are a lot of advantages to text as opposed to image heavy design, but I see both sides of both sides. There was a trend for a couple of years to have website home pages be huge, highly detailed, saturated images, or even video. The idea there seemed to be create this feeling of lushness or richness. This has obvious costs- bandwidth, latency, etc., but there is also something decadent about it. We have all of this bandwidth, let's use it. A picture is worth a thousand words. However, these are fleeting feelings that leave me empty. I need the words to create a rational idea out of the picture.

I need the command line.

I love the episode of the Netflix series on design "Abstact" that featured Paula Scher. https://www.netflix.com/watch/80093802?trackId=14277283 There is a lot you can do with typography. I do see evidence that people are coming back to text favored design.  https://cheapskatesguide.org/articles/beauty-of-text.html.

I am sick of fighting over which icon to use on a toolbar when a word would do-- although this does bring another battle. Which word? Which language? We live in a world where we fight over words. Do symbols let us agree to disagree? Do they let us put our own interpretations into those symbols, while disagreeing under the surface about what they mean?

I kinda want to switch from using blogger to get more fonts.

Purist Programming

Programming is about making things. Purity is about adhering strictly to a set of rules. It seems that one should favor the programming over the purity. Abstractions should be minimal.

Think about this thing from John Cook on "Pretending OOP Never Happened".

That has been my experience. I hardly ever write classes anymore; I write functions. But I don’t write functions quite the way I did before I spent years writing classes.
And while I don’t often write classes, I do often use classes that come from libraries. Sometimes these objects seem like they’d be better off as bare functions, but I imagine the same libraries would be harder to use if no functions were wrapped in objects.

Cook then quotes James Hague's "Follow-up to "Functional Programming Doesn't Work":

100% pure functional programing doesn’t work. Even 98% pure functional programming doesn’t work. But if the slider between functional purity and 1980s BASIC-style imperative messiness is kicked down a few notches — say to 85% — then it really does work. You get all the advantages of functional programming, but without the extreme mental effort and unmaintainability that increases as you get closer and closer to perfectly pure.

Getting to the point of it, the pure functional people are the most annoying people in the world. The world has state. You need state.

I really get annoyed by the programming paradigms in place now on the web, particularly the front end paradigms. There is a very simple event driven model that works surprisingly well. The component is an object, with state, that responds to events...anyway probably worth exploring the world of JavaScript separately.

GPT-3

I am not sure if people who haven't been doing machine learning can appreciate how weird GPT-3 is or what "few-shot learner" means.

In normal NLP machine learning, you might start with a pre-trained language model that encodes the relationships between words of a language. You then build a training set of data, probably 1000s of items where you have labels applied to text, correct translations, answers to questions, and things like that. You then train the model to minimize error on that training set. Then, with the trained model, you send it new samples of text and it spits out a label, translation, or answer as appropriate.

With GPT-3, a much bigger language model, trained to just predict the next word, you don't have to do any of that. You construct the whole task in the last bit, where you would normally be sending a trained model new samples of text. The trick is, you send in a description of the task in with the text. So you could send in:

 translate from English to French: hat => chapeau, cat => chat, hello => 

and it would send back "bonjour".

It learned enough about language to be able to have examples of what typically follows "translate from English to French" to be able to get good performance on that task. This wouldn't be surprising if it had been trained on that task, but there is no task specific training (aka fine-tuning). It was just trained to predict the next word. Having a big enough model (a mind boggling 175 billion parameters) it just picks up that whole task as a pattern.

Read the paper.

Decisions

Thinking about decisions

Check out this product from Osparna that supports investment diligence and decision-making. The idea is that we should take a systematic approach to decision making around investments. Rather than only base them on our intuition, we decide what the basis for a good decision is, and then gather the information. There's a lot more to it...

Black box

https://news.ycombinator.com/item?id=20826979

Unpopular quote from my image and video processing professor - “The only problem with machine learning is that the machine does the learning and you don’t.”

While I understand that is missing a lot of nuance, it has stuck with me over the past few years as I feel like I am missing out on the cool machine learning work going on out there.

There is a ton of learning about calculus, probability, and statistics when doing machine learning, but I can’t shake the fact that at the end of the day, the output is basically a black box. As you start toying with AI you realize that the only way to learn from your architecture and results is by tuning parameters and trial and error.

Of course there are many applications that only AI can solve, which is all good and well, but I’m curious to hear from some heavy machine learning practitioners - what is exciting to you about your work?

This is a serious inquiry because I want to know if it’s worth exploring again. In the past university AI classes I took, I just got bored writing tiny programs that leveraged AI libraries to classify images, do some simple predictions etc.

The complaint that a neural network, or deep learning model, is a "black box", where you can't see why the model is making a particular output seems quite weak to me. Not being interested in it because of that is worse, because it is like saying that we shouldn't study neuroscience because we don't understand how memories are stored in the nervous system.  The box is not black. It is just a very complicated box, with lots of weights and layers. You can look at the weights and inputs and layers and literally see exactly what simple math applied to the inputs led to the output. It might take you a while, but it is all there. We are building tools all of the time to make it easier to make this process more efficient, but, unlike a brain, no one is stopping you from reaching into the box and taking a look.

Keras

I've been working hard to keep up to date with recent progress in machine learning.  The progress in Deep Learning is exciting to me, especially because it builds on all of the things I learned about neural networks in 1995-99 in college in grad school (particularly some work with Dr. William Levy at UVA), but never really had the opportunity to put into practice directly, except in some simple classifier type situations.

In any case, last year I took a short course from Miner and Kasch taught by Florian Muellerklein that was quite good. You can check out the slides and code on github.  I'd previously been messing around with Torch, Caffe, TensorFlow, cuDNN and a variety of other libraries, while easier than the old days of finding eigenvalues in c++ or running out of MatLab, they required a lot of configuration and such.  In the course, we jumped right into using keras. Wow, so much easier. It's a bit like a Ruby on Rails for neural networks, giving you some sensible defaults and get going right away, minimizing common errors, but different in that it is just a simpler interface overlaying other libraries.  If anyone is diving in this stuff, the keras path is the best path that I've tried.

To help one out on this path, there are now a couple of books on keras. Deep Learning with Python is the one that I recommend. It is written by Francois Chollet, who is the creator and maintainer of keras, and is now working for Google. Respective of that, I think it does the best job of communicating how the library is intended to be used, and puts things in the right context of experimentation with hyper-parameters and other topics that can take up all of your time.

In any case, I was recently asked what I thought about the joint project Gluon from Amazon and Microsoft. I think it is trying to do the same thing as keras, and I am hoping that it doesn't mean more divergence in the stack.  Already, the deep learning stack is pushing in the hardware direction. There were already a level of matrix math operations that were implemented in hardware, which is why Nvidia GPUs were such a early accelerator for progress based on similar needs in graphics processing, and AMD and Intel are not far behind, with Intel's acquisition of Nervana systems a key recent purchase in designing chips built for purpose. Google's TensorFlow Processing Units (TPUs) take this a step further, and are designed to push more of the TensorFlow code into hardware.  Obviously Microsoft and Amazon don't want to be left behind, as they won't be buying TPUs for AWS or Azure. Even Tesla is looking at building their own chips for image analysis in cars.

That leaves those of us at software level trying to find the right API to code against. Right now, the answer is for me keras, but keeping an eye on the whole stack is necessary to see what choice is the right one to make at the top of it.

Programming Languages

Why are some programming languages more popular than others? The number one reason is libraries. If you are going to be building something for the iPhone, you would have used Objective-C and now Swift because they are supported the tools on that platform. If you are doing Microsoft development, you will probably have a similar preference for C#, because that's where the libraries needed to connect to various applications live. Python is successful in data science, because it has a great set of libraries for numerical analysis which have formed the basis for more complex. Ruby became quite popular because of the Rails framework. Java has a number of open source projects that have kept it alive, things like Lucene and Hadoop, so we can blame Doug Cutting for its continued existence (in addition to a virtual machine that supports a number of platforms). And, of course, we are all stuck with JavaScript because it is the only thing that runs in the web browsers- despite the numerous and ever changing set of libraries that JS developers have been rapidly moving through over the past five years.

 You'll notice that what is largely missing here are any features of the language, whether a language is functional or object-oriented, dynamic or compiled. In the post-modern programming world where things are built from other blocks, these kinds of features of the language tend to be dwarfed in importance, because the average enterprise project includes more code written elsewhere than it does code written specifically for the project.

Another post about nothing

This post by Jeff Atwood really gets at why I don't blog much any more. There is a certain kind of pressure when people are actually or might be actually reading your blog that you don't get when it is just sitting there. It's the pressure that what I am doing at the moment is not as interesting or useful to other people as the things they are doing so I don't want to waste people's time even skimming it to decide that themselves. I also ran into some incidents where idiots took everything I said in the way they wanted to, or used it to strategic advantage against me to supposedly reveal my hidden preferences or lack of commitment to their chosen path in The Neverending Battle over Which Technology to Use. And then there are the other things, the Twitters and Google+... anyway, maybe I'll find something useful to say again someday. Just reading through my old ideas, there aren't too many embarrassing ones that would prevent me from ever looking at this stuff again.

Subsidy

Counterparties: The lessons of tuition inflation

Felix Salmon dismisses the effect of loan subsidies on rising tuition costs by pointing out that the loans get repaid with interest, so that means it is not a subsidy.  Besides being a very weak argument, whether or not it is labeled a subsidy is immaterial. It is credit which allows people to pay more. Without a cap, colleges wouldn't be able to charge as much, and hire so many useless administrators. The cost of college is out of control. Watching the online disruption is fun. Would love to find a way to get more involved....

Money for Music

Reading this semi-widely shared piece by Damon Krukowski- I really think the analogy is wrong. Pandora is like having your song played on the radio, not like selling 7" of vinyl. When I buy a piece of vinyl, I can listen to it as many times as I want, or at least until the grooves run down. He tries to compare the royalty for 7,000 plays of a track on Pandora to 7,000 sales of a single?? It's more like a radio station with 7,000 listeners playing it once. How much is Galaxie 500 making in radio royalties? I haven't heard them on the radio for 15 years. They should be comparing their $1.21 to $0.00.

Domain Modeling

Should the domain model be separate from the serialization format? For example, if you are using something like Avro or Thrift as your serialization format, does it make sense to have all of the objects in your domain modeled as objects generated by these technologies or is it better to just model them as Plain Old Java Objects and then find a way to serialize those?

This question has recently come up, and I find it reminiscent of the older question of whether your XML data model should be your domain model or whether your database model should your domain model. The basic difference effect of using this pure data model approach is that you end up with classes that do not contain any behavior- basically structs. In the other case, you end up with objects that have both data and behaviors that depend on that data. The additional wrinkle is that generating classes from schema, be it thrift, Avro, ProtocolBuffers or XML adds the additional wrinkle of working with generated code. You typically don't modify generated code, so this can make adding methods, functions, etc. to this data a little challenging.

There is a great body of knowledge out there that supports not putting a dependency in your code on a particular technology implementation. In particular, modeling your domain effectively and independent of other components follows a lot of solid principles, and lets you pursue a domain driven design. The opposite, an Anemic Domain Model has its limitations- a procedural design, generally missing out on OO.

So The unsatisfying answer, of course, is "it depends". To make this answer a bit more satisfying, we need to look at what it depends on, and how those dependencies affect the answer.

The first "depends" I would consider is the programming paradigm. If you are working in a functional programming paradigm, not just a procedural one, for example in a language like Clojure, separating data model from behavior is a natural pattern. Many of the object oriented advantages to a rich domain model simply do not apply. If you are working in an object oriented paradigm, for example in a language like Java, it makes more sense to put the behavior with the data.

These are still generalities though. Generated code is less of a challenge in languages other than Java. Some languages, such as C#, allow partial classes, so you can combine generated code with code that you write in the same class. Dynamically typed languages like Ruby provide a lot of flexibility, so having model classes that are generated by a framework like ActiveRecord, or a pattern like Active Record)is not a problem, because the classes contain no code.

The question then comes up of things like the Hadoop MapReduce framework. While your code may be in Java, the programming paradigm of map reduce is functional. (Some would even say that it is giving us an opportunity to get away from the object oriented programming model- and that it is sad that we produce CompSci graduates that only know Java.) Thus the benefits of having behavior associated with your objects is greatly reduced.

However, in the particular case we are considering, we are not just working in MapReduce, we have a number of application layers that can use the domain object. We are working in Java, so we can't do a lot of cute things with partial classes or mixins. We do want to use Avro as a serialization format though, so there is some momentum for using that as the medium of our domain model. It forces a certain weakness into our domain model, and an awkwardness to our programming. It is not as noticeable when writing functional style Map Reduce code, but it is quite noticeable when we try to abstract behavior such as validation and mapping operations that naturally go on the model. We can get something close to the

Of course, if we go to a more pure, POJO domain model, it then becomes a little more tricky to go in reverse and make the serialization occur with Avro. Still thinking on this one...

Patents

The patent system, and intellectual property rights claims in general, are out of control. You may think I am referring to the current Apple v. Samsung case where the invention of the rectangle is in question. I am actually more concerned about the White v. Heinz case. We get no benefit from allowing these independent inventors to hold companies who came up with the same idea hostage...just as we get no benefit from more organized patent trolls. It's a game lawyers are playing to exploit the legal system to take money from people. I came home yesterday and my daughter had created a shop for Patented fashion designs, along with a licensing scheme if you actually wanted to make it. The sewing machine was sitting there unused. The designs were great though, but the inspiration was clear. I guess Everything is a Remix.