The Flub Paradox

Blub and Flub

Paul Graham's classic essay Beating the Averages is well worth re-reading. It is the story of how, twenty years ago, Paul Graham and Robert Morris built an online store generator called Viaweb and out-manoeuvred their many competitors using their secret weapon, Lisp. But it is much more than a success story with a fairytale ending.

He identified what he called the Blub Paradox: why people persist in using so-so programming languages when they could be so much more productive using something more powerful. He works from the assumption that there is a hierarchy of power of programming languages, starting with Assembly and working up to your favourite language, the one you would love to use in your job. He calls the so-so, median language Blub [1] The paradox is that when Blub users looks down at 'inferior' languages, they can't imagine using such obsolete technology; but when they look up the power hierarchy they don't see greater power, but weirdness and attitude. The reason, Mr Graham thinks, is because they think in Blub.

For example, the Blubbist looks at Haskell, and first sees the different surface syntax, the dismaying lack of curly braces and the mathematical terseness of the function definitions. Then they see that data is immutable, and think: "These variables don't vary! How is it possible to write programs without mutable state? And why?" The point is that they don't recognize the power, and hence don't acknowledge the hierarchy.

Mr Graham's essays about programming tend to assume the natural superiority of Lisp over all other languages; you don't have to agree with him about this to appreciate his arguments. I'll call the 'obviously' superior language in any comparison Flub, for much the same reasons he calls the so-so language Blub. I can then isolate the common characteristics of ueber-languages and their proponents, and emphasize that both Blub and Flub are time-dependent variables. In other words, Lisp was the Flub of its day. A few years ago Flub was Haskell, and recently the new contender appears to be Rust, judging from the buzz. (Note: this is about tracking fashion, not a comment on the actual virtues of these languages.) There has been a lot of new language design happening in the last seven years or so, so I don't doubt that there will be a new value of the variable Flub in a few years. (These are just observations gleaned from reading Reddit and Hacker News, which is probably the closest I get to following sport.) [2]

Joe Hacker and The Continuum of Power

Mr Graham is of course careful to qualify his assertion that there is a natural continuum of power in programming languages. He would not himself use Lisp for everything, since Perl is a better tool for the text slicing-and-dicing of system administration - cooks have more than one pot. He was (after all) a guy getting things done, who wasn't going to let purity and aesthetics slow him down. Contrast with this quote about Edsgar Dijkstra in an interview with Donald Knuth: "It would make him physically ill to think of programming in C++". Which is appropriate; Dijkstra was a computer scientist [3]. But a bit ... weird to those of us who enjoy programming as a means of making vague exciting ideas come alive and making impact. After all, Mr Graham ended up selling Viaweb to Yahoo for $50 million in shares in 1998.

Any closer examination of 'power' in this context reveals it to be a slippery concept. Is BASIC more powerful than Assembly? I doubt it, you can do anything in assembly, and BASIC is deliberately limited [4]. Assembly is very long-winded however, and involves constantly playing with sharp knives; if a job can be done adequately in BASIC, then it should. Using assembly for a BASIC kind of problem is sheer premature optimization and requires a much higher skill level. So BASIC is more expressive than assembly. It will take you even less lines to do the job in Python, so Python is more expressive than BASIC. A better working definition of 'power' here is therefore 'expressiveness'. In the hands of a master, Lisp is more expressive still, because of its meta powers (code that writes code). So Paul Graham and Robert Morris could out-code their competitors; a small posse of smart hackers in tune with their tools can always run rings around larger gangs of programmers led by non-technical people. And that was their secret weapon, ultimately.

So the more expressive a language the better? Not necessarily; consider APL, famous for its one-liners. This represents 'too much of a good idea' at least for people who like typing with normal keyboards. Expressiveness has a lot to do with library support, not just ideas-per-line; Python is well-known for having libraries to do just about anything and finds a lot of use in the scientific community.

'Power' and 'Expressiveness' turn out to be separate concepts. In the end, there is no simple continuum that you can use to line programming languages up, nose to tail, feeble to awesome.

We prefer to reduce vector spaces to scalars, perhaps because of this psychological need to rank things and people. In particular, the multi-dimensional nature of programming space undermines the central Blub argument: "By induction" he writes, "the only programmers in a position to see all the differences in power between the various languages are those who understand the most powerful one".

Except there isn't such a beast.

Blub: Language or Community?

Blub is probably more an attribute of a language's community than the language itself.

Consider Java. In the "Enterprise", only the architect gets to have fun; the developers are not expected to take pleasure in their work, and the feelings of the users are irrelevant, since it is only their managers that matter.[5] This is the very opposite of the Flub spirit.

But Java is a pretty productive language, if used with an independent mind. The proverbial 'small posse' could do very disruptive things with it. To use it effectively, however, requires using an IDE, and Flubbists hate IDEs, partly because their finger muscle memory has been overspecialized from spending too much time in Eighties power editors, but also because using an IDE is too closely associated with Blub practices. (People who can't ride a bicycle without training wheels, basically.) Having recently made my peace with IDEs again, after a long sabbatical, I can attest that they are irritating - like a pedantic pair programming partner. But you learn to ignore the yellow ink, and other fussy ways, to get the productivity boost.

Now I don't doubt you can do better than Java the language, and still remain in Java the ecosystem; Scala shows signs of actual adoption, and Clojure is an attempt at re-Flubbing Lisp. But a underwhelming language is no obstacle to getting things done by determined talent, if the talent is pragmatic enough and does not get physically ill like Dijkstra. Knuth himself wrote TeX in 'literate' Pascal; his idea was to embed the code in a clear narrative, and incidentally get around the limitations of the language (it did not support separately compiled modules, for instance.)

The Flub Paradox

The Flub Paradox can be stated like this: although Flub is self-evidently more powerful and productive than Blub, relatively few people use it and its effect on the real landscape of innovation appears minimal. Where is Flub being used as a secret weapon, in the way described by Paul Graham? Why aren't its practitioners exploiting their natural advantage?

An example of the Flub paradox is the Haskell web framework Yesod; that is not an earth-shattering list of uses, when you take away the Haskell-related sites.

A disruptive startup is more likely to use Blub in creative ways, focusing on the idea, not the implementation. Facebook dominated its market - using PHP - which everyone agrees is crap. Google built up the world's biggest advertising agency using Java and C++; Android is a Java-like ecosystem that runs in most pockets these days.

(At this point, "The Unreasonable Effectiveness of Blub" seems like a good title for an article: Simon Newcomb's analysis of heavier-than-air flight confronted by the tinkerings of bicycle makers. Be my guest.)

Mr Graham says (and it's still true) that "when you're writing software that only has to run on your own servers, you can use any language you want". But the modern web is a dance between the server and the client, and that client is running JavaScript.

The vitriol surronding JavaScript is interesting and revealing. Granted, it is not well designed and was knocked together in an ridiculously short period of time; but it has good influences, like Scheme's closures[6]. It could have easily been something much worse, like Tcl. Yet the Flubbists hate it, it's untidy and used by yahoos (pun intended.) It is an obstacle to the march of Flub in a way Blub never is.

That's also perhaps why so many words are wasted on flaming Go; a deliberately simplified, anti-theoretical language that is gaining ground, particularly in the business of writing server software. Go is like a predatory shark appearing amongst dolphins; competition for resources that also eats your babies. It is that terrible thing, a rising Blub.

If you are a Flub practioner, you want there to be Flub job opportunities; it is thus very important to establish the idea that Flub is the next best thing. A veritable deluge of articles appears, explaining the beauties of Flub and giving crash courses on the mathematical background needed to understand it. For languages that arise in academia, this is necessary work to establish your reputation and secure a comfortable position from where you can continue to write more articles. If they remain in academia, it is necessarily and also sufficient.

That's partly why there is a Flub Paradox; people like Paul Graham who are sharp hackers and prepared to risk their livelihood to follow a startup dream aren't common. Plus, although they may ascribe their success to Flub, it is actually only one of many factors, and the most important of these are talent and motivation. You put a group of merely good Lisp programmers on a project, and embed them in a corporate environment, and the stellar results are likely to be not reproducible. (I don't doubt they will be more productive than Java programmers. But will the result blend?) Big companies understand this problem well, and prefer Blub programmers, who are easier to source and come with less attitude.

Part of the problem is over-selling Flub as a universal panacea (Fred Brooks' "No Silver Bullet" again.) For example, there has been a great deal of heat round the new language Rust from the Mozilla foundation; a strongly-typed systems language with better guarantees of safety.

To use Rust for writing programs that can afford the overhead of garbage collection would be premature optimization. An engineering analogy: titanium is a fantastic metal, stronger than steel yet half the weight - it does not rust; but it's a bitch to work with. and expensive. By Mr Graham's argument, we should prefer it over aluminium, all other things being equal, except that they ain't. Using a more advanced language could be significantly more expensive than just doing it in Java. I'm not just thinking of the developer cost, but the time cost. For instance, it is reported that a 2.4 kloc Rust program takes 20sec for a dev build; that's very long for such a dinky program. Java, Go and C would be practically instantaneous; for comparison, a 10 kloc C++ program takes under 10sec for a full rebuild, and average rebuild times are about a second. It is important that Rust works on its incremental build story. We hear the authentic voice of despair: "getting blocked for several minutes every time I need to recompile is a big productivity killer after a while."[7]

I don't doubt the world needs safety guarantees, given the messes created by cowboy firmware. C is famous for letting you shoot your own foot; in safety-critical applications, you are shooting someone else's foot. Rust is a sincere attempt at solving that kind of problem, as well as other systems software, like operating systems.

However, these are fairly niche areas of development; it is unlikely that major operating systems will emerge written entirely in Rust, and embedded engineers are a conservative bunch - they usually come from an EE background. Also remember that theoretical correctness and safety guarantees are old preoccupations of the industry, and Ada is well entrenched in some parts. I suspect that the idea of a magically safe language will turn out to be yet another silver bullet; it would be interesting to be proven wrong.

Edit: It was probably a mistake to mention actual concrete values for Flub, again see[1]. The point I wish to emphasize has nothing to do with the value of Flub or Blub, and everything to do with not seeing Flub used as a disruptive force of change, as Lisp was in Paul Graham's original essay. It is probably simply too early to tell.

[1] Trashing a developer's language of choice by name is like insulting a patriot's country or his mother's apple pie - the nerd equivalent of bumping the pool cue of a big man in a bar. It makes further rational discussion impossible.

[2] It would be interesting to do sentiment analysis on these august websites and selected social-media feeds, and track the buzz surrounding popular programming languages. Not exactly science, but numbers are good fuel for an argument. The Tiobe Index is approximate and not good at capturing changes in the 'long tail' where Flub lives.

[3] It's entirely possible that programming in C++ can make you physically ill, but the point is that you have to learn it and use it to know. Dijkstra arrived at this conclusion from first principles.

[4] BASIC was originally a teaching language from the University of Dartmouth, baby FORTRAN. Niklaus Wirth's Pascal is another example of a language designed to teach a very structured way of thinking. So criticisms of Pascal are a bit unfair, because it wasn't intended for large scale application. The dialects that appeared later that some of us remember with affection were much more capable.

[5] The Marxist concept of 'alienation of labour' is applicable here, since it always was as much a psychological as an economic concept. Like a craftsperson becoming a factory worker, enterprise programmers are both separated from the value they create and divorced from the pleasure of creating value. No fun, and no big bucks either.

[6] Closures are functions that carry around their surrounding state with them; they are probably the most powerful thing I experienced through Lua. A pity that JavaScript was standardized quickly in such an uncooked state, since the steady development of Lua produced a much cleaner, simpler design. In many ways it is like JavaScript without braces: prototype-based object system, field references are lookups (a.f == a['f']) but with proper lexical scoping. The Web would be a better place if it ran on Lua, but modern Lua arrived too late. As an example of how surface syntax has such an unreasonable hold on developer opinion, Lua indexes from 1, not 0. This is apparently a problem. Besides, it does not have curly braces, which are an obvious marker of good language design.

Fortunately, modern C++ does have closures ('lambdas') and this has made me a happier man.

[7] Personally, this would be a deal breaker, because it would mess with my flow. Everyone has different strengths and patience isn't one of mine. I spent too much energy railing against the situation in C++, but learned to accept given reality when I got better hardware. But I still miss Delphi.