How Search Will Affect Programming Language Design

How Search Will Affect Programming Language Design

Global Recall Over Recognition: A Practical Use for blekko’s Date Range Slashtag

After daydreaming for a while about what to write my first blog post about, I decided just to pick a small but useful productivity hack and share it. But then this daydreaming got me really thinking. Why is this small hack significant in the first place? And is it important that this small hack will become even more relevant six months from now?


My first programming language was Common Lisp. I taught myself by struggling all the way through Paul Graham’s excellent but dense book ANSI Common Lisp, then forcing myself to tackle SICP and On Lisp late at night after my normal workdays. It took me a lot of work to become productive enough to build complex systems.
My recent experience diving into Ruby and Rails has been quite different: It’s only been three weeks and I’m already happily hacking around inside the fairly complex production codebase of an SEO reporting and analytics platform. Granted, this is no longer my first programming language. But this brings up a big contrast between the two situations that really sticks out to me: I haven’t read either of the books I bought on Ruby/Rails (the Pickaxe book and the Rails 3 Guide). I’ve also never even seen any of the official Ruby/Rails documentation (yet). All the questions I have had so far have been answered by quick Google searches, leading directly to task-relevant blog posts, tutorials, StackOverflow threads, or even other people’s Github accounts.
Not everything about Ruby/RoR resonates with me, in fact some of it is pretty annoying and makes me wish I was hacking lisp, but I must admit that I’ve found myself productive and in the thrill of purposeful creativity with RoR in a very short time.

Making Tools Useful

I believe it was Donald Noman in The Design of Everyday Things who said that good design is the act of transferring insight from your mind into the external world in the form of a useful physical object (I don’t remember the quote exactly, accidentally I left my copy of the book in the Dublin/Pleasanton BART station, let me know if you found it 😉
So this means that designing a useful tool involves taking an insight about performing that task and encoding it directly into the tool via its design. That insight will then hopefully leap into the mind of the next user, and thus gain a life of it’s own.
Designing a complex programming language is the same in this respect. The designer of the programming language has a deep insights and lesson learned about performing an algorithmic task, and thus chooses to include certain features to enable, encourage, or even force the next user to grasp that insight while performing similar algorithmic tasks with this language.
Much like the design of Erlang programming language includes process monitoring features that can restart errant processes to finish failed tasks, the design of a hammer includes a simple notch on the back that can be used to pry out errant nails and thus enable the user to bang in new ones.
The hammer is a bit easier to figure out, however, since the affordances provided by its intuitive design naturally suggest all possible uses.
Fortunately for Erlang, affordances aren’t the only thing that store human insight for posterity. So do blog posts and how-to guides on the Internet. Maybe programming languages no longer need to be designed as intuitively as a hammer, because now programmers have instant access to a global network of human insight whenever they perform the act of programing.
I wonder if the biggest fundamental difference between the design of the 21st century’s greatest hammer and the design of the 21st century’s greatest programming language won’t be defined by the inherent complexity of the intended action, but rather by proximity of the intended end user to a search engine.

Recognition Over Recall

It is well known that the human mind can recognize a pattern faster than it can recall it from memory.
This principle can be seen at work in any good programming environment. For example:


  • Fuzzy completion in SLIME means that you don’t actually have to accurately remember the names of functions, just type a few letters you think might be in the name, hit tab and you will be presented with a list recognize the right function thanks to consistent naming conventions.
  • Live printing of function signatures means that you don’t actually need to remember how to use a function until the last second. You can usually recognize what arguments it should take from the names of the parameters, which usually follow recognizable conventions.

Programming language Design

  • There are 972 different symbols in ANSI Common Lisp, each with a unique name. But working with this number of symbols is manageable because their names and meanings adhere to easily recognizable patterns. nreverse is the composition of the symbol reverse and the “n” prefix convention used for naming destructive operators (similar to the “!” suffix in other languages).
  • There are three different generic equality operators, each with different behavior (equal, eql, eq), but once discovered they are easy to remember since they follow a pattern: the more condensed the spelling the faster and more specific the operation.

Without these patterns and recognition tools, it would be extremely hard normal person like me to recall how to use Common Lisp’s 972 symbols.

Search Engines, Operant Conditioning, & the Blogosphere

I have been using TextMate instead of Emacs for Ruby/Rails hacking so far. I haven’t missed the fancy IDE features (yet), because so far I have found all the language help I have needed with Google. In fact, I haven’t stumbled upon a Ruby/Rails question yet that I haven’t been able to find the answer to with Google. It has been as integral to my experience with Ruby/Rails as any individual feature of the language itself.
Sure using TextMate and IRB running in terminal means I will get 346 possible completions if I do a method completion of a simple instance of a subclass ActiveRecord. And sure they will be displayed in giant alphabetically-sorted columns instead of broken up in a logical way. But Just Googling It means that I in fact still don’t have to remember the names of any of the operators required to perform a certain task, nor even learn too much about how they work until I need them.
This makes me wonder if the aid to human recognition and recall provided by search engines will prove to be a more powerful force dictating the design of the 21st century’s programming tools than traditional programming language design itself.
In his talk Growing a Language, Guy Steele writes:

Language design is not at all the same kind of work it was thirty years ago, or twenty years ago. Back then, you could set out to design a whole language and then build it by your own self, or with a small team, because it was small and because what you would then do with it was small.

Now programs are big messes with many needs. A small language won’t do the job. If you design a big language all at once and then try to build it all at once, you will fail. You will end up late and some other language that is small will take your place.

It would be great if there were some small programming language that felt large, the way Basic English is small but feels large in some ways. But I don’t know how to do it and I have good cause to doubt that it can be done at all. In its day, APL was a small language that felt large; but our needs have grown and APL did not have a good pattern for growth.

So I think the sole way to win is to plan for growth with help from users. This is a win for you because you have help. This is a win for the users because they get to have their say and get to bend the growth to their needs. But you need to have one or more persons, too, or one or more groups, to take on the task of judging and testing and sifting what the users do and say, and adding what they think best to the big pile of code, in the hope that other users will trust what they say and not have to go to all the work to test and judge and sift each new claim or each new piece of code, each for his own self.

Parts of the language must be designed to help the task of growth…

When examined under this light, examples such as the quick, unanimous adoption of RVM, then RVM+Bundler over the original module system make the Ruby/Rails Borg really shine. Rails’ absorption of Merb is another great example of how the system has managed to successfully “grow” in the way described by Guy Steele. (it does also strangely remind me of the creation Abraham Lincoln’s Cabinet in 1861 😉
Getting back to the point about search engines:
In addition to facilitating the growth of the language, the Borg-like nature of the Ruby/Rails entity has an additional secondary effect: significant operant conditioning for the authors of useful web content. Unlike a dense textbook which requires a lot of time and great sacrifice to write, all that is needed to inspire crowds of Ruby/Rails hackers to produce an endless universe of useful blog posts, how-to tutorials, StackExchange threads, etc is positive feedback in the form of recognition, thank you’s, and frequent job offers.
Since this “soft documentation” is usually written in task-relevant, inverted pyramid style, and is well indexed by Google (unfortunately for us lispers, usenet posts and AI textbook and not indexed very well by Google and are usually kind of complicated to read), Ruby/Rail’s “Just Google It” JIT learn/hack workflow is characterized by a surprisingly low signal-to-noise ratio and a natural pattern of progressive disclosure .

The Practical Part of the Blog Post

So… The only problem I am having with Ruby/Rails’ JIT “Just Google It” Learn/Hack algorithm is not that there isn’t enough information, but rather that there is TOO MUCH information available. Specifically, I’m always finding information about versions of Ruby and Rails that I’m not using. And given what a moving target Ruby/Rails is, different versions of the same libraries/tools can turn out to be radically different.
James Ludlow over at Working as Intended has already demonstrated that blekko’s “/ruby” and “/rails” slashtags still have a long way to go to beat Google. Still, blekko has one more feature that is pretty useful when searching for Ruby/Rails resources: the date range slashtag (“/dr=YYYYYYYY”). This somewhat esoteric blekko feature is actually useful when context switching between different versions of Ruby/Rails, since you can filter out all the content from other points in time clogging up all the search results.
eg: “/ruby /dr=2009-2011 activerecord”
vs: “/ruby /dr=2007-2009 activerecord”

My (Subjective) Conclusion

Times have changed. In the words of Guy Steele: “Language design is not at all the same kind of work it was thirty years ago, or twenty years ago”. The fact that now one can learn a new programming language without reading a single book or formal documentation using the JIT learn/hack algorithm, and that sometimes one already needs a special hack to deal with the amount of quality resources available on the Internet, means that the Search Engine Effect really is now part of the language design equation.
Rubbing shoulders with the Ruby/Rails Borg has been a pleasure, and makes me appreciate the power of the Borg approach. But it also makes me appreciate the power of small gem languages like newLisp, which I have found incredibly useful in side projects requiring a lightweight distributed scripting solution despite having an active userbase that is estimated to be under 473.
It also makes me think about and be impressed by Emacs/Elisp, which has managed to stay practically useful and malleable for MORE THAN 30 YEARS without namespaces or even a decent module system.
More than anything it makes me excited to learn about the 21st century’s new generation of tools which both have both an emphasis on language design and leverage the power of the Borg, such as Clojure (which capitalizes on the Java platform) and Node.js (which takes the worlds most popular programming language and turns it on it’s head).
One way or the other, whether you prefer to wield a meticulously crafted language gem, or prefer to leverage the power of the Borg (or are lucky enough to be doing both), there are plenty of opportunities for purposeful creativity to go around. And plenty of productive tools to work with while doing it. What an exciting time to be a hacker!