I’m Done Poking Around

August 30, 2013

I’m up to my ears!

The title of this blog is no longer quite fits . Turns out I’m not just poking around Erlang anymore. I’m on a mission. So, today I’m creating a new blog on Word Press that better reflects my commitment and day-to-day journey. I’m calling it Erlang Lab Notes.  I hope to dig into concurrent Erlang, clusters, ARM servers, and more.

Come visit!





Hello Again Erlang

August 23, 2013

Something more than three years ago, in my most recent post to this blog,  I wrote:

“So here’s what I’ve learned about Erlang. It’s a fine language, but not quite ready for guys like me who know just enough to shoot themselves in the foot. My hat is off to the Erlang developers posting increasingly interesting elements of Erlang-based web infrastructure. But until they get more community support and far better user documentation they’ll be sucking the fumes of the Joomlas, Drupals, and Djangos of the world.

“There’s still far too big a gap between the fluency of human language and the brittle tyranny of Erlang code.”

Well, though I still stand by some of the above sentiments, it turns out I never truly left the Erlang fold. I simply buried my head and dug into every source I could lay my hands on to learn the language sufficiently well to build useful applications. I have, I can modestly say, made progress.

“Three and a half years to learn Erlang!” I can hear the snickers. “This dude must be seriously retarded.”

I won’t argue. I’m certainly NOT a professional programmer. My best day of code production no doubt falls short of your worst. I’m a word guy— a novelist and small press publisher. I’m motivated by the need for a whole suite of web sites and tools to help market my books. As an army of one with insufficient budget to support an IT department or, even, competent professional consulting, I have little choice but to build the tools and systems I need myself. But what with many calls on my limited time and energy, it’s slow going.

OK, so here’s where I stand. At this point I have a reasonably good handle on sequential Erlang. You can judge for yourself by my first GitHub post:



Zpt_gridz is not perfect— it no doubt has many warts and cross-grain outrages against Erlang convention. If you spot such, by all means please let me know so I can mend the error of my ways.

And this brings me to why I’m resurrecting this blog. Zpt_gridz is but one small component in a much more ambitious project buzzing around my brain. But it’s brought me to outer edge of my Erlang competencies. I think I get the ideas and principles of concurrent Erlang. But I’m way fuzzy on how to IMPLEMENT those principles in practice.

So here I ask for help. I need a few generous mentors to help me bring my, perhaps, dunderheaded  ideas  about web development to fruition. I’ll expound on these ideas in subsequent posts. I’m not asking for any big-time commitments. Just a gentle nudge or critique here and there to keep on a productive path.

In exchange for your guidance, I pledge to post resulting applications to GitHub under liberal Open Source license, document my stumbles and triumphs along the way in this blog, and write tutorials on topics not well covered in the Erlang literature.

Back soon— and all the best,











Checklists for Erlang I

February 28, 2010

Dr. Atul Gwande set out to study clinical and surgical errors and how to reduce them.

He concluded that many clinical and surgical procedures are so complex that even experienced specialists can’t keep every essential step in mind during the heat of practice.

The cost?

Clinical and surgical mistakes, patient injury, death, lawsuits, ruined careers, and rising healthcare costs.

What to do?

Dr. Gwande’s research led him back to WWII. Boeing Aircraft had developed the B-17 bomber as one of the most sophisticated, high-performance aircraft ever to come off the boards. But, when three men died as result of an early test-flight accident, many argued that the B-17 was just too demanding for human pilots. Influential congressmen argued that the B-17 program should be scuttled. The survival of Boeing itself was at stake.

But Boeing came up with a very simple solution: A checklist. Boeing implemented a simple checklist, then went on to fly 1.8 million hours with 18 B-17s without incident. Eventually nearly 13,000 were built.

I argue that checklists are an answer to an Erlang noobie’s prayers — particularly in the area of user documentation for web frameworks and other relatively complicated applications.

Noobies trip and fall over a relatively small handful of user doc deficiencies:

1) Assumed knowledge – the docs presume users have knowledge or skills that, in fact, they don’t.

2) Missing steps – the docs omit crucial steps.

3) Confusing language – the docs are poorly written or ambiguous.

4) Missing context – the docs fail to mention crucial hardware, software, or software version prerequisites.

The result:

Noobies waste hours trying to untie the Gordian knot. They turn away in frustration. Fine software fails to gather critical mass community. Erlang gets a bad rap.

Over the next few posts I’ll recount my experience in turning above-average Erlang user documentation into a checklist and the surpising results.

Stay tuned.

Recent comments from Ivan Uemlianin and Marc Worrell have vexed my thinking. Marc says, “We didn’t add ‘full blown social network site’ support (to Zotonic) for one major reason: user support.” And Ivan noted, “…user documentation is hard to write, and — it has to be tested (by a real user).”

So here’s a challenge to the Erlang community: How can we best help our most promising developers improve user documentation of best-of-breed Erlang programs and frameworks?

Do we need:

1) Better standards for user documentation?
2) Better documentation tools?
3) More enlightened attitudes about user documentation?
4) Annual awards for best user documentation?
5) An endowed foundation to support development of better user documentation?
6) None of the above. Let the user flounder.

Incidentally, just for fun, try to find the Erlang standard libraries through Google searches such as: “Erlang + R13B + libraries.” Indeed, try to find the standard libraries from the http://erlang.org/doc.html.

What do you think?

Happy New Year to all.


Continued from December 29:

Q: You say that you “…want (Zotonic) to be a spider in the web of information?” Can you tell us more?

A: Nowadays we have information on, or gather information from, multiple web sites. Think of Twitter, Flickr, and so on. We like to be able to collect all that, personal, information at one place. That one place can then be your own site, a true center of your personal, or organization’s, internet. From that site you are then able to publish to other sites, or the other sites by subscribing to your content.

Q: Given your vision of Zotonic, in what ways is it, or will it be, superior to competitive open-source Content Management Systems?

A: Speed and connectivity.

And this without sacrificing ease of use, extensibility and ease of site development. Our web designer just loves the ease with which he can develop and deploy sites using Zotonic.

Q: Under what circumstances might it be better to choose a Content Management System other than Zotonic?

A: When you can’t install Erlang on your web server.

Or when you want to have a full blown social network site, we don’t do that.

Q: Why did you choose Erlang as language of choice for building Zotonic?

A: Erlang is a simple language with a couple of unique features. First is its ease of building robust software. Next is its incredible support of multiprocessing, it is the only language with support for thousands upon thousands of concurrent processes. And don’t forget that it is an old language, it has already proven to work and be stable.

What was also interesting is that the main purpose of Erlang was to program telephone switches. A telephone switch is not unlike a modern website with its APIs, html pages, Comet connections, XMPP connections etc.

Erlang turns out to be a perfect match for building web sites.

Q: Why did you choose PostgreSQL rather than Mnesia, Riak, or Couchdb as back-end database?

A: Quite simple, we needed a database and not a key-value/document store where we have to program the database part ourselves.

It becomes more clear when you check the requirements for a data store:

1. Easy to use and program against
2. Reliable
3. Data integrity
4. Able to copy a database from test to staging to production (and vice versa)
5. Scales linearly with more cores
6. Fast enough for the majority of sites
7. Easy to query
8. Simple full text search available that is good enough for small(er) sites
9. Multiple independent data stores (for every site one)

And we have customers. When we say “we use PostgreSQL, you can access your data”, then they are very happy. Their technical people know PostgreSQL (and MySQL), they can use it, they know they can always access the data contained in the database.

Don’t forget that our data model is very simple. Most fields are serialized in a blob using erlang’s external term format. Selecting, packing and unpacking a SQL row is very fast because we don’t have many columns.

There are other advantages that come with a RDBMS. Easy to query with SQL, foreign key constraints to keep data integrity, easy to express relations (hey, it is a Relation-DBMS!), triggers and stored procedures. All stuff that makes building a system on top of the data very easy.

Why not MySQL? We had some stability problems, some erroneous queries, scaling issues and saw bad join performance. We never had those issues with PostgreSQL. Besides that there is a great Erlang binary driver for PostgreSQL and only a text driver for MySQL.

Q: Do you see any scalability limitations arising from your choice of PostgreSQL?

Not yet 🙂

At least not until we hit a million documents or so.

RDBMS systems are not that great for big write volumes. Which is one of the reasons why key-value stores are so popular right now.

For read performance there is not really a problem. We do have quite aggressive caching, keeping the data where it is needed instead where it is stored. So we mostly hit the database for queries, not for reads. Testing by an external party showed that during request bursts the database is almost inactive.

The best way to scale is to prevent hitting the database.

Q: Tell us about the core Zotonic development team?

A: We know each other from the company Mediamatic, where I used to be the CTO and (co-)owner. Tim and Arjan still work at Mediamatic. Zotonic and freelance projects is what they do in their non-Mediamatic time.

Tim is a front end engineer and designer. He really brings together an eye for the design and the technical expertise to make it happen with css, javascript and templates. Tim also likes to play music with his band.

Arjan is a back end engineer and artist. He is a very smart guy with a wide field of interest. He builds his own hardware for art projects and is fluent in many programming languages, concepts and ideas.

I know Peet already a long time. He is the guy on the sideline, cheering us up, reflecting what we make with the real world, and making video/pod casts about Zotonic. Audio is really his thing. Besides that he is a software engineer and works as an architect for Mirabeau. He knows what he is talking about.

Q: Tell us about yourself and the experience and strengths you bring to Zotonic?

My own background is quite wide.

I have been making software since 1983 or so. In the mid 90s I worked on development environments for embedded systems, with customers like Intel, Motorola and Siemens. After that on multimedia database systems for news papers and others publishers. The last ten years I have been developing software for the web. Notably building the information management system anyMeta. anyMeta is an inspiration for Zotonic.

My biggest strength is a very wide and extensive experience. Which helps a lot in abstracting problems and prioritizing solutions.

Q: What do you see as the biggest challenge ahead in building Zotonic?

Keeping it focused and the development funded.

Q: How can the Erlang community be of most help in realizing your vision for the future of Zotonic?

Use it. Write about it. Report problems.

And write modules. There are so many features that people want, and only a couple of core people working on it. So when you want something, then check if you can add it and send us an inclusion request.

We used the Apache License to make it possible for everybody to use, extend and enjoy Zotonic. We hope it will also encourage people to give back extensions and fixes to the community.

Well, here I was headed out on sabbatical from Erlang and this blog when Russell Brown and Marc Worrell talked me out of it. Well, at least, talked me into delaying my departure.

The bait? Zotonic–Erlang’s new Content Management System.

I looked. I liked. I started a conversation with Marc Worrell, the creative spark plug firing Zotonic development.

Q: Marc, when you set out to build Zotonic, what was the main problem you were striving to solve?

A: In the past years I was confronted with two developments: real time web with xmpp publish/subscribe and Comet-like connections on the one side and on the other a quick evolution of hardware into multicore CPUs. The technologies we were using did not match both developments.

When checking what could support both the software requirements and hardware developments I re-discovered Erlang.

The basic requirements we set were:

1. Support of Ajax and Comet (Websockets).
2. Can withstand a storm of requests for a small set of pages (known as the slashdot effect).
3. Integration of xmpp: publish/subscribe.
4. Easy templating system for front end designers.
5. Rich and flexible data model, extensible by editors.
6. Easy to maintain and extend by programmers.
7. Efficient enough to serve a fairly large and popular website from a single server.
8. Serve many sites from the same server.

Q: With all of the full-featured open-source Content Management Systems out there, like Django, Joomla, and Drupal, why another?

A: I know the existing frameworks, they are very nice indeed. They solve the problem of generating a html page.

  1. They don’t solve the problem of generating many pages at the same time in an efficient manner.
  2. They don’t solve the problem of sharing information between systems using many interfaces, of being a spider in the web of information.
  3. They can’t stay connected with other services, as is needed for websockets and xmpp.

They didn’t match our requirements.

Let’s take a look at one of the problems, handling multiple requests.

In the usual setup with PHP, Rails or Django you will see a single process handling a single request. When you have 10 simultaneous requests there will be 10 processes. When you have 100 requests there are 100 processes. Each process is quite large, so this will consume a lot of memory. Each process will connect to the database and to the memcache server. Each process will fetch all the information it needs from those sources and start building a html page.

Now imagine that they all handle render the same page or page elements. All those processes are independent so they don’t know about each other. They can’t share the parts they are rendering. They are all doing the same thing at the same time. Isn’t that a shame? Isn’t that wasting valuable resources, time and energy?

By handling all those requests with a single multithreaded process we are able to share parts and information between different requests. That is one of the reasons why Zotonic is fast.

Q: As Zotonic stands now, what kind of sites can be easily built with minimal additional programming?

A: General publication sites, blogs, company inter- and extranets and event sites. Examples are the site we made for the New Island Festival in New York and a site for the Dutch Mediafonds we are now finishing.

Where Zotonic really shines is handling rich and interconnected information. It is possible to define relationships between all pages. And every page can represent anything, a person, a news article, a tweet, a photo, a keyword.

Q: I see that Zotonic has a modular design. Tell us what modules beyond core functionality are production ready now?

A. Modules are independent processes that contain everything for a certain functionality. They can contain css, javascript, templates, resources (controllers), dispatch rules, service apis, template actions and more. They can also start processes, for example to check for incoming tweets or connect to an xmpp server.

The modules we have right now, and which we are using in production, are:

  1. Admin, the administrative environment. This includes some modules that extend the admin for specific functionalities.
  2. Atom feed – generates atom feeds by category or search query
  3. Broadcast – a simple example of pushing messages to all open sessions.
  4. Calendar – a weekly calendar viewer
  5. Development – automatically loads recompiled modules
  6. Emailer – queues, renders and sends e-mail as a background process
  7. Mailinglist – send any page directly from the admin to a mailinglist, schedule mailings and add/remove subscribers
  8. Menu – a simple site menu that can be changed using drag & drop
  9. OAuth
  10. Search – implements different queries that can be used directly from the templates or Erlang code
  11. SEO – search engine optimization, meta tags, Google Analytics, Google Webmaster tools.
  12. Sitemap – generates a sitemap.xml for better indexing of your site
  13. Twitter – follow persons on twitter and import their tweets
  14. Video embed – handles embedding video and audio

Automatic image/pdf manipulation (crop, grayscale, resize etc.) is part of the Zotonic core.

Q: What additonal modules would you hope to see production-ready one year from now? Three years from now?

A: Right now we are working on a couple of modules or core features.

  1. Google Maps
  2. Geo support
  3. Websockets
  4. XMPP Publish/Subscribe
  5. XMPP Chat groups

And we are also working on an example blog site. This will become the default site for a fresh Zotonic install.

For the coming years we are looking into realtime and better publish/subscribe support for sharing content between web sites. We want to be a spider in the web of information, being able to easily share anything on the web with other web sites.

To Be Continued

Fairwell Erlang

December 13, 2009

With considerable regret, this is my last post to this blog — at least, for the immediate future.

My interest in Erlang has in no way diminished. Indeed, in all ways but one, my explorations have more than confirmed initial impressions. But, fact is, time has tracked me down. I must, for now, focus elsewhere.

A year ago I set out to learn Erlang. I created this blog to document my stumbles and progress. I came clean from the beginning — I am not a professional programmer. But I studied diligently. I set a goal — build my own Erlang-based blog. I worked my way through the Getting Started with Erlang User’s Guide. I read Joe Armstrong’s thesis, I studied Joe’s book Programming Erlang: Software for Concurrent World, subscribed to Kevin Smith’s fine screencasts Erlang in Practice. I attended Erlang University in Palo Alto. And spent hours with Cesarini/Thompson’s Erlang Programming and the Early Access Edition of Logan et. al. Erlang and OTP in Action.

I also set out to study the landscape of Erlang web servers and frameworks — installed them on my computer, studied source code, booted them up, explored how to build on them. (See previous posts) Many generous Erlang wizards mentored me along the way, for which I’m deeply grateful.

Now, I can enthusiastically say that the above course of study should have been more than enough to guide any Erlang newbie to journeyman competency. But I fell short. In fact, I stumbled well short of my goal. I managed to code a few mnesia-based modules that enabled me to store and retrieve blog posts. But they were toys — far short of production quality.

Truth is, I had an ulterior motive for learning Erlang. I wanted to build a web platform for supporting my writing career. When I started this blog I was well into my second novel — well, third, if you count my graphic novel Aya Takeo. Creative writing was my first priority. Programming was second. Consequently, while I was diligent in studying Erlang, I spent far too few hours actually programming. Had I imposed the same discipline on programming as I did on writing, I venture that I would have surpassed my goal by now.

But, now, time has tracked me down. Not counting Aya Takeo, my first novel, Freein’ Pancho, is on the verge of publication. I’m into final revision on my second novel, The Gospel of Ashes. And I’m several chapters into El Tiburon.

So, it’s now time to get serious about marketing Aya Takeo and Freein’ Pancho. If the novelist doesn’t do the heavy lifting of marketing these days, few, if any, novels get notice. And, of course, I must get The Gospel of Ashes out the door and keep forward progress on El Tiburon.

Thus, I’ve spent the past week bringing up a marketing site for Aya Takeo. I’ll be bringing up another site soon to market Freein’ Pancho. When published, I’ll also need a site for The Gospel of Ashes. I’d hoped to be able to bring up these sites in Erlang — but I simply couldn’t climb the learning curve fast enough.

The Aya Takeo site took me about a week to build. I used Joomla, a PHP/SQL open-source content management system. As an old Cold Fusion guy, I’d hate to get down and dirty with the Joomla’s PHP code. But, with help of a Joomla developer, I was able to bring up the Aya Takeo site in very short order and minimal cost.

So here’s what I’ve learned about Erlang. It’s a fine language, but not quite ready for guys like me who know just enough to shoot themselves in the foot. My hat is off to the Erlang developers posting increasingly interesting elements of Erlang-based web infrastructure. But until they get more community support and far better user documentation they’ll be sucking the fumes of the Joomlas, Drupals, and Djangos of the world.

There’s still far too big a gap between the fluency of human language and the brittle tyranny of Erlang code.

But when the day comes, and the gap is bridged, it will be Katy bar the door. I hope it comes soon.