OTP and Erlang in Action

Martin Logan is one of the many bright people I met at Erlang Factory Palo Alto.

Martin has been programming Erlang for more than 10 years — full time for the last five. He’s the core developer at Erlware and the primary developer of the Faxien package management system.

During one of the Erlang Factory breaks Martin patiently defined “referential transparency” for me. Fancy phrase for a fairly simple concept it turns out — an expression that always returns the same result given the same input is said to be “referentially transparent.”

Anyway, I recently purchased from Manning Publications the MEAP edition of OTP and Erlang in Action by Martin Logan, Eric Merritt, and Richard Carlson. And it’s sure to be an Erlang classic.

The cool thing about MEAP is that you pop in your credit card number and get back an instant *.pdf download of the book in progress. As the authors add content, you get e-mail notification of updates from Manning. You can order hard copy of the work, of course, but you won’t get it until the formal publication date. But, I say that OTP and Erlang in Action is too important to wait. If you’re striving for professional excellence in Erlang, order both *.pdf and hard copy today.

Joe Armstrong’s Programming Erlang is, of course, the Erlang gold standard. But OTP and Erlang in Action (OTP for short) belongs right beside it. And, arguably, you might find skimming through it before you even crack open Armstrong quite profitable. Most certainly, once you’ve digested Armstrong, you’ll want to read and re-read OTP, and enter code as you go along, until the pages fall off.

OTP and Erlang in Action first leads you through the core concepts of Erlang — processes and concurrency, process communication, fault tolerance, distributed Erlang, functional programming, higher order functions, iteration vs. recursion, single assignment, and pattern matching.

All these fancy shmancy words — sounds like a trudge. But OTP is written in lucid and lively English — just the kind of plain-talk accessible language that tells you that the writers really know their stuff.

Chapter 2 introduces language basics. It’s not nearly as thorough as the Erlang tutorials or Armstrong, but it complements them nicely and, more to the point, sets you up for some dazzling hands-on programming.

To wit, the remaining 11 chapters, guide you through solid OTP production-level code — a telnet server all packaged up as an OTP application and a distributed caching system. This is code you can build and use. Along the way you learn in great detail about OTP packaging and organization, processes and linking, building a production system, distributed Erlang, tracking and managing distributed code, packaging and deployment, hot code loading and upgrading, non-native Erlang distribution, drivers and multi language interfaces — all the good stuff that was left to your imagination in other texts.

As a newbie, I’ve got to say, this book is a real find. Can’t wait for the authors to finish it.

Erlang/OTP Callback Modules

Apologies for my April 19, 2009 post. For one thing, it’s much too dense. And worse, it’s wrong — at least seven of the last eight paragraphs.

Reading Erlang source really shouldn’t be rocket science. But I’m still struggling to follow the flow-of-control through Erlang applications.

One problem, for me, is this concept of “callback module.” I think I understand it, but the more I think about it the less confident I feel. I’ve tried to find a definition in Google, but failed. It seems to be one of those terms that everyone tosses around thinking that everyone else understands it.

But if I’m hazy, chances are there’s another Newbie or two out there scratching his/her head as well.

Problem for this Newbie is that the concept of “callback module” implies that something is going on behind the curtains. If I’m not all that clear about what’s going on back there, how can I be clear about what’s going on in the foreground?

So here’s my Newbie insight for the day. Caution: it may be wrong. If so, I beseech any passing Erlang wizard to set me straight.

Behaviours in Erlang/OTP are functional templates of oft-repeated code patterns, e.g., servers, finite-state machines, events, supervisors, and applications.

Callback modules fill in the slots in these templates to round out functionality for specific purposes. The “behaviour” is the generic, or abstract, part of the functionality. The “callback module” is the means through which the programmer tailors a behavior for specific application.

So how do we find out what’s going on behind the curtain? Erlang/OTP docs should certainly be our first stop:

Gen_Server Behaviour

“The client-server model is characterized by a central server and an arbitrary number of clients. The client-server model is generally used for resource management operations, where several different clients want to share a common resource. The server is responsible for managing this resource.”

Gen_Fsm Behaviour

“A finite state machine, FSM, can be described as a set of relations of the form:

“State(S) x Event(E) -> Actions(A), State(S’)

“These relations are interpreted as meaning:

“If we are in state S and the event E occurs, we should perform the actions A and make a transition to the state S’.”

Gen_Event Behaviour

“In OTP, an event manager is a named object to which events can be sent. An event could be, for example, an error, an alarm or some information that should be logged.”

Supervisor Behaviour

“A supervisor is responsible for starting, stopping and monitoring its child processes. The basic idea of a supervisor is that it should keep its child processes alive by restarting them when necessary.”

Applications

“When we have written code implementing some specific functionality, we might want to make the code into an application, that is a component that can be started and stopped as a unit, and which can be re-used in other systems as well.”

I’ll pick up on iserve again tomorrow; and try to view it from high overhead with these “behaviours” in mind.

Newbie Test: iserve – start-up logic

In our last two posts, March 30 and April 3, we poked into Erlang/OTP conventions, supervisor processes, and application resource files.

Yes, I know, this is supposed to be a Newbie Test. Well, it is. But the opportunity to study how iserve is set up as an Erlang/OTP application is just too good to pass up.

Newbies need to nibble down nifty knowledge at every pass.

iserve’s application resource file, iserve.app, tells us that we must poke into iserve_sup.erl to see how iserve is fired up.

As noted in our March 30 post, you can recognize iserve_sup.erl as a supervisor callback module by its name, *_sup.erl, and the -behviour(supevisor) directive near the top of the iserve_sup.erl module.

iserve_sup.erl exports two functions: start_link/0 and init/1.

Here’s start_link/0.

start_link() -/>
supervisor:start_link({local, ?MODULE}, ?MODULE, []).

supervisor:start_link/3 creates a supervisor process as part of a supervision tree and ensures that the supervisor is linked to the calling process (its supervisor).

Note that the first parameter of supervisor:start_link/3 is of the form {local,Name}.

This means that the supervisor is registered locally as Name using register/2.

register/2 expects a name, Name, and a process identity, P. It associates Name with the port or process identity P.

The second parameter is the name of the callback module, in this case, iserve_sup.erl.

The third parameter is an argument passed to init/1.

So, we fire off iserve:start_link/0. This kicks off a locally registered supervisor process with the name iserve_sup. Our new supervisor now calls init/1 to find out about restart strategy, maximum restart frequency and child processes.

In our case, init/1 starts two new child processes, a supervisors, iserve_server_sup, and a gen_server, iserve_master.

We looked at how supervisor processes are configured in our March 30 post.

Christian kindly provides source comments to help us understand the functions of these two processes:

%% The servers supervisor is for keeping individual servers,
%% adding and removing them dynamically

%% The master is for book keeping all running servers and
%% to hold information about them.

So what do we find when we poke into iserve_server_sup?

Well, my eyes are too blurry to find out. But I’ll try to get back to this when the dust settles. Promise.

Back anon.

Note to wizards: corrections/comments welcomed.

Newbie Test: iserve – application resource file

March 30 Poking Around Erlang touched on iserve’s application resource file, iserve.app.

If you’re an Erlang newbie, it’s worth studying this file closely. Much can be gleaned about the application you’re setting out to understand.

At risk of boring wizards beyond tears, here is iserve.ap in all it’s glory:

{application, iserve,
[{description, "Web Server"},
{vsn, "%ISERVE_VSN%"},
{modules, [ iserve_sup,
iserve_app,
iserve_server,
iserve_socket
]},

{registered, [ iserve_sup]},
{applications, [kernel, stdlib, sasl]},
{mod, {iserve_app, []}},
{env, [{port, 8080}, {callback, iserve_test}]}]}.

Why is this file important and what does it tell us?

The Erlang app man page is a good place to go for answers.

Erlang/OTP conventions call for one application resource file for each application in the system. The file should be named Application.app, where Application is the name of the application. It should be located in the application’s ebin directory.

The application resource file is read by the application controller when an application is loaded/started. It is also used by functions in systools when generating start scripts.

Note that the structure of the iserve’s resource specification is a series of comma-delimited tuples, Each tuple is prefaced with an atom, italicized below.

I’d hoped to have decoder rings made up, but the following will have to do in the interim:

application (name of the application): iserve

description (one-line description of the application): Web Server

version (version of the application): “%ISERVE_VSN%” NOTE: instantiated elsewhere

modules (all modules introduced by the application): iserve_sup, iserve_app,
iserve_server, iserve_socket

registered (all names of registered processes started in this application): iserve_sup

applications (all applications which must be started before this application is allowed to be started): kernel, stdlib, sasl

mod (application callback module and start argument): iserve_app, []

env (configuration parameters used by the application): port, 8080; callback, iserve_test

So, now we know that among all the files in the iserve directories, we must pay closest attention to iserve_server and iserve_socket to understand functionality. We must start up kernel, stdlib, and sasl before we start iserve. To start iserve, we need to call iserve_sup.

Useful overview for one short file. And the good news is that you’ll find a similar file in the ebin directory of every Erlang/OTP application.

Newbie Test: iserve – poking deeper into the code

My March 20 post pointed out how Christian Sunesson’s fork of iserve incorporates OTP system principles.

This must have seemed obvious and a tad tedious to Erlang wizards, but it’s been useful for me as a newbie to stand the rather abstract statements of principle in the official Erlang OTP docs against an actual implementation.

In my experience, you can’t have too many examples to drive an abstract idea home. If I have one generic criticism of the otherwise admirable official Erlang documentation it would be just this: too few code-level examples.

I also found the comparison of iserve source against the OTP docs useful in another way: As a newbie poking into a non-trivial Erlang system, it’s a challenge to know where to start among the plethora of directories, modules, and supporting files as one strives to understand what’s going on.

An understanding of OTP conventions whittles down the challenge considerably.

So, today I’m poking a bit deeper into the mysteries of OTP and iserve. Then, in another post or two I’ll try to lay out my understanding of how iserve works and, maybe, point out a few interesting code phases and idioms that I’ve noticed along the way.

My usual caution: I’m a newbie. If I get something wrong feel free to scold me, but better, please help us rise to better understanding with detailed what and why.

Let’s look first at two files in the iserve src directory: iserve_sup.erl and iserve_server_sup.erl.

When you dive into OTP Design Principles, the first thing you learn is that the supervision tree is a basic concept in Erlang/OTP.

Since you can read as well as I, no need to go too deeply into it here other than to identify iserve_sup.erl and iserve_server_sup.erl as examples of callback functions that tell a more generic OTP supervisor module what to supervise and how. The clue? Note the *_sup.erl module names and the -behaviour(supervisor) directives in each of the modules.

If you open iserve_sup.erl you’ll note that the init/1 function starts and supervises two worker processes, iserve_server_sup.erl and iserve_master.erl.

With guidance from the OTP docs we can further decipher what’s going on:

Here’s specification of the first worker process:

Servers = {servers_sup,
{iserve_server_sup, start_link, [Supervisor]},
permanent, 2000, worker, [iserve_server_sup]},

- servers_sup is the id of the first worker process
- iserve_server_sup, start_link tells the supervisor how start the worker
- Supervisor … frankly, this puzzle me, perhaps you can clear it up for us
- permanent says that the child function should always be restarted
- the integer 2000 is a timeout value.

The OTP docs tells that: “An integer timeout value means that the supervisor tells the child process to terminate by calling exit(Child, shutdown) and then waits for an exit signal back. If no exit signal is received within the specified time, the child process is unconditionally terminated using exit(Child, kill). “

- worker tells the supervising process that the child function is a worker process
- iserve_server_sup is the name of the callback module

Here’s the other child process specified in iserver_sup.erl:

Master = {master,
{iserve_master, start_link, [iserve_server_sup]},
permanent, 2000, worker, [iserve_master]},

See if you can decode it. Now open iserver_server_sup.erl and see if the code doesn’t make more sense.

Now let’s look at iserve.app and iserve_app.erl.

The key to understanding these two files may also be found in the Erlang/OTP docs.

When you check it out, you’ll find that iserve_app.erl is an application callback module which specifies how to start and stop the application and iserve.app is an application resource file which specifies which modules the application consists of and the name of the callback module.

So, between the supervision and application files we have a fine road map for understanding the rest of the application.

But, enough for today. My bride will be home soon and it’s my night to bring up dinner.

Best to all.

Tidbits: Mochiweb… Nitrogen

Mochiweb

In case it slipped by, Jamaal posted a comment the other day that brings to our attention a meaty Mochiweb tutorial:

Design and Analysis of Mochiweb

An English language translation is available.

Unfortunately for this Newbie the garbled machine-translated English is almost as challenging as Mochiweb itself.

My how I would love to see a fluent English-speaking human take on the task of translation, since this tutorial promises much. Even more, I’d love to be able to read the original Chinese.

Most of all, I’d love to see an in-depth English language user guide to Mochiweb that shows us how to build beautiful apps on the Mochiweb foundation.

Some days I fear that Erlang fluency may be as far off for me as mastery of Chinese.

Many thanks for the links Jamaal.

And thanks to refactor for sharing your insights into Mochiweb.

Nitrogen

Nitrogen develop Rusty Klophaus tossed off this provocative comment in the Nitrogen Google Group the other day:

“Overall, Nitrogen is rapidly maturing as a framework. The interfaces
are much less volatile than they were even a few months ago.

“Within the next few month I hope to start labeling releases with major
and minor version numbers. At that point, interfaces would only change
as part of a major release, and the changes will be well documented.”

May the Force be with you, Rusty!

Newbie Test: iserve

In days of yore,* circa 2007, trapexit published A fast web server demonstrating some undocumented Erlang features.

The posting, in form of a code walk-through, is terrific. Much to be learned.

The server, called iserve, promises simplicity and speed.

The author, noted only as Sean in the posting, deserves thumbs up on both counts. The author’s full name, says Torbjorn Tornkvist, is Sean Hinde.

In any event, thank you, Sean.

But there’s only one problem. Sean’s posting is just shy of sufficient detail to enable a newbie, at least this newbie, to bring up iserve and build on it.

Fortunately, Torbjorn Tornkvist took interest and put the code into a working OTP application. Then, Christian Sunessons mirrored Tobbe’s code and, as recently as February 28, 2009, has added patches and various tweaks.

See: http://frihjul.net/iserve

Torbjorn Tornkvist notes in private correspondence, “I would recommend you to use mochiweb. It is similar to iserve but much more developed and battle-proven. But for studies and experiments iserve is ok.”

And that’s what I intend to do.

I’m setting out today, in all newbie humility, to see if I can bring up Christian’s incarnation of iserve to see what I can learn from it.

Turns out, bringing up iserve is a newbie delight:

1) I opened a console to ~/erlang, the directory I use for messing around in erlang.
2) I entered git clone git://github.com/noss/iserve.git
3) From here, I followed the steps in the iserve/README file, which follow.

- cd iserve
- make test
- erl -pa ./test
application:start(sasl).
application:start(iserve).
iserve:add_server(iserve_master, 6464, iserve_test, x).
- Point your browser to http://localhost:6464/

And, by George, it works.

What a joy.

This newbie extends additional thumbs up to Tobbe and Christian. They light the way.

Now, what can I learn from here?

Stay tuned.

Random Tips for Erlang Doc Writers: 1

This from Notesgirl, a professional technical writer:

“…subject matter experts (that’s me, now) sometimes leave out steps because they’re just too darn obvious. Unfortunately, that makes for sucky documentation or training materials or presentations… “

See 12:27 PM post

Sucky Documentation: Who’s to blame?

The more I study Erlang, the more impressed I am with the potential of Erlang as a language and any number of exciting open-source projects written in Erlang.

If you’ve dipped into this blog now and again, however, you’ll note that I’ve been critical of the quality and/or sufficiency of user documentation available in support of various Erlang web servers and frameworks.

I haven’t mentioned it, but I also have trouble understanding how to productively employ hosts of useful-sounding modules and functions in the standard library.

In many areas Erlang documentation is excellent. The folks at Erlang.org deserve a standing ovation. And scores of generous and gifted writers have published and posted invaluable tutorials, reference guides, and other materials to light the way.

But, in my experience, there are still many dark and arcane stretches along the road to Erlang mastery.

My position is this: Erlang is a tool. I want to master it as quickly and painlessly as possible so I can get on with the work of the world.

It could be just me. But I’ll venture the hypothesis that sucky user documentation considerably impedes the progress and productivity of many programmers striving to learn Erlang and, I’d even venture, an embarrassing number of Erlang veterans as well.

So who’s to blame?

My guess is that it’s a cultural thing — not only within the open-source Erlang community, but across the open-source community as a whole.

If so, this raises any number of provocative questions:

Does it matter?

Maybe not, since the problem persists and open-source thrives.

But I say yes because the more wizards we have crafting new tools and applications, talented programmers polishing up the jewels in the Erlang firmament, and enthusiastic newbies contributing new perspectives, insights, and exciting code, the more fun, productivity, and career opportunities for all.

If nothing else, the more vibrant the community, the easier it is to convince management that there are enough talented Erlang programmers out there to support that new project you’d like to propose.

To beef up the ranks, however, we need to project an enticing and friendly environment. Quality user docs go a long way toward achieving that.

What’s at the root of the problem?

In my experience, when we’re talking about significant software tools or ambitious software applications, producing quality user documentation is as difficult, if not more so, as writing the code to begin with. Why? Because you need to know how to install and configure for various platforms, how the interfaces work, how to write good use cases and examples, and how to effectively communicate all that with a population of users with wide ranging needs and competencies.

Creating quality user documentation, in other words, is just plain difficult and time-consuming.

Moreover, when we’re talking open-source, we’re talking for the most part, volunteer effort. Most techies would rather write or play with code than document it.

What do we mean by quality user documentation?

This question, I think, gets close to the heart of the matter.

There doesn’t seem to be much agreement about what constitutes quality user documentation. One sees little discussion of the differences between source documentation, reference documentation, and user documentation.

Nor do we see much effort toward consensual definition or quality standards when it comes to user documentation.

If we don’t know what it is, how can we do it? And “I know when I see it,” I contend, isn’t good enough.

Who’s responsible for creating user documentation?

Frankly, I think it’s asking too much to expect the core programmer of a significant package to produce quality user documentation. It’s a big enough job to conceptualize, implement, test, and maintain a large corpus of code.

Nor do I think we can expect some “documentation fairy,” as one of my readers suggested, to come along and produce good documentation for us.

Then who?

I suggest that it’s the responsibility of everyone of us who choose to adopt or use a body of code to contribute whatever we can toward better user documentation. But we well know, that if everyone has responsibility, then no one has responsibility.

So what can be done?

1. Hammer out a standard and framework for quality user documentation.

We do this for protocols like HTTP and XML. Why can’t we do it for documentation?

2. Produce/adapt/adopt a tool set to automate to the fullest extent possible the documentation task.

3. Publish all user documentation in wiki form so it can be edited/revised/extended by users.

4. Run an annual awards program for excellence in user documentation.

5. Work toward a culture that recognizes the value of quality user documentation, gives credit to those who advance it, encourages individuals to contribute what they can.

Look at it this way:

Imagine an on-line template that outlines in generic sense what a new user needs to know to successfully bring up and use and/or build out from a software tool or application. Now imagine ten, 20, or 30 skilled users taking ten minutes each to plug in the specifics for a given package, including use cases and examples. Let’s say each contibutor devotes 20 to 30 minutes to the task. No big sacrifice anyones part. But just imagine how valuable that would be to all who come behind.

With such a culture we can have our cake and eat it too… Spend most of our hours and days on code, but still have excellent user docs.

Many people seem to feel that Erlang deserves to become a leader in the Web 2.0/multi-core processor world. My contention is that energetic focus on quality user documentation is an essential step toward that goal.

Good folks behind Erlang.org, as well as Joe Armstrong, Kevin Smith, Hughes Waroquier, and others too numerous to name, have pointed the way.

I say, let’s all jump on board. When it comes to user documentation, we get what we deserve.

An Erlang Newbie Salutes Ricky Ho

When you survey the constellation of intellect, you’ll likely observe two notable types of stars. I call them dark stars and brilliant stars.

Dark stars, smart and often accomplished, conserve their knowledge like misers for fear that some interloper may steal their dark light.

I once worked with a dark star video engineer. He wrapped his arms around his small corporate video studio like a penny-pincher guarding his money bags. NO ONE else could touch a switch or control in his studio. NO ONE could even know which black box was patched into which.

Brilliant stars share their knowledge widely. They encourage others to surpass them.

My wife worked with the late Dr. Judah Folkman at Children’s Hospital in Boston. Dr. Folkman, a brilliant surgeon, studied the role of angiogenesis in tumor formation. In early years he confronted considerable scientific skepticism. His theories now promise to end cancer as a mortal disease. He mentored and launched the careers of scores of outstanding scientists. He published widely, sharing credit all around. He gave of himself stintlessly to patients. When he died, he was mourned around the world.

http://en.wikipedia.org/wiki/Judah_Folkman

Dark stars are the legacy of the ancient medeival guilds, creatures of the dark ages.

Science, as we know it, would not be possible without brilliant stars to light the way.

I’ve just discovered a new star to light up significant areas of darkness along my path as a software developer.

Ricky Ho

An hour skimming through the archives of Ricky’s blog, Pragmatic Programming Techniques, is a roller-coaster ride of exciting insights into cutting-edge web technologies.

Interested in the implementation of couchDB? See: http://horicky.blogspot.com/2008/10/couchdb-implementation.html

How about parallelism with Map/Reduce? See: http://horicky.blogspot.com/2008/04/spam-filtering-with-mapreduce.html

Or maybe you just want to explore Erlang with Map/Reduce. See: http://horicky.blogspot.com/2008/06/exploring-erlang-with-mapreduce.html

Ricky is as concise as can be, but communicates challenging concepts as clearly as a newly tuned silver bell.

Check him out.

Thanks, Ricky. This newbie has put your blog near the top of his list.