OTP and Erlang in Action

Martin Logan is one of the many bright people I met at Erlang Factory Palo Alto.

Martin has been programming Erlang for more than 10 years — full time for the last five. He’s the core developer at Erlware and the primary developer of the Faxien package management system.

During one of the Erlang Factory breaks Martin patiently defined “referential transparency” for me. Fancy phrase for a fairly simple concept it turns out — an expression that always returns the same result given the same input is said to be “referentially transparent.”

Anyway, I recently purchased from Manning Publications the MEAP edition of OTP and Erlang in Action by Martin Logan, Eric Merritt, and Richard Carlson. And it’s sure to be an Erlang classic.

The cool thing about MEAP is that you pop in your credit card number and get back an instant *.pdf download of the book in progress. As the authors add content, you get e-mail notification of updates from Manning. You can order hard copy of the work, of course, but you won’t get it until the formal publication date. But, I say that OTP and Erlang in Action is too important to wait. If you’re striving for professional excellence in Erlang, order both *.pdf and hard copy today.

Joe Armstrong’s Programming Erlang is, of course, the Erlang gold standard. But OTP and Erlang in Action (OTP for short) belongs right beside it. And, arguably, you might find skimming through it before you even crack open Armstrong quite profitable. Most certainly, once you’ve digested Armstrong, you’ll want to read and re-read OTP, and enter code as you go along, until the pages fall off.

OTP and Erlang in Action first leads you through the core concepts of Erlang — processes and concurrency, process communication, fault tolerance, distributed Erlang, functional programming, higher order functions, iteration vs. recursion, single assignment, and pattern matching.

All these fancy shmancy words — sounds like a trudge. But OTP is written in lucid and lively English — just the kind of plain-talk accessible language that tells you that the writers really know their stuff.

Chapter 2 introduces language basics. It’s not nearly as thorough as the Erlang tutorials or Armstrong, but it complements them nicely and, more to the point, sets you up for some dazzling hands-on programming.

To wit, the remaining 11 chapters, guide you through solid OTP production-level code — a telnet server all packaged up as an OTP application and a distributed caching system. This is code you can build and use. Along the way you learn in great detail about OTP packaging and organization, processes and linking, building a production system, distributed Erlang, tracking and managing distributed code, packaging and deployment, hot code loading and upgrading, non-native Erlang distribution, drivers and multi language interfaces — all the good stuff that was left to your imagination in other texts.

As a newbie, I’ve got to say, this book is a real find. Can’t wait for the authors to finish it.

Thinking Erlang – 3

Among the many impressive folks I met at Erlang Factory – Palo Alto was Robert Virding. Robert is a certified Erlang guru with scores of important contributions to the language under his belt. You can read Robert’s blog here.

Robert patiently answered my numerous newbie questions and, later, sent me a most thoughtful email. Here’s the gist:

“I think you are right in that one major problem with learning erlang is processes and modules, how they relate to each other.

“Another major problem for many which I will take first is the functional aspect of the language, which usually manifests itself in the lack of global data and immutable data structures. In principle this problem is easy to solve, all you need is practice and after a while you get the hang of it and then it all seems easy, natural and self evident. Everything you know is always easy, natural and self-evident. In practice it can take time to get the hang of it, and I don’t know of any good short cuts which can help. I was used to programming like this from even before Erlang so I honestly can’t remember how I initially reacted to it.

“A similar problem for many is the syntax. I have found from reading various blogs about people’s reaction to the syntax that they fall into 4 groups:

“1. They see and love it immediately.
“2. They find it strange at first but after a while they find it natural and right.
“3. They understand it and can use it but never like it.
“4. They never get it. This is most often accompanied by loud moaning and complaining about how it isn’t like the language they are used to, why can’t it be like that, can’t it be changed …

“Group 1 people are very rare and often have used similar languages before. Group 2 people seem pretty common in the projects which use Erlang, I remember Bob Ippolito said something close to this in a talk he gave when he mentioned the syntax. I can understand group 3 people although I think they are wrong. Group 4 people I just can’t understand at all. Of course it is different, it is a different language with different semantics and everything. Expecting it to be the same is at best naive and at worst plain stupid. The whole point of learning something new is that it is new and different, otherwise why do it? It is like travelling to a another country and complaining that it is different. Sorry for the diatribe but this way of thinking really gets me.

“I think peoples reaction to other aspects, like the functional side, generally falls into the same groups.

“Processes and modules.
“The way I see it modules are about *what* to do and *how* to do it, while processes are about *who* is to do it. As in the real world where you can hand out the same job to many people in erlang you can have many processes running the same code. This is allso a complete rethink for most people. In traditional systems you generally try to avoid creating to many processes/threads and you learn a lot of techniques to get around it.

“Erlang goes the other way and says create as many processes as you need.The question is then, of course, well how many do I *need* and how many *should* I use? The simple answer, which we blithely give, is as many as there are concurrent things in your problem – one process per concurrent thing. How many that is depends on how you structure your problem. This is definitely not trivial! As in real life working out the optimal number of “workers” you should have is difficult. All I can say is to try and restrict each process to doing one thing, this is usually the best, but of course sometimes some tasks just work better if they are together.

“Concurrency is definitely not easy. It also requires clear thinking in what is concurrency and how does it relate to my problem. Many times we have problems in convincing people that there is a fundamental difference in concurrent and non-concurrent objects. Just because you “send” a message to them all doesn’t mean they behave in the same way.

“Callbacks
“Callback can mean different things and can be handled differently. gen_server splits functionality into two separate parts: concurrency and some management which is handled by the library; and the functionality required by the application which is handled by user-code. It is the user code which defines what is to be done. The user code is in a module with a number of predefined functions. When an event occurs, a message arrives, the server will call a user function to handle that event. This function will do what it has to and return a value saying how it went. This is what is meant by a callback in OTP, it is a user function which called at a specific time by a server process. As the servers are generic they must handle user code and data in a very standardized way. So for example the server passes around user state in an argument to user callbacks and updated state in there return values. The server has no knowledge of what is in the state. It is the same with calling a server, the system provides some standardized ways of calling the server, that is sending request data to the server, which is then processed by the user callback and returning a result value from the callback to the original requester. The server knows *how* it is to pass the data around while the user code knows *what* to do with it.

“A final comment about your problems with inets is that you ran into many issues at the same time: what the user code is to do; how to run the OTP application, processes etc.; and how to set up an OTP application, the .app file. The last one is recognized as being tricky. Even for Joe it took a long time to work it out when writing his book, and he was one of the original developers of OTP.

“I am not an OTP expert, though I realize that I will have to learn it someday.”

Many thanks, Robert!

Seeing Erlang Anew

Mea culpa for neglecting this blog so woefully. If memory serves I was on my way to Erlang Factory in Palo Alto when I last posted. That was in… May? I promised to post again upon return from Palo Alto “…as soon as the jet lag wears off.”

Well… here we are in August. Some case of jet lag you say. And you’re right. A bit more than jet lag has distracted me from this blog and, I’m sorry to say, Erlang.

Permit me to share a bit since it does bear on the future directions of this blog.

First off, Jamaal, bless him, asked in a comment if I’d soured on Erlang. And the answer is, absolutely not! Erlang Factory was wonderfully inspirational. It gave me a much deeper sense of the Erlang community. And I was blown away by the many exciting Erlang projects in the making. On the other hand, I boarded the airport shuttle bus for home with a deep sense of humility just thinking about how much I have to learn.

Six or so hours later I stepped off the airplane and into the Boston Logan terminal with all intentions of posting my Erlang Factory encounters and impressions first thing after a good night of sleep. But something more than jet lag and less than flu knocked me out for the better part of a fortnight. I couldn’t string three coherent words together or get out of my own way. Total listlessness.

Then, just as my mitochondria started pumping out energy again I got a call from my computer hardware supplier. My new Linux workstation is ready. Good thing since Cube1, my four-year-old Linux machine, was near death. Definitely doing goofy things. Thus ensued a painful two weeks of reconfiguring Reliance, my new custom built workstation, transferring files, updating my development network, and all the good stuff you go through when you change PCs in midstream. A good eighty percent of my time was spent in man-page hell. I still haven’t figured out how to get my Infrant ReadyNAS network attached storage device to talk with Reliance. So I’m still without automated back-up. And, I must say, the Netgear gurus have been no help at all. Lord I hate computers some times.

So now we’re into the season of my son’s wedding, numerous family graduations, most requiring stressful travel, yada yada.

But the big news is this. Just as things settled down on the family events front my writing muse kicked me into an intense sprint to finish the first draft of my political thriller. Working title: Carmichael. But I’m seriously thinking about calling it God’s Wrath if somebody hasn’t already jumped on it. Waiting now for critical feedback. Then, after a rewrite/revision I’ll start the risky and arduous process of trying to bring it into print. If you’re a good critical reader and you like thrillers, drop me a line.

So now it’s time to start thinking deeply about Erlang again. I need to start working seriously toward building a site to help market my writing. My plan is to set up my own mini-on-line book store with a promotional blog and various other devices to bring in a few shekels for my words. Yes, I know I can use any number of polished open-source and/or proprietary programs to accomplish this. But I want to learn Erlang, dammit! And I always seem to go the long way around the barn. So, please, wish me luck.

The upside is this: I need to focus. Till now I’ve grazed promiscuously about the fields of Erlang. My plan now is to zero in on Nitrogen… see what I can do with it. I’ll need to invest serious programming time, which may mean somewhat less time posting to this blog, well, relative to my pace prior to early May.

But I’ll do my best to share what I learn along the way.

A Gift of Time

Thanks to a fortunate turn of events I’ve been enjoying a gift of time, a sort of sabbatical following many years of hard work and deadline stress.

But what to do when the pressure is off and the agenda wide open?

My first project was goofy on the face of it, but had a serious objective. I wanted the experience of running an on-line store so I wrote a simple little Python web framework and teamed up with a world-class artist, Sonia Leong from over Cambridge, England way, to create a web manga or, if you will, an on-line graphic novel. We’ve faithfully posted a new full-color panel every week for near two years now. The manga has been moderately successful. The webstore less so. You can see it all at http://ayatakeo.com.

Once the demands of Aya.Takeo settled down I again had time on my hands.

My undergraduate degree, all too many decades ago, was in creative writing with a heavy seasoning of social sciences and humanities. My writing skills have served me well over the years, but the necessity of feeding a family gave me all too little time to serve my writing, that is, to embark on any serious extended writing project that wasn’t dictated by a client.

So my second project, still ongoing, was to write a novel, actually my second. I’m pleased with the way it’s been going and see the end in sight… well first draft at least… and it’s given me a kind of inner creative joy that’s only visited me in fits and starts heretofore throughout my career as a software developer.

But if you know anything about the fiction market you’ll know that the chances of selling a novel these days are somewhere south of the odds on winning a trifecta.

And, who knows, maybe someday this gift of time will come to an end.

So some four months ago I set out on a third project… to develop journeyman skills as an Erlang programmer/developer and that, in turn, led to this blog. Fact is, next to writing, I truly love software development. So learning Erlang is anything but an imposition. But how will I use my Erlang skills should I develop sufficient mastery? Who knows. But I do have a bunch of applications rattling around in my head.

I’m sharing all this by way of explanation. My various projects put me in the position of the bounder with three mistresses. If the cad sleeps with one he feels guilt toward the other two. And that’s the way it is with me and my three projects.

Over the past week the novel muse has been particularly insistent, shouting long passages of dialogue and description in my ear. What can one do, but submit.

So, in the case you’ve been wondering why no posts to pockingaroundererlang over the past week… now you know.

But the Erlang muse is jumping up and down at my side. As Rumpole would say, “She who can’t be denied.”

Hope to attend to her soon.

Meanwhile, thanks for listening.

Newbie Tests: “Hello World” Webapp

For simplicity and instant gratification Matthew Foemmel’s inets-based “Hello World” Webapp in Erlang gets a newbie A+.

See:
http://blog.foemmel.com/2008/05/hello-world-webapp-in-erlang.html
http://blog.foemmel.com/2008_05_01_archive.html

I had Matt’s first version up and running in about two minutes flat.

The next question, however, is, “Where do we go from here?”

Surely we’d like to:

Add css styling
Add a additional pages
Build and publish forms
Validate form input
Return requested data
Redirect pages
Run Javascript
Etc.
Etc.

Matt doesn’t leave many bread crumbs. He simply points toward reference docs for inets, httpd, and mod_esi.

OK. I’ll bite. Let’s look into the reference sources to see what we can find.

First, what are they?

Inets: “…provides the most basic API to the clients and servers, that are part of the Inets application, such as start and stop.”

http://erlang.org/doc/apps/inets/index.html

httpd: “…handles HTTP requests as described in RFC 2616 with a few exceptions such as gateway and proxy functionality.”

http://erlang.org/doc/apps/inets/http_server.html

mod_esi: “…defines the API – Erlang Server Interface (ESI). Which is a more efficient way of writing erlang scripts for your Inets web server than writing them as common CGI scripts.”

See:

http://erlang.org/doc/man/mod_esi.html
http://erlang.stacken.kth.se/doc/r8b/lib/inets-2.6.7/doc/html/mod_esi.html
http://www.woodpecker.org.cn:9081/doc/ErLang/KsErlounge/erl-vol9-Inets/Erlang%20Inets.pdf.

It seems that inets sets up the web server, httpd configures it to handle HTTP requests, and mod_esi gives us a tool for writing scripts.

Cool.

We note that Matt’s configuration of httpd calls additional modules. What’s with these?

mod_alias: “Erlang Webserver Server internal API for handling of things such as interaction data exported by the mod_alias module.”

http://erlang.org/doc/man/mod_alias.html

mod_auth: “User authentication using text files, dets or mnesia database.”

http://erlang.org/documentation/doc-5.2/lib/inets-3.0/doc/html/mod_auth.html

mod_actions: “This module runs CGI scripts whenever a file of a certain type or HTTP method (See RFC 1945) is requested.”

http://erlang.org/documentation/doc-5.0.1/lib/inets-2.5.3/doc/html/mod_actions.html

mod_cgi: “…execute vanilla CGI (Common Gateway Interface) scripts in the server.”

http://erlang.org/documentation/doc-5.2/lib/inets-3.0/doc/html/mod_cgi.html

mod_dir: “Basic directory handling.”

http://erlang.org/documentation/doc-5.0.1/lib/inets-2.5.3/doc/html/mod_dir.html

mod_get: “Handle GET requests.”

http://193.144.51.148/~cvarela/docs/erlang/lib/inets-3.0.3/doc/html/mod_get.html

mod_head: Handles HEAD requests to regular files.

http://erlang.org/documentation/doc-5.2/lib/inets-3.0/doc/html/mod_head.html

mod_log: Standard logging using the “Common Logfile Format” and text files.

http://www.erlang-fr.org/erlang.org/lib/inets-3.0/doc/html/mod_log.html

mod_disk_log: Standard logging using the “Common Logfile Format” and disk_log(3).

http://erlang.org/documentation/doc-5.2/lib/inets-3.0/doc/html/mod_disk_log.html

I’ve noted “Hello World” Webapp’s simplicity, but these modules references suggest substantial fire power under the surface if we can only understand more concretely what they do and how to use them.

For all that, it looks like mod_esi holds the keys to the kingdom. It exports the function deliver/2 and the callback function Module:Function/3. It also exports Module:Function/2, but it is deprecated.

But there’s a mystery here: Two differing man pages for mod_esi. What’s with this?

http://erlang.org/doc/man/mod_esi.html
http://erlang.stacken.kth.se/doc/r8b/lib/inets-2.6.7/doc/html/mod_esi.html

It looks like there is much to get the old head around here. But my newbie attention span has expired. Maybe I can get back to this with more mental energy tomorrow.

Just When Things Are Getting Interesting

First, tip of the hat to Philip Robinson for pointing the way out of my confusion at the end of my last post. See his comment.

Second, I’m chomping at the bit here to learn more Erlang, but in the interest of domestic peace I must, let’s emphasize,  MUST  take a day or two to bring order out of the chaos up here in my garret attic.  This, in turn, will required tearing down my development network and building it back up.

If I’m not back on task by tomorrow, please know that I’m buried under a pile of books, papers, dust, tools, various black boxes, and network cables.

In any event, I’ll be back just as quick as I can.

Meanwhile fellow travelers, I’ll do my best to buff up on the fun.damentals of Funs.

And Yet Another Question

One can argue that a blog engine is not a particularly good application for showing off the strengths of Erlang.

OK, then, how can we leverage Erlang’s strengths to turn the blog world on its head?