Newbie Test: iserve – plunging deeper down the start-up sequence

Our recent (March 30, April 3, and April 6) explorations into Christian Sunesson’s fork of iserve taught us much about OTP/Erlang functionality and conventions, including directory structures, supervision, and gen-server functionality.

In the April 6 post we explored how iserve’s start-up logic created two child processes, a supervisor, iserve_server_sup, and a gen_server, iserve_master.

Now we’re in a position to plunge deeper into the heart of iserve. So let’s poke into iserve_server_sup.

There’s repetition of material from earlier posts here. Think of it as cram review before your midterm.

And, all due cautions here: beware possible Newbie misunderstandings or misinterpretation in what follows. As always, I invite corrections.

First, note the -behavior(supervisor) directive. This tells us that iserve_server_sup is an interface to the supervisor pattern in OTP/Erlang.

iserve_server_sup exports four functions: start_link/0, start_link/1, add_server/2, and init/1.

Recall that during the iserve start_up sequence, iserve_sup created a supervisory process registered as iserve_sup. iserve_sup’s initialization process, iserve_sup:init/1, in turn, called iserve_server_sup:start_link/1, passing in iserve_sup’s PID in the variable Supervisor.

The function iserve_server_sup:start_link/1 creates and registers a new second-level supervisory process called iserve_server_sup. The name is defined in the macro SERVER declared in the -define directive at the top of the module source listing.

When the supervisory process iserve_server_sup is created, supervisor:start_link/3 calls init/1, which initializes the new supervisory process.

iserve_server_sup:init/1 first instantiates the variable AChild with a tuple that defines a call to iserve_server, then fires off the message {ok, {{simple_one_for_one,10,300},[AChild]}}.

So who’s waiting with abated breath for this message?

The OTP/Erlang generic supervisor. The message defines a simple_one_for_one restart strategy for iserve_server_sup, with no more than 10 restarts within 300 seconds. AChild defines which child processes iserve_server_sup should start and monitor.

The value of AChild is:

{iserve_server,{iserve_server,start_link,[iserve_master]},
temporary,2000,worker,[iserve_server]}

So how do we interpret all this? We’ve gone through this before, but let’s it again (nothing like repetition to sharpen the Newbie mind):

iserve_server – a name used to identify the child specification by the supervisor.

iserve_server, start_link,[iserve_master] – Start function in form of M,F,A; e.g. iserve_server:start_link(iserve_master).

temporary – restart function.

2000 – Shutdown. The supervisor will tell the child process to terminate by calling exit(Child,shutdown) and then wait for an exit signal with reason shutdown back from the child process. If no exit signal is received within the specified time, 2000 seconds in our case, the child process is unconditionally terminated using exit(Child,kill).

worker – child type

iserve_server – Module; used by the release handler during code replacement to determine which processes are using a certain module. Module is the callback module, if the child process is a supervisor, gen_server or gen_fsm.

(April 22, 2009 — Mea Culpa!: I think I stepped off the track when I wrote the following. Don’t believe a word of it. I’ll revisit as soon as I’m more confident that I know what I’m talking about. LRP)

So it’s looking like the supervisor iserve_server_sup is going to be looking after the gen_server iserve master.

Let’s check it out by poking into iserve_master.

First off, Christian tells us that iserve_master provides “Bookkeeping of running servers.”

Next, the -behavior(gen_server) tells us that it provides the callback functions for gen_server.

I hesitate to plow through the bloody details of gen_server. Been there. Done that. So I won’t. You can read about it here.

For our purposes, the things to note are the internal functions call_add_server and call_del_server.

Now these look interesting, but by now you’re undoubtedly bored to tears and I’m running out of pep.

So let’s pick it up another day.

Web Templates for Erlang: Continued from previous post

“8) If we fore go fancy GUI content editing and styling tools, then this system can be fairly simple to implement; puts no added burden on the one-man-band web site developer.

9) In my experience with systems that conceptually approach this model I’ve seen substantial acceleration of productivity.

So, I argue, divide and conquer! “

Fact is, this is the kind of system I’m working toward. Would welcome Erlang wizard input.

Web Templates for Erlang: A Different Take

Much discussion of web templates on the Google Groups – Nitrogen forum recently.

I stuck in my two cents and generated some interesting discussion.

This is a topic I care a bunch about based on experience and hard knocks.

Today I posted the rationale behind my thinking, thought I’d share it.

Note, I define a “Part” as an encapsulated single-function black box of code that can be configured to display CSS stylized web content. You can think component, applet, mini-application, or what have you.

So here it is:

“I’d venture that the truest thing one can say about templates is, ‘different horses for different courses.’

Page-level, monolithic templates are fine for relatively simple pages.

But I argue that there is a conceptual mismatch between the way many web developers, particularly those on the programming side, see the web and the way many many sites are presented today, indeed, the way many page content developers and designers conceptualize a web page.

On the one hand, the conceptual unit is the ‘page.’ Makes sense, since the page is the basic unit of HTML.

But for well-trained designers the page is simply a frame for a ‘page grid.’ Squint your eyes and look at any magazine, newspaper, or even well-designed brochure, and you’ll see a mathematical “grid” structure, where each panel in the grid is a container for content. True, many designers spread content over two or more panels, but the basic structure holds.

The grid system is a fundamental concept in many graphic design text books. Why? Because it provides a coherent way of organizing disparate elements of content.

Squint at many, if not most, well-designed, high-traffic websites and you’ll see the grid system at work organizing the content. The grid system is like a well-organized closet with shelves and cubby holes — everything has a place and everything is in its place, whereas a page-oriented design is more like one of my wife’s kitchen drawers, everything tossed in a jumble with everything else (I should talk; my closets and drawers are even more disorganized).

Now take a look at amazon.com. I have no idea of how Amazon constructs its pages. Maybe as someone suggested it’s a big hairy content management system. Doesn’t matter to my basic argument.

Call up page source for the Amazon home page. Now squint your eyes and substitute variables for every substantive element; e.g. in your mind, turn the source into a template. Now imagine a programmer giving you a list of variable names for different dynamic content elements and working through the template to match content item with place-holder in your big, long, hairy template file.

I’m sure no one does it quite this way, but you get the idea.

Now take another look at Amazon’s home page. Squint your eyes and you’ll see a simple three-column grid, or “make-up” as a designer would say; e.g. left and right side bars surrounding a main content panel.

Look more closely at the top of the grid and you’ll see three content elements (personalization line, personal menu, and search bar) that span the content column and right side bar. They’re still consistent with the basic grid, but they add visual diversity and interest while keeping content well organized.

Further note that the basic columns contain simple lists of content elements stacked one atop another.

Now, look at the variety of content elements that fill the panels in the grid and think about the code, HTML formating, and CSS styling tags needed to create each element. You’ll note that some elements are straight text, some are text and graphics, some made up of hyper links, some form elements, some buttons, and some report data drawn up from a database.

When I talk about ‘Parts,’ this is essentially what I’m talking about. Content elements can be abstracted into a set of ‘Parts,’ distinguished by underlying logic.

Now imagine a page construction process that goes like this:

1) Programmers provide an inventory of Parts, single-function black boxes of code that can be configured with content and styling to generate content elements.

2) Select the Parts you need to convey the information you want. Enter content for each.

3) Design (or select from an inventory) a page grid, essentially a list of DIVs (see my earlier post for example).

4) Develop (or select from inventory) CSS file(s) for your page

5) Plug parts into the page grid

6) Tweak styles in each part if necessary to maintain stylistic consistency with page.

A system like this:

1) Is conceptually consistent with the way content developers and trained designers think

2) Does not necessarily require the page designer to adopt any additional tools beyond HTML and CSS.

3) Breaks down a potentially confusing task into a set of relatively simple tasks.

4) Content and styling can be configured with a simple text editor on the one hand, or sophisticated GUI tools on the other. Indeed, GUI tools could be an additional layer on top of the simpler system.

If the base system were open-source, then specialized parts and GUI systems could add value to meet specialized requirements or support a variety of business models.
E.g., Parts could be open-source or proprietary; offering the best of both worlds.

5) Encourages substantial code reuse.

6) Pushes code complexity down into the Parts. With careful factoring, code complexity may never even be an issue but for the rarest and most algorithmically challenging Parts.

7) Offers great flexibility. Change the page grid and/or the high-level CSS file and you change the basic look of the page. Reuse content elements across pages. Change a content element with little to no interaction with other content elements.”

(Getting too long here… continued next post)

Newbie Test: iserve – Oh oh… Beware!

Just spotted this on Christian Suneson’s blog…

“I probably broke iserve build-simplicity tonight.”

Don’t know if Christian has fixed it yet. But if you have trouble bringing up iserve, it’s a good guess that Christian’s on the case.

Newbie Test: iserve – start-up logic

In our last two posts, March 30 and April 3, we poked into Erlang/OTP conventions, supervisor processes, and application resource files.

Yes, I know, this is supposed to be a Newbie Test. Well, it is. But the opportunity to study how iserve is set up as an Erlang/OTP application is just too good to pass up.

Newbies need to nibble down nifty knowledge at every pass.

iserve’s application resource file, iserve.app, tells us that we must poke into iserve_sup.erl to see how iserve is fired up.

As noted in our March 30 post, you can recognize iserve_sup.erl as a supervisor callback module by its name, *_sup.erl, and the -behviour(supevisor) directive near the top of the iserve_sup.erl module.

iserve_sup.erl exports two functions: start_link/0 and init/1.

Here’s start_link/0.

start_link() -/>
supervisor:start_link({local, ?MODULE}, ?MODULE, []).

supervisor:start_link/3 creates a supervisor process as part of a supervision tree and ensures that the supervisor is linked to the calling process (its supervisor).

Note that the first parameter of supervisor:start_link/3 is of the form {local,Name}.

This means that the supervisor is registered locally as Name using register/2.

register/2 expects a name, Name, and a process identity, P. It associates Name with the port or process identity P.

The second parameter is the name of the callback module, in this case, iserve_sup.erl.

The third parameter is an argument passed to init/1.

So, we fire off iserve:start_link/0. This kicks off a locally registered supervisor process with the name iserve_sup. Our new supervisor now calls init/1 to find out about restart strategy, maximum restart frequency and child processes.

In our case, init/1 starts two new child processes, a supervisors, iserve_server_sup, and a gen_server, iserve_master.

We looked at how supervisor processes are configured in our March 30 post.

Christian kindly provides source comments to help us understand the functions of these two processes:

%% The servers supervisor is for keeping individual servers,
%% adding and removing them dynamically

%% The master is for book keeping all running servers and
%% to hold information about them.

So what do we find when we poke into iserve_server_sup?

Well, my eyes are too blurry to find out. But I’ll try to get back to this when the dust settles. Promise.

Back anon.

Note to wizards: corrections/comments welcomed.

Newbie Test: iserve – application resource file

March 30 Poking Around Erlang touched on iserve’s application resource file, iserve.app.

If you’re an Erlang newbie, it’s worth studying this file closely. Much can be gleaned about the application you’re setting out to understand.

At risk of boring wizards beyond tears, here is iserve.ap in all it’s glory:

{application, iserve,
[{description, "Web Server"},
{vsn, "%ISERVE_VSN%"},
{modules, [ iserve_sup,
iserve_app,
iserve_server,
iserve_socket
]},

{registered, [ iserve_sup]},
{applications, [kernel, stdlib, sasl]},
{mod, {iserve_app, []}},
{env, [{port, 8080}, {callback, iserve_test}]}]}.

Why is this file important and what does it tell us?

The Erlang app man page is a good place to go for answers.

Erlang/OTP conventions call for one application resource file for each application in the system. The file should be named Application.app, where Application is the name of the application. It should be located in the application’s ebin directory.

The application resource file is read by the application controller when an application is loaded/started. It is also used by functions in systools when generating start scripts.

Note that the structure of the iserve’s resource specification is a series of comma-delimited tuples, Each tuple is prefaced with an atom, italicized below.

I’d hoped to have decoder rings made up, but the following will have to do in the interim:

application (name of the application): iserve

description (one-line description of the application): Web Server

version (version of the application): “%ISERVE_VSN%” NOTE: instantiated elsewhere

modules (all modules introduced by the application): iserve_sup, iserve_app,
iserve_server, iserve_socket

registered (all names of registered processes started in this application): iserve_sup

applications (all applications which must be started before this application is allowed to be started): kernel, stdlib, sasl

mod (application callback module and start argument): iserve_app, []

env (configuration parameters used by the application): port, 8080; callback, iserve_test

So, now we know that among all the files in the iserve directories, we must pay closest attention to iserve_server and iserve_socket to understand functionality. We must start up kernel, stdlib, and sasl before we start iserve. To start iserve, we need to call iserve_sup.

Useful overview for one short file. And the good news is that you’ll find a similar file in the ebin directory of every Erlang/OTP application.

Newbie Test: iserve – poking deeper into the code

My March 20 post pointed out how Christian Sunesson’s fork of iserve incorporates OTP system principles.

This must have seemed obvious and a tad tedious to Erlang wizards, but it’s been useful for me as a newbie to stand the rather abstract statements of principle in the official Erlang OTP docs against an actual implementation.

In my experience, you can’t have too many examples to drive an abstract idea home. If I have one generic criticism of the otherwise admirable official Erlang documentation it would be just this: too few code-level examples.

I also found the comparison of iserve source against the OTP docs useful in another way: As a newbie poking into a non-trivial Erlang system, it’s a challenge to know where to start among the plethora of directories, modules, and supporting files as one strives to understand what’s going on.

An understanding of OTP conventions whittles down the challenge considerably.

So, today I’m poking a bit deeper into the mysteries of OTP and iserve. Then, in another post or two I’ll try to lay out my understanding of how iserve works and, maybe, point out a few interesting code phases and idioms that I’ve noticed along the way.

My usual caution: I’m a newbie. If I get something wrong feel free to scold me, but better, please help us rise to better understanding with detailed what and why.

Let’s look first at two files in the iserve src directory: iserve_sup.erl and iserve_server_sup.erl.

When you dive into OTP Design Principles, the first thing you learn is that the supervision tree is a basic concept in Erlang/OTP.

Since you can read as well as I, no need to go too deeply into it here other than to identify iserve_sup.erl and iserve_server_sup.erl as examples of callback functions that tell a more generic OTP supervisor module what to supervise and how. The clue? Note the *_sup.erl module names and the -behaviour(supervisor) directives in each of the modules.

If you open iserve_sup.erl you’ll note that the init/1 function starts and supervises two worker processes, iserve_server_sup.erl and iserve_master.erl.

With guidance from the OTP docs we can further decipher what’s going on:

Here’s specification of the first worker process:

Servers = {servers_sup,
{iserve_server_sup, start_link, [Supervisor]},
permanent, 2000, worker, [iserve_server_sup]},

- servers_sup is the id of the first worker process
- iserve_server_sup, start_link tells the supervisor how start the worker
- Supervisor … frankly, this puzzle me, perhaps you can clear it up for us
- permanent says that the child function should always be restarted
- the integer 2000 is a timeout value.

The OTP docs tells that: “An integer timeout value means that the supervisor tells the child process to terminate by calling exit(Child, shutdown) and then waits for an exit signal back. If no exit signal is received within the specified time, the child process is unconditionally terminated using exit(Child, kill). “

- worker tells the supervising process that the child function is a worker process
- iserve_server_sup is the name of the callback module

Here’s the other child process specified in iserver_sup.erl:

Master = {master,
{iserve_master, start_link, [iserve_server_sup]},
permanent, 2000, worker, [iserve_master]},

See if you can decode it. Now open iserver_server_sup.erl and see if the code doesn’t make more sense.

Now let’s look at iserve.app and iserve_app.erl.

The key to understanding these two files may also be found in the Erlang/OTP docs.

When you check it out, you’ll find that iserve_app.erl is an application callback module which specifies how to start and stop the application and iserve.app is an application resource file which specifies which modules the application consists of and the name of the callback module.

So, between the supervision and application files we have a fine road map for understanding the rest of the application.

But, enough for today. My bride will be home soon and it’s my night to bring up dinner.

Best to all.

Tidbits: Webmachine, Ngnix, and More

Webmachine

Thanks to alert correspondent Jamaal we’ve just learned that Justin Sheehy has released version 1.0 of Webmachine.

Justin positions Webmachine as “an application layer that adds HTTP semantic awareness on top of the excellent bit-pushing and HTTP syntax-management provided by mochiweb, and provides a simple and clean way to connect that to your application’s behavior.”

He further notes “From the way that Webmachine’s decision core works, it follows that Webmachine’s HTTP behavior is transactional. Each HTTP Request is fully received, and the resulting HTTP Response is then fully constructed before being returned. This means that while Webmachine is suitable for a great many Web applications it is not a good fit for an application that will gradually or continuously stream responses back to clients inside the context of a single HTTP Request.”

Jamaal is smitten by Webmachine, finds it easy to work with. He’s promised notes on his experiences to date; we’re eagerly looking forward to them.

I’m thinking about moving Webmachine up the list as a Newbie Test candidate, but first need to finish up explorations of iserve.

Erlang and Ngnix and ???

Nick Gerakines is experimenting with a C/C++ Ngnix module as a front-end to Erlang.

JLouse says “Pre-process the data in other languages and make them into erlang terms which can be read fast by erlang.”

And adds “… don’t get bitten by the mnesia bug, please. It is almost always the case that mnesia and ETS are the wrong way to go.”

As a Newbie, one the attractions of Erlang was the impression that I didn’t have to pile heterogeneous software environments atop one another to get a job done. But these writers, and others, suggest that Erlang is not up to my hopes and expectations.

So what gives? What’s a Newbie to make of this? Is Erlang:

1) Immature? Give it time and it’ll deliver on the promise.
2) Incomplete? Give it time and it’ll embrace ever more functionality.
3) Specialized? It is what is; can’t be all things to all people; live with it.
4) OK as is? These guys may have a point, but by and large it’s sour grapes.

Love to hear from the gray-beard wizard contingent on this topic.

Best Practices

While we’re on the subject of Nick Gerakines, bookmark his list of Erlang Library Best Practices.

Can’t tell you how much this Newbie appreciates words of wisdom from the wise.

Formatting Dates in Erlang

Here’s another one to bookmark… a handy little function for formatting dates.

Newbie Test: iserve – the code

A newbie could do worse then spend a few days studying iserve source code.

This newbie is not wise enough to second guess Sean Hinde, Torbjörn Törnkvist, and Christian Sunesson on code quality. But taken as a whole, the iserve project provides a non-trivial model of web server construction. Of perhaps greater newbie value, it demonstrates Erlang and Erlang OTP conventions that every Erlang newbie should understand and emulate.

One could start with Sean Hinde’s original www.trapexit.org posting. But, as a first step, review of Christian Sunesson’s updated source side-by-side with the OTP system and design principles documentation may well be more productive.

Erlang OTP System Principles Section 1.5 lists the file types defined in Erlang OTP. An understanding of the form and function of these file types goes far toward understanding any Erlang OTP project.

Section 7.4 of the Erlang OTP Design Principles documentation suggests a directory structure for OTP applications. This structure makes it easier to make your way around OTP project source listings.

If you download and compile the latest version of iserve, you’ll note the Christian follows the OTP conventions.

We have an ebin directory plump with *.beam object files created during the compilation process.

Ignore the *.beam files in ebin, however, and you note the iserve.app file. This is an Application Resource File described in OTP Design Principles Section 7.3. In this file you’ll find the name of the application, a short description, version, modules, and other useful information.

In the include directory we find iserve.hrl file. See Section 7.1 of the Erlang Reference Manual for include file details. If you open iserve.hrl, you’ll find a record structure that defines a connection between a browser and a server. Sean provides more details.

For more on records, check Section 8 of the Erlang Reference Manual.

We peek now in the src directory and see a handful of *.erl Erlang source files.

But, hold the phone! There’s a second iserve.app file in src. We open it to find that it’s identical to the iserve.app file in the include directory. Why this should be and which is operative… beats me. Maybe one of our wizards can explain.

Among the *.erl files we find two supervision files, iserve_server_sup.erl and iserve_sup.erl. The structure and function of supervision files is documented in Section 5 of OTP Design Principles. Again, beats me why we need two. Is our wizard still around?

Another really really important thing to note can be found in iserve_sup.erl and iserve_server.erl. Grok in on the lines -behavior(xxxx). These lines tell you that you’re looking at callback modules that export pre-defined functions to to implement a common code pattern, in our case, supervision and server functionality.

There’s more to learn here, but the shadows are long and I’m headed out to the movies with my love.

Back soon fair winds willing.

Tidbits: Mochiweb… Nitrogen

Mochiweb

In case it slipped by, Jamaal posted a comment the other day that brings to our attention a meaty Mochiweb tutorial:

Design and Analysis of Mochiweb

An English language translation is available.

Unfortunately for this Newbie the garbled machine-translated English is almost as challenging as Mochiweb itself.

My how I would love to see a fluent English-speaking human take on the task of translation, since this tutorial promises much. Even more, I’d love to be able to read the original Chinese.

Most of all, I’d love to see an in-depth English language user guide to Mochiweb that shows us how to build beautiful apps on the Mochiweb foundation.

Some days I fear that Erlang fluency may be as far off for me as mastery of Chinese.

Many thanks for the links Jamaal.

And thanks to refactor for sharing your insights into Mochiweb.

Nitrogen

Nitrogen develop Rusty Klophaus tossed off this provocative comment in the Nitrogen Google Group the other day:

“Overall, Nitrogen is rapidly maturing as a framework. The interfaces
are much less volatile than they were even a few months ago.

“Within the next few month I hope to start labeling releases with major
and minor version numbers. At that point, interfaces would only change
as part of a major release, and the changes will be well documented.”

May the Force be with you, Rusty!