My Day of Terror — Compiling Erlang on Lenny

Much encouraged by kind wizards (see Erlang — One Step Forward; Two Back; August 28, 2009; comments) I I set off this morning to compile Erlang on Reliance, my Debian Lenny workstation.

I keep trying to train myself to keep careful notes when I venture into new territory. But inevitably I get caught up in the moment and, in the end, am inevitably caught short with substantially useless historic records. Anyone else have this problem?

Just so, I’ll try to do my best to recount my experience from memory and what scanty notes that have made it through the ordeal — er, experience.

Going in, I had already copied the Erlang source into /usr/local/src and unpacked it into otp_src_R13B0.

So, first thing this morning, I followed Jamaal’s advice — just ran ./configure with no options. Scanning through the feedback I noted several packages missing: curses library, flex, lex, yywrap, odbc library, and something about a missing wx driver.

Grantmichaels implied that I could solve dependency problems with two apt commands:

(sudo) apt-get install sun-java6-jdk -y
(sudo) apt-get build-dep erlang -y

Tried the first one. System came back with:

Couldn’t find any package whose name or description matched “sun-java6-jdk”
No packages will be installed, upgraded, or removed.

Googled Debian + sun-java6-jdk — found that there was indeed such a package for Debian Lenny, though non-free. Tried to load it again, but no dice. No doubt my aging brain is missing something here.

Nevertheless I skipped on to the apt-get build-dep erlang -y step. This seemed to do a bunch of stuff that whizzed up my screen faster than a speeding bullet. So, I tried ./configure once more.

Now the system tells me I’m missing a Java compiler (can’t remember the exact note) and, again, something about a missing wx driver.

Decided to look into this wxWidgets business. Chaining through links I came to Installing xwWidgets and wxPython on Ubuntu or Debian. Followed the directions and, once again, tried ./configure — and wx driver is still among the missing.

At this point irrationality set in. Let’s go for broke. I entered make then make install and let her rip. Took awhile all the while various warnings and unknowns flashing up my screen so fast I can barely catch them in flight. Yes, I know that I’ve failed to account for a Java compiler, curses library, flex, lex, yywrap, and odbc library. No doubt my foolhardy omissions will haunt me to the end of days.

When my bash command returned, just for the hell of it, I entered erl.

And it worked!

Now, I don’t know if I have a crippled Erlang environment or not. I don’t know if or when it’ll come back and kick me in the teeth. But the Erlang shell came up and I could enter 2 + 2. and get 5.

Now, if I knew how, I’d uninstall Erlang and start over a bit more methodically. But I don’t know how to uninstall locally compiled code. So, for now, I’ll live with it.

Oh how I wish I had the skill and environment to back port Erlang R13 to Lenny. It would make it so much easier for future newbies in my position.

And thanks to the wizards for your encouragement. All dunderhead mistakes above are strictly of my own doing.

:

OTP and Erlang in Action

Martin Logan is one of the many bright people I met at Erlang Factory Palo Alto.

Martin has been programming Erlang for more than 10 years — full time for the last five. He’s the core developer at Erlware and the primary developer of the Faxien package management system.

During one of the Erlang Factory breaks Martin patiently defined “referential transparency” for me. Fancy phrase for a fairly simple concept it turns out — an expression that always returns the same result given the same input is said to be “referentially transparent.”

Anyway, I recently purchased from Manning Publications the MEAP edition of OTP and Erlang in Action by Martin Logan, Eric Merritt, and Richard Carlson. And it’s sure to be an Erlang classic.

The cool thing about MEAP is that you pop in your credit card number and get back an instant *.pdf download of the book in progress. As the authors add content, you get e-mail notification of updates from Manning. You can order hard copy of the work, of course, but you won’t get it until the formal publication date. But, I say that OTP and Erlang in Action is too important to wait. If you’re striving for professional excellence in Erlang, order both *.pdf and hard copy today.

Joe Armstrong’s Programming Erlang is, of course, the Erlang gold standard. But OTP and Erlang in Action (OTP for short) belongs right beside it. And, arguably, you might find skimming through it before you even crack open Armstrong quite profitable. Most certainly, once you’ve digested Armstrong, you’ll want to read and re-read OTP, and enter code as you go along, until the pages fall off.

OTP and Erlang in Action first leads you through the core concepts of Erlang — processes and concurrency, process communication, fault tolerance, distributed Erlang, functional programming, higher order functions, iteration vs. recursion, single assignment, and pattern matching.

All these fancy shmancy words — sounds like a trudge. But OTP is written in lucid and lively English — just the kind of plain-talk accessible language that tells you that the writers really know their stuff.

Chapter 2 introduces language basics. It’s not nearly as thorough as the Erlang tutorials or Armstrong, but it complements them nicely and, more to the point, sets you up for some dazzling hands-on programming.

To wit, the remaining 11 chapters, guide you through solid OTP production-level code — a telnet server all packaged up as an OTP application and a distributed caching system. This is code you can build and use. Along the way you learn in great detail about OTP packaging and organization, processes and linking, building a production system, distributed Erlang, tracking and managing distributed code, packaging and deployment, hot code loading and upgrading, non-native Erlang distribution, drivers and multi language interfaces — all the good stuff that was left to your imagination in other texts.

As a newbie, I’ve got to say, this book is a real find. Can’t wait for the authors to finish it.

Erlang/OTP Callback Modules II

Hynek (Pichi) Vychodil kindly pointed me to Joe Armstrong’s thesis to help deepen my understanding of callback modules.

Starting on page 98, Joe talks about “abstracting out concurrency.”

Keeping track of many concurrent process is mentally challenging and error-prone, he says. For this reason he advocates a library of generic components, written by skilled Erlang programmers, and “plugins,” written by programmers competent in Erlang sequential programming. This component library, of course, has become OTP.

Says Joe, “The generic component should hide details of concurrency and mechanisms for fault-tolerance from the plugins. The plugins should be written using only sequential code with well-defined types.”

My quick reading did not turn up the phrase “callback module.” May be there. But I take it that “callback module” and “plugin” are synonymous.

Check out Joe’s thesis for yourself. Tons of invaluable background straight from the man himself.

One crucial point still clouds my aging brain: When you call a function in, say, a gen_server callback module and, say, it returns a tuple like {ok, yada yada}, who’s listening and expected to respond? I definitely need to think this one through before I get back to my walk-through of iserve.

Meanwhile, I’m packing up for Erlang University. Plan to spend three days learning what I can from Chris Anderson and Jan Lehnard about CouchDB.

The following two days, Thursday and Friday, promise many hours of brain food. I’m particularly looking forward to Rusty Klophaus’s Nitrogen presentation. If you haven’t been following Google Groups Nitrogen Web Framework for Erlang, drop in. Lively exchange of performance profiling data has been particularly interesting.

While rubbing shoulders with Erlang greats at Erlang University, I also hope to learn much more about Web Machine and Erlyweb.

Can’t promise, but maybe I’ll come back with a few pearls of wisdom. Maybe even a photo or two. I certainly plan to get back to my dissection of iserve.

So, be back sometime in early May just as soon as the jet lag wears off.

Erlang/OTP Callback Modules

Apologies for my April 19, 2009 post. For one thing, it’s much too dense. And worse, it’s wrong — at least seven of the last eight paragraphs.

Reading Erlang source really shouldn’t be rocket science. But I’m still struggling to follow the flow-of-control through Erlang applications.

One problem, for me, is this concept of “callback module.” I think I understand it, but the more I think about it the less confident I feel. I’ve tried to find a definition in Google, but failed. It seems to be one of those terms that everyone tosses around thinking that everyone else understands it.

But if I’m hazy, chances are there’s another Newbie or two out there scratching his/her head as well.

Problem for this Newbie is that the concept of “callback module” implies that something is going on behind the curtains. If I’m not all that clear about what’s going on back there, how can I be clear about what’s going on in the foreground?

So here’s my Newbie insight for the day. Caution: it may be wrong. If so, I beseech any passing Erlang wizard to set me straight.

Behaviours in Erlang/OTP are functional templates of oft-repeated code patterns, e.g., servers, finite-state machines, events, supervisors, and applications.

Callback modules fill in the slots in these templates to round out functionality for specific purposes. The “behaviour” is the generic, or abstract, part of the functionality. The “callback module” is the means through which the programmer tailors a behavior for specific application.

So how do we find out what’s going on behind the curtain? Erlang/OTP docs should certainly be our first stop:

Gen_Server Behaviour

“The client-server model is characterized by a central server and an arbitrary number of clients. The client-server model is generally used for resource management operations, where several different clients want to share a common resource. The server is responsible for managing this resource.”

Gen_Fsm Behaviour

“A finite state machine, FSM, can be described as a set of relations of the form:

“State(S) x Event(E) -> Actions(A), State(S’)

“These relations are interpreted as meaning:

“If we are in state S and the event E occurs, we should perform the actions A and make a transition to the state S’.”

Gen_Event Behaviour

“In OTP, an event manager is a named object to which events can be sent. An event could be, for example, an error, an alarm or some information that should be logged.”

Supervisor Behaviour

“A supervisor is responsible for starting, stopping and monitoring its child processes. The basic idea of a supervisor is that it should keep its child processes alive by restarting them when necessary.”

Applications

“When we have written code implementing some specific functionality, we might want to make the code into an application, that is a component that can be started and stopped as a unit, and which can be re-used in other systems as well.”

I’ll pick up on iserve again tomorrow; and try to view it from high overhead with these “behaviours” in mind.

Newbie Test: iserve – application resource file

March 30 Poking Around Erlang touched on iserve’s application resource file, iserve.app.

If you’re an Erlang newbie, it’s worth studying this file closely. Much can be gleaned about the application you’re setting out to understand.

At risk of boring wizards beyond tears, here is iserve.ap in all it’s glory:

{application, iserve,
[{description, "Web Server"},
{vsn, "%ISERVE_VSN%"},
{modules, [ iserve_sup,
iserve_app,
iserve_server,
iserve_socket
]},

{registered, [ iserve_sup]},
{applications, [kernel, stdlib, sasl]},
{mod, {iserve_app, []}},
{env, [{port, 8080}, {callback, iserve_test}]}]}.

Why is this file important and what does it tell us?

The Erlang app man page is a good place to go for answers.

Erlang/OTP conventions call for one application resource file for each application in the system. The file should be named Application.app, where Application is the name of the application. It should be located in the application’s ebin directory.

The application resource file is read by the application controller when an application is loaded/started. It is also used by functions in systools when generating start scripts.

Note that the structure of the iserve’s resource specification is a series of comma-delimited tuples, Each tuple is prefaced with an atom, italicized below.

I’d hoped to have decoder rings made up, but the following will have to do in the interim:

application (name of the application): iserve

description (one-line description of the application): Web Server

version (version of the application): “%ISERVE_VSN%” NOTE: instantiated elsewhere

modules (all modules introduced by the application): iserve_sup, iserve_app,
iserve_server, iserve_socket

registered (all names of registered processes started in this application): iserve_sup

applications (all applications which must be started before this application is allowed to be started): kernel, stdlib, sasl

mod (application callback module and start argument): iserve_app, []

env (configuration parameters used by the application): port, 8080; callback, iserve_test

So, now we know that among all the files in the iserve directories, we must pay closest attention to iserve_server and iserve_socket to understand functionality. We must start up kernel, stdlib, and sasl before we start iserve. To start iserve, we need to call iserve_sup.

Useful overview for one short file. And the good news is that you’ll find a similar file in the ebin directory of every Erlang/OTP application.

Tidbits: Webmachine, Ngnix, and More

Webmachine

Thanks to alert correspondent Jamaal we’ve just learned that Justin Sheehy has released version 1.0 of Webmachine.

Justin positions Webmachine as “an application layer that adds HTTP semantic awareness on top of the excellent bit-pushing and HTTP syntax-management provided by mochiweb, and provides a simple and clean way to connect that to your application’s behavior.”

He further notes “From the way that Webmachine’s decision core works, it follows that Webmachine’s HTTP behavior is transactional. Each HTTP Request is fully received, and the resulting HTTP Response is then fully constructed before being returned. This means that while Webmachine is suitable for a great many Web applications it is not a good fit for an application that will gradually or continuously stream responses back to clients inside the context of a single HTTP Request.”

Jamaal is smitten by Webmachine, finds it easy to work with. He’s promised notes on his experiences to date; we’re eagerly looking forward to them.

I’m thinking about moving Webmachine up the list as a Newbie Test candidate, but first need to finish up explorations of iserve.

Erlang and Ngnix and ???

Nick Gerakines is experimenting with a C/C++ Ngnix module as a front-end to Erlang.

JLouse says “Pre-process the data in other languages and make them into erlang terms which can be read fast by erlang.”

And adds “… don’t get bitten by the mnesia bug, please. It is almost always the case that mnesia and ETS are the wrong way to go.”

As a Newbie, one the attractions of Erlang was the impression that I didn’t have to pile heterogeneous software environments atop one another to get a job done. But these writers, and others, suggest that Erlang is not up to my hopes and expectations.

So what gives? What’s a Newbie to make of this? Is Erlang:

1) Immature? Give it time and it’ll deliver on the promise.
2) Incomplete? Give it time and it’ll embrace ever more functionality.
3) Specialized? It is what is; can’t be all things to all people; live with it.
4) OK as is? These guys may have a point, but by and large it’s sour grapes.

Love to hear from the gray-beard wizard contingent on this topic.

Best Practices

While we’re on the subject of Nick Gerakines, bookmark his list of Erlang Library Best Practices.

Can’t tell you how much this Newbie appreciates words of wisdom from the wise.

Formatting Dates in Erlang

Here’s another one to bookmark… a handy little function for formatting dates.

Sucky Documentation: Who’s to blame?

The more I study Erlang, the more impressed I am with the potential of Erlang as a language and any number of exciting open-source projects written in Erlang.

If you’ve dipped into this blog now and again, however, you’ll note that I’ve been critical of the quality and/or sufficiency of user documentation available in support of various Erlang web servers and frameworks.

I haven’t mentioned it, but I also have trouble understanding how to productively employ hosts of useful-sounding modules and functions in the standard library.

In many areas Erlang documentation is excellent. The folks at Erlang.org deserve a standing ovation. And scores of generous and gifted writers have published and posted invaluable tutorials, reference guides, and other materials to light the way.

But, in my experience, there are still many dark and arcane stretches along the road to Erlang mastery.

My position is this: Erlang is a tool. I want to master it as quickly and painlessly as possible so I can get on with the work of the world.

It could be just me. But I’ll venture the hypothesis that sucky user documentation considerably impedes the progress and productivity of many programmers striving to learn Erlang and, I’d even venture, an embarrassing number of Erlang veterans as well.

So who’s to blame?

My guess is that it’s a cultural thing — not only within the open-source Erlang community, but across the open-source community as a whole.

If so, this raises any number of provocative questions:

Does it matter?

Maybe not, since the problem persists and open-source thrives.

But I say yes because the more wizards we have crafting new tools and applications, talented programmers polishing up the jewels in the Erlang firmament, and enthusiastic newbies contributing new perspectives, insights, and exciting code, the more fun, productivity, and career opportunities for all.

If nothing else, the more vibrant the community, the easier it is to convince management that there are enough talented Erlang programmers out there to support that new project you’d like to propose.

To beef up the ranks, however, we need to project an enticing and friendly environment. Quality user docs go a long way toward achieving that.

What’s at the root of the problem?

In my experience, when we’re talking about significant software tools or ambitious software applications, producing quality user documentation is as difficult, if not more so, as writing the code to begin with. Why? Because you need to know how to install and configure for various platforms, how the interfaces work, how to write good use cases and examples, and how to effectively communicate all that with a population of users with wide ranging needs and competencies.

Creating quality user documentation, in other words, is just plain difficult and time-consuming.

Moreover, when we’re talking open-source, we’re talking for the most part, volunteer effort. Most techies would rather write or play with code than document it.

What do we mean by quality user documentation?

This question, I think, gets close to the heart of the matter.

There doesn’t seem to be much agreement about what constitutes quality user documentation. One sees little discussion of the differences between source documentation, reference documentation, and user documentation.

Nor do we see much effort toward consensual definition or quality standards when it comes to user documentation.

If we don’t know what it is, how can we do it? And “I know when I see it,” I contend, isn’t good enough.

Who’s responsible for creating user documentation?

Frankly, I think it’s asking too much to expect the core programmer of a significant package to produce quality user documentation. It’s a big enough job to conceptualize, implement, test, and maintain a large corpus of code.

Nor do I think we can expect some “documentation fairy,” as one of my readers suggested, to come along and produce good documentation for us.

Then who?

I suggest that it’s the responsibility of everyone of us who choose to adopt or use a body of code to contribute whatever we can toward better user documentation. But we well know, that if everyone has responsibility, then no one has responsibility.

So what can be done?

1. Hammer out a standard and framework for quality user documentation.

We do this for protocols like HTTP and XML. Why can’t we do it for documentation?

2. Produce/adapt/adopt a tool set to automate to the fullest extent possible the documentation task.

3. Publish all user documentation in wiki form so it can be edited/revised/extended by users.

4. Run an annual awards program for excellence in user documentation.

5. Work toward a culture that recognizes the value of quality user documentation, gives credit to those who advance it, encourages individuals to contribute what they can.

Look at it this way:

Imagine an on-line template that outlines in generic sense what a new user needs to know to successfully bring up and use and/or build out from a software tool or application. Now imagine ten, 20, or 30 skilled users taking ten minutes each to plug in the specifics for a given package, including use cases and examples. Let’s say each contibutor devotes 20 to 30 minutes to the task. No big sacrifice anyones part. But just imagine how valuable that would be to all who come behind.

With such a culture we can have our cake and eat it too… Spend most of our hours and days on code, but still have excellent user docs.

Many people seem to feel that Erlang deserves to become a leader in the Web 2.0/multi-core processor world. My contention is that energetic focus on quality user documentation is an essential step toward that goal.

Good folks behind Erlang.org, as well as Joe Armstrong, Kevin Smith, Hughes Waroquier, and others too numerous to name, have pointed the way.

I say, let’s all jump on board. When it comes to user documentation, we get what we deserve.

Newbie Tests: Inets – and the Newbie Score is…

This rather intermittent and meandering Newbie Test started out as a look at Mathew Foemmel’s “Hello World” Webapp.

http://blog.foemmel.com/2008/05/hello-world-webapp-in-erlang.html

But somehow it morphed into a Newbie Test of Inets.

See:

Newbie Tests: “Hello World” Webapp; February 15, 2009
Newbie Test: “Hello World” Webapp Round 2; February 17, 2009
Newbie Test: Inets – yet another frustrating day; February 23, 2009
Newbie Test: Inets – smidgen of progress; February 25, 2009
Newbie Test: Inets – breakthrough!; February 26, 2009

All at:

http://pokingarounderlang.wordpress.com

The reason is simple enough. “Hello World” is a jewel as far as it goes. Delivers on it’s promise. But it doesn’t promise much. And it delivers just enough to tease the wayward newbie into looking more deeply into Inets.

I don’t know the history behind Inets, but I do know that it stands tall in the Erlang OTP libraries. So, I presume, it must be important in the Erlang world.

The Inets User Guide promises powerful functionality including FTP, TFTP, and HTTP clients and HTTPD, an HTTP server. And HTTPD is an option, if not at the very heart, of several popular Erlang web frameworks.

http://erlang.org/doc/apps/inets/part_frame.html

My look at Inets focused on HTTPD server and all comments here relate only to that component. As noted in previous posts, my experience was fraught with considerable frustration, some of my own making.

Nevertheless, I’m going to say it. And it may hurt. And many an Erlang wizard may rant and flame. I just hope they don’t send ninjas in the night to assassinate me and my family.

The Inets User Guide… More specifically, Section 4 of the Inets User Guide, HTTPD server, REALLY SUCKS!

There, I’ve said it.

I’ve read and re-read that cursed chunk of cryptic prose numerous times. Throughout the first ten readings I was willing to accept that my failure to understand was simply a failure of intellect, experience, and imagination on my part. But then I got it. Section 4 of the Inets User Guide, HTTPD server, REALLY SUCKS!

My, it feels good to get that off my chest.

So here’s my updated Newbie Score Card.

Clarity and comprehensiveness of user docs

BeepBeep             8
Erlang Web           4
Inets HTTPD          1

Clarity of code and comments

BeepBeep             8
Erlang Web           -
Inets HTTPD          6

Conceptual clarity

BeepBeep            10
Erlang Web          10
Inets HTTPD          8

Ease of installation	

BeepBeep             10
Erlang Web            7
Inets HTTPD*          8

Ease with which I can build on it

BeepBeep              6
Erlang Web            2
Inets HTTPD           4

Likelihood that I would consider it further

BeepBeep              6
Erlang Web            2
Inets HTTPD           9

Rating scale: 10 – Totally Awesome; 1 – Total Turnoff
* Based on Mathew Foemmel’s post

Note that I’m highly motivated to come back to Inets. Why? By all lights it’s a powerful package.

But boy do I dread it.

My last feeble experiments suggest that Inets is not really as difficult to master as one would think struggling with the User Guide. But from where I stand today, it’s most definitely not Newbie friendly.

For more on Newbie Tests, see:

Newbie Tests; January 19, 2009

Newbie tests coming your way soon:

iserve: A fast web server demonstrating some undocumented Erlang features
MochiWeb: an Erlang library for building lightweight HTTP servers
Nitrogen: Web Framework for Erlang
Nitrogen: 5 Minute Blog Using Nitrogen and CouchDB
Yaws: Yet Another Webserver

Joe Armstrong on Nitrogen

Check out most recent posts on the Nitrogen forum for a lively discussion between Joe Armstrong and the developers of Nitrogen:

http://groups.google.com/group/nitrogenweb

The Old Tigers of Erlang have handed down a wonderful language. I, for one, am deeply grateful for their willingness to share their hard-won experience. Nitrogen, and the Erlang community as a whole, can only be stronger for it.

Newbie Test: Inets – yet another frustrating day

I once thought I was as bright as the next guy. But now I wonder. Clearly some people get it. But I don’t.

I just spent another frustrating day trying to understand the Inets documentation.

http://erlang.org/doc/apps/inets/part_frame.html

Confession: I really want to understand this stuff.

And I’m having trouble even understanding what I don’t understand.

I started the day trying to bring up the examples referred to in the mod_esi docs:

http://www.erlang.org/documentation/doc-5.3.6.13/lib/inets-3.0.9/doc/html/mod_esi.html

The mod_esi doc suggests that I:

“Take a look at httpd_example.erl in the code release (UNIX: $INETS/src, Windows: %INETS\src) for a clarifying example. Start an example server as described in httpd:start/0 and test the following from a browser”

OK. I jump over the the httpd docs and faithfully follow these instructions:

“start/0 and start/0 starts a server as specified in a hard-wired config file, that is start(“/var/tmp/server_root/conf/8888.conf”). Before utilizing start/0 or start_link/0, copy the example server root (UNIX: $INETS_ROOT/examples/server_root/, Windows: %INETS_ROOT%\examples\server_root\) to a specific installation directory (UNIX: /var/tmp/, Windows: X:\var\tmp\) and you have a server running in no time. “

Well, I didn’t.

=ERROR REPORT==== 23-Feb-2009::17:18:37 ===
Failed to start service:
“/var/tmp/server_root/conf/8888.conf”
due to: {missing_property,server_name}

Easy enough, I think. I bring up 8888.conf in my browser, see that the server name is commented out, and so add my own server name, Test.

Now I enter httpd:start() once again in my erl console, only to be greeted with this:

=ERROR REPORT==== 23-Feb-2009::17:24:01 ===
Failed initiating web server:
undefined
“mod_disk_log: Can’t create /var/tmp/server_root/logs/access_disk_log_8888, {file_error,\”/var/tmp/server_root/logs/access_disk_log_8888.siz\”,enoent}”

** exception exit: shutdown

At this point I gave up.

But I’m a persistent cuss. So I decided to go back to first principles, the inets User Guide. I’ve visited this material before; thought I’d start fresh to see if I missed something the first time around.

See: Newbie Test: “Hello World” Webapp Round 2; February 17, 2009

I’ve been focusing on Section 4: HTTP server. Section 4.2: Configuration is quite baffling. What’s an “erlang node application configuration file?” How is the file named? And where does it go?

I spent an hour or more on Google trying to track this down. Best I can tell it refers to OTP conventions. I spent considerable time skimming through OTP Design Principles striving for daylight, but no cigar. But I need to come back to this since there’s a great deal to absorb.

http://erlang.org/doc/design_principles/part_frame.html

Next I moved on to Section 4.3 of the inets User Guide: Using the HTTP Server API. And here, I actually experienced success.

Well, sort of. I was able to replicate all six steps. But when, after step 3, I tried to connect with the server through the browser… I get an Internal Server Error and am told to “Please contact the server administrator.”

Since that’s me, I don’t expect much relief.

So what do you think? Should I keep pounding my head against the wall?

I will, but I think I need to find another section of wall. This one is too bloody.

Incidentally, I did poke through the inets examples source and picked up a few clues. Whether or not they’ll lead to enlightenment, time will tell.

s t