Erlang Web Development with modlib

I tend to view “web applications” as “HTTP interfaces” to underlying, independent functionality. So, you build an application and then expose that over HTTP to clients — either as browser application or an API end point.

This isn’t just a semantic shift — it drives the entire approach to building software!

For many developers, building software means building web applications. I’ve seen “web apps” used for everything from monitoring to batch processing. For them, every software problem is cast as “how do I solve this using my favorite web framework?”

This is the application view for web-bound developers:

This is the traditional application view:

Maybe this is a trivial difference. Who cares, right?

There are a lot of reasons why a systems view is the Right Way to approach software development. For one, a well designed application hosting environment (e.g. an operating system) encourages independence. Web frameworks generally don’t list “application independence” as a priority — and if they do, they probably leverage the operating system (e.g. fork-exec).

Independence in software is a topic for many, many blog posts :)

In this post, we’ll aspire only to use a simple tool to HTTP-enable an existing Erlang application!

modlib

modlib is a small library that makes it easy to build web applications that run in Erlang’s built-in http server.

What’s so special about Erlang’s http server?

• It comes with Erlang, so you can avoid a dependency
• It’s designed like Apache, using “mods” that can be configured as needed to quickly add features to your application
• It works well within the “systems” view — i.e. you can easily maintain independence across your application components

The big downside of Erlang’s http server is that it’s painful to work with.

modlib makes it easy.

The Application

We’re not going to start with a web application — we’re going to start with the underlying functionality. Once that’s in place, we’ll add an HTTP interface.

Let’s borrow some code from e2. From an OS shell:

$ git clone https://github.com/gar1t/e2
$ cp -a e2/examples/mydb modlib-example

Build the project:

$ cd modlib-example
$ make

This will compile the “mydb” example — a simple key/value store accessible using a TCP socket protocol. You can read all about this example in the e2 tutorial.

Start the mydb application in an Erlang shell:

$ make opts="-config test -s mydb" shell

This make target starts Erlang using the test.config file and starts the mydb application.

You can test the mydb data API in the Erlang shell (at each > prompt, type the text as listed below and press ENTER — you should see the corresponding output printed in the Erlang shell):

1> mydb_data:get("msg").
error
2> mydb_data:put("msg", "Hello mydb").
ok
3> mydb_data:get("msg").
{ok,"Hello mydb"}
4> mydb_data:del("msg").
ok
5> mydb_data:get("msg").
error

This is an example of an independently running application. The mydb database is running as an Erlang process — and we use the API defined in mydb_data to access it.

The database also handles TCP connections. Let’s see how this works.

From an OS shell, connect to the database using telnet:

$ telnet localhost 2222

To see how the TCP in action, type the commands below that follow the >> prompt (NOTE the >> prompt is not actually displayed in telnet — e.g. to execute the first command below, just type “GET msg” and then ENTER):

>> GET msg
-ERROR
>> PUT msg Hello mydb
+OK
>> GET msg
+Hello mydb
>> DEL msg
+OK
>> GET msg
-ERROR

This is an example of a TCP interface to the database. This interface supports multiple simultaneous connections, making mydb a concurrent key/value store!

Adding an HTTP Interface

We’ve just seen the “real” app. It works perfectly well on its own, but we also want to support client access over HTTP. This is where modlib comes in.

Let’s add modlib to as a build dependency. Edit rebar.config, located in the modlib example directory, so that it looks like this:

%% -*- erlang -*-
{erl_opts, [debug_info]}.
{deps,
 [{e2, ".*", {git, "git://github.com/gar1t/e2.git"}},
   {modlib, ".*", {git, "git://github.com/gar1t/modlib.git"}}]}.

Get the new dependency by running this from the OS shell:

$ make -B deps

mydb_http stub

I tend to use the naming convention that Erlang modules that handle HTTP requests end with “_http”.

Create the file src/mydb_http.erl that looks like this:

-module(mydb_http).

-include_lib("modlib/include/webapp.hrl").

-export([request/3]).

request("GET", Key, _Info) ->
    {ok, ["TODO: GET ", Key]};
request("PUT", Key, Info) ->
    {ok, ["TODO: PUT ", Key, " = ", modlib:body(Info)]};
request("DELETE", Key, _Info) ->
    {ok, ["TODO: DELETE ", Key]}.

This is a modlib “web app” — it’s a simple stub for an HTTP interface. It’s hard to imagine a simpler interface! But we’ll see in a moment that this is all we need.

Compile the new module by running this OS command in the mydb example root directory:

$ make

Configure modlib

Next we need to setup the HTTP server. This is done by defining a server in the modlib configuration. Modify test.config, located in the root mydb example directory, to look like this:

[{mydb, [{server_port, 2222}, {db_file, "test.db"}]},
 {modlib, [{servers, [{8080, [{modules, [mydb_http]}]}]}]}].

The modlib entry defined a single server running on port 8080. The server uses one “mod” — our new mydb_http web app!

Exit the Erlang shell by typing CTRL-C twice. Restart it using the same command:

$ make opts="-config test -s mydb" shell

This starts the mydb application as before: the same Erlang and TCP socket interfaces are available.

Let’s now start the modlib application. In the Erlang shell, run:

1> application:start(modlib).
ok

As before, type the Erlang statements following the “>” prompt and press ENTER.

When the modlib application starts, it starts the servers specified in the configuration file. In this case, an HTTP server is running on port 8080.

Use an HTTP client like curl to test the interface:

$ curl localhost:8080/foo
TODO: GET /foo
$ curl localhost:8080/foo -X PUT --data hello
TODO: PUT /foo = hello
$ curl localhost:8080/foo -X DELETE
TODO: DELETE /foo

Sweet!

Finish mydb_http

Modify mydb_http to look like this:

-module(mydb_http).

-include_lib("modlib/include/webapp.hrl").

-export([request/3]).

request("GET", Key, _Info) ->
    handle_get(mydb_data:get(Key));
request("PUT", Key, Info) ->
    handle_put(mydb_data:put(Key, modlib:body(Info)));
request("DELETE", Key, _Info) ->
    handle_del(mydb_data:del(Key)).

handle_get({ok, Value}) -> {ok, Value};
handle_get(error) -> {not_found, "Not Found"}.

handle_put(ok) -> {ok, ""}.

handle_del(ok) -> {ok, ""}.

Believe it or not, this is a complete HTTP interface — 13 lines of code!

Compile the changes:

$ make

The example is setup to automatically reload the changed module — you don’t need to restart the Erlang shell for the change to take effect.

Use your HTTP client (e.g. curl) to test your new interface!

Wrapping Up

If you like the idea of tacking on HTTP interfaces to your apps, as needed, modlib is a drop dead easy way to do it.

But don’t let this ultra-simple approach fool you! You can build very advanced web applications using what you’ve seen here, plus a little extra, depending on what you need:

• mod_get to serve static content from a directory
• mod_auth to add HTTP authentication
• ErlyDTL
• Various Erlang database bindings, or perhaps boss_db if you like ORMs
• A great JavaScript library like YUI

You can get from “simple” to “advanced” using small steps. Add only what you need when you need it!

A Note On Middleware

The “mods” design of the Erlang http server uses a middleware pattern: you can insert software components into the middle of a client/server exchange.

This is a Very Good Pattern.

Python’s rich web ecosystem owes its success to an effective middleware strategy in WSGI.

Erlang needs this.

The “mod” interface defined by httpd is an example, though a terrible one — it’s is blindingly complicated. modlib could serve as a step toward improving it, though it was never envisioned as a generalized HTTP interface ala WSGI.

To achieve WSGI like success in Erlang, the Erlang community needs a drop-dead simple interface that can be used to chain HTTP handlers together. If this was done right and promoted effectively, I think we’d see Erlang emerge as a top tier platform for “web development”.

References

• modlib
• Erlang HTTP server