Lessons from Erlang

Strange Loop 2012

Garrett Smith, CloudBees

@gar1t

• Pseudo-parallel, event-driven real-time programming language
• Dedicated for AXE telephone exchanges
• Built in the 1970s by Göran Hemdahl at Ericsson
• Effective, but expensive to use (low level, complex)

• OS independent virtual machine
• Massive fine grained concurrency
• Asynchronous message passing
• Reliability over performance
• Functional pragmatism over purity

• Isolation
• Fault detection and recovery
• Separation of concerns
• Back box design
• State management
• Avoid complexity

• Memory
• Threads
• Files
• Disks
• CPU Cores
• Network Interfaces
• Networks
• Racks
• Data Centers

• Have to be able to detect failure
• "Fail fast"
• Avoid defensive measures
• Limit the scope of failure

Courtesy of South Park

• Unplug the Internet
• Wait five seconds
• Plug Internet back in

• Easier to reason about
• Easier to test
• Isolation effect - limited scope for change

• Easy to setup (just plug in?)
• Start button
• Minimal controls
• Reboot to fix

"

• Durability -> Recovery
• Replication -> Failover
• Integrity -> Repair
• Consistency -> Synchronization

Courtesy Oracle

• Dependencies
• Nesting / Hierarchies
• Resource Sharing
• Lots of Code
• Fear

• No shared memory
• Communicate via "message passing" (stdio, sockets, pipes)
• Process terminate (i.e. "fault") detection
• Techniques
  • Standard IO "servers"
  • 0MQ (light weight inter process communication via messages)
  • TCP / HTTP

• No shared memory (semantically)
• Queues to process messages
• Inter thread communication via queue inserts (message passing)
• Direct language support: Scala, Go, Erlang
• Libraries: Kilim (Java), Pykka (Python), Celluloid (Ruby), libcppa (C++)

• Avoid defensive practices
• Let exceptions propagate as far as possible
• Use assertions and leave them in!
• Exiting the process is not a bad idea

• Process monitors: runit, launchd
• Standard IO "servers"

• Narrowing the scope of an “application”
• Appliance oriented development
• Micro SOA
• Functional style programming (e.g. limit avg functions to < 4 lines)

• If it's not obvious, work until it becomes obvious
• Take small steps, doing what's clearly the next thing
• Avoid building for the "future"