Bloggity Blog

We love our work so much, we want to tell the world about it.

Littlelines Roadtrip (Summer Edition)

Written by Matt Sears

The Littlelines crew will taking the show on the road this week. We always like to take the opportunity to stretch our legs and get out of the city for a few days to meet folks in the industry. This time round we're loading up the car and sticking to the great Midwest.

Steel City Ruby - Pittsburgh, PA

Steel City Ruby Logo

Our first stop will be Pittsburgh for the Steel City Ruby conference. Steel City is entering it's third year and the speaker line up looks like it's going to be another great one. We had the pleasure once again of designing the conference tee shirts and we think they look really great.

Weapons of Mass Creation - Cleveland, OH

WMCFest Logo

Next up, we're heading north to Cleveland for Weapons of Mass Creation. This is the premier art, design, and music event in the country. This is the fifth year running and has grown to a 2,000+ attendee event. Checkout the amazing speaker line up - can't wait to listen to all the great talks!

If you see us, stop and say hi, we'd love to meet you! And hey, you never know - we might have goodies to hand out.

Welcome Dustin Armstrong

Written by Matt Sears

Dustin Image

It is my great pleasure to welcome Dustin Armstrong to our development team! Dustin brings his clean, responsive, front-end coding skills to team with a design talent to boot. Dustin is a 2009 graduate of School of Advertising Art and has been writing solid code for the past five years. Dustin's experience in design and development will help our clients products be a glowing success. You may follow Dustin on Twitter and on Github

Welcome aboard Dustin!

A Survival Guide for Legacy Rails Apps

Written by Matt Sears

Hello, my name is Matt and I love working with legacy Rails apps. Ruby on Rails is now over 10 years old. That means there are a lot of (old) Rails applications running out there. At Littlelines, we've worked on hundreds of Rails projects. Most of them we build from the ground up, but often we have the opportunity to work with legacy Rails apps - some as old as 2006! More often than not, we discover these apps include many many common mistakes made back in the day and it's our job to fix them. Over the eight years of writing Rails apps, I've picked up a few tips and tricks that can help us get through the agony of legacy code.

1. Take Stock of the Situation

It is impossible to understand the present or prepare for the future unless we have some knowledge of the past. So first things first, let's take an assessment of where we're at in terms of test coverage and code quality. If we're lucky enough to take on a project that has tests, the first thing I do is install SimpleCov to measure how well our app is tested. Once I have the SimpleCov report, I'll take a screenshot and save the report. Then we can use CodeClimate to get an overall grade on quality and security. CodeClimate will analyze our app and report on all the hotspots and security violations in the code. Finally, I'll write this number down and take a screenshot.

Now that we have metrics to work with, we can do some Opportunistic Refactoring. In other words, always leave code behind in a better state than you found it. The fun part is seeing how far we can improve the code's test coverage and quality and we can challenge ourselves to take the score from an F to an A.

2. Sharpen the Saw

If I had eight hours to chop down a tree, I'd spend six sharpening my axe. – Abraham Lincoln

As developers, we spend most of our day buried in our text editor. Luckily for us, most text editors allow us to customize and fine tune it to make it work more effectively for us. Even better, some editors like Vim and Emacs allow you to create custom functions to help automate repetitive and complex tasks. These can be incredibly useful when working with legacy code. One of my favorite functions converts old Ruby 1.8's hash syntax to 1.9's new syntax.

Ruby Hash Converter

Some text editors like Vim and Emacs ship with built-in Macros support. Macros allow us to record keystrokes and play them back. These can be great tools to automate simpler tasks with less effort. The nice things about Macros is we don't need to write a function, we just need to hit the record button and play it back.

Let's take look at an example. Let's say we are upgrading a Rails application and we discover that it's using the old style of validating ActiveRecord fields. We can create an ad hoc macro to convert the first line of the validation code to the new validation syntax and play it back for the next three lines.

Emacs Macro and Playback

3. Learn Something New

As soon as we stop try new things, we stop learning. We can always be better Rubyist and so as a general rule, I try to learn something new when starting a new project. It doesn't have to be anything huge. In fact, it's usually a small thing and something that fits with the project. It can be anything from replacing our view templates with a new template engine like Slim or something as small like using Ruby 2.0's new keyword arguments. Most likely our legacy app is using some old and unmaintained gems. This is a great opportunity to see what we can replace them with. Head on over to the The Ruby Toolbox or Awesome Ruby and see the latest and greatest libraries available.

To give you an example, A few years ago I was tasked with upgrading an old Rail 1.2 app to the latest and greatest version. During the upgrade process, I discovered a lot of embedded SQL and the project owner had complained about how slow the searching has been on the site. So, I thought it would be a good opportunity to learn more about how to do full-text searching in Rails. And this lead me to discover the great Sunspot gem. With Sunspot, I was able to eliminate all the embedded SQL and make the search perform much much faster at the same time.

4. Have a Plan

As a rule for every new project, I make a list of things I'd like to accomplished by the end of the project. It's usually a small list that contains some very high level goals. In most cases, the goals coincide with making a better Rails app and me a better a developer in the end. For example, here is a list I made on my last project.

  1. Upgrade application to Rails 4 and Ruby 2.
  2. Raise Code Climate GPA to 4.0.
  3. Increase test code coverage by 20%.
  4. Watch RubyTapas episode on Null Object and apply it.
  5. Try out Byebug gem and see how it stacks up to Pry.
  6. Write a new Emacs Lisp function that converts erb to haml across multiple buffers.

5. Keep Pushing

As many of you may know, working with legacy projects can be boring and frustrating. But, we can take steps to make it a little more fun and learn something new in process. Even when the code is horrendous and we're cursing who ever wrote this pile of #@$%, we can still learn something new and challenge ourselves. And it isn't always new tricks or tools we learn, it's also the past mistakes that teach us. If you commit to taking these steps, you'll improve your skills at a much faster rate and you'll find yourself stepping out of your normal routine and applying new solutions that will ultimately lead to you becoming a better developer.

How about you? If you have any tips for working on legacy Rails apps, please add a comment below.

Elixir vs Ruby Showdown - Phoenix vs Rails

Written by Chris McCord

Phoenix vs Rails

This is the second post in our Elixir vs Ruby Showdown series. In this latest installment, we're exploring the performance of the Phoenix and Rails web frameworks when put up against the same task. Before we get into code samples and benchmark results, let's answer a few common questions about these kinds of tests:

tl;dr Phoenix showed 10.63x more throughput over Rails when performing the same task, with a fraction of CPU load


Isn't this apples to oranges?

No. These tests are a direct comparison of our favorite aspects of Ruby and Rails with Elixir and Phoenix. Elixir has the promise to provide the things we love most about Ruby: productivity, metaprogramming, elegant APIs, and DSLs, but much faster, with a battle-tested concurrency and distribution model. The goals of this post are to explore how Elixir can match or exceed our favorite aspects of Ruby without sacrificing elegant APIs and the productive nature of the web frameworks we use.

Are benchmarks meaningful?

Benchmarks are only as meaningful as the work you do upfront to make your results as reliable as possible for the programs being tested. Even then, benchmarks only provide a "good idea" of performance. Moral of the story: never trust benchmarks, always measure yourself.

What are we comparing?

Elixir Phoenix Framework

  • Phoenix 0.3.1
  • Cowboy webserver (single Elixir node)
  • Erlang 17.1

Ruby on Rails

  • Rails 4.0.4
  • Puma webserver (4 workers - 1 per cpu core)
  • MRI Ruby 2.1.0

We're measuring the throughput of an "equivalent" Phoenix and Rails app where specific tasks have been as isolated as possible to best compare features and performance. Here's what we are measuring:

  1. Match a request from the webserver and route it to a controller action, merging any named parameters from the route
  2. In the controller action, render a view based on the request Accept header, contained within a rendered parent layout
  3. Within the view, render a collection of partial views from data provided by the controller
  4. Views are rendered with a pure language templating engine (ERB, EEx)
  5. Return the response to the client

That's it. We're testing a standard route matching, view rendering stack that goes beyond a Hello World example. Both apps render a layout, view, and collection of partials to tests real-world throughput of a general web framework task. No view caching was used and request logging was disabled in both apps to prevent IO overhead. The wrk benchmarking tool was used for all tests, both against localhost, and remotely against heroku dynos to rule out wrk overhead on localhost. Enough talk, let's take a look at some code.



defmodule Benchmarker.Router do
  use Phoenix.Router
  alias Benchmarker.Controllers

  get "/:title", Controllers.Pages, :index, as: :page


Benchmarker::Application.routes.draw do
  root to: "pages#index"
  get "/:title", to: "pages#index", as: :page


Phoenix (request parameters can be pattern-matched directly in the second argument)

defmodule Benchmarker.Controllers.Pages do
  use Phoenix.Controller

  def index(conn, %{"title" => title}) do
    render conn, "index", title: title, members: [
      %{name: "Chris McCord"},
      %{name: "Matt Sears"},
      %{name: "David Stump"},
      %{name: "Ricardo Thompson"}


class PagesController < ApplicationController

  def index
    @title = params[:title]
    @members = [
      {name: "Chris McCord"},
      {name: "Matt Sears"},
      {name: "David Stump"},
      {name: "Ricardo Thompson"}
    render "index"


Phoenix (EEx)

    <h4>Team Members</h4>
      <%= for member <- @members do %>
          <%= render "bio.html", member: member %>
      <% end %>
<b>Name:</b> <%= %>

Rails (ERB)

    <h4>Team Members</h4>
      <% for member in @members do %>
          <%= render partial: "bio.html", locals: {member: member} %>
      <% end %>
<b>Name:</b> <%= member[:name] %>

Localhost Results

Phoenix showed 10.63x more throughput, with a much more consistent standard deviation of latency. Elixir's concurrency model really shines in these results. A single Elixir node is able to use all CPU/memory resources it requires, while our puma webserver must start a Rails process for each of our CPU cores to achieve councurrency.

  req/s: 12,120.00
  Stdev: 3.35ms
  Max latency: 43.30ms

  req/s: 1,140.53
  Stdev: 18.96ms
  Max latency: 159.43ms


$ mix do deps.get, compile
$ MIX_ENV=prod mix compile.protocols
$ MIX_ENV=prod elixir -pa _build/prod/consolidated -S mix phoenix.start
Running Elixir.Benchmarker.Router with Cowboy on port 4000

$ wrk -t4 -c100 -d30S --timeout 2000 ""
Running 10s test @
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.31ms    3.53ms  43.30ms   79.38%
    Req/Sec     3.11k   376.89     4.73k    79.83%
  121202 requests in 10.00s, 254.29MB read
Requests/sec:  12120.94
Transfer/sec:     25.43MB


$ bundle
$ RACK_ENV=production bundle exec puma -w 4
[13057] Puma starting in cluster mode...
[13057] * Version 2.8.2 (ruby 2.1.0-p0), codename: Sir Edmund Percival Hillary
[13057] * Min threads: 0, max threads: 16
[13057] * Environment: production
[13057] * Process workers: 4
[13057] * Phased restart available
[13185] * Listening on tcp://

$ wrk -t4 -c100 -d30S --timeout 2000 ""
Running 10s test @
  4 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    21.67ms   18.96ms 159.43ms   85.53%
    Req/Sec   449.74    413.36     1.10k    63.82%
  11414 requests in 10.01s, 25.50MB read
Requests/sec:   1140.53
Transfer/sec:      2.55MB

Heroku Results (1 Dyno)

Phoenix showed 8.94x more throughput, again with a much more consistent standard deviation of latency and with 3.74x less CPU load. We ran out of available socket connections when trying to push the Phoenix dyno harder to match the CPU load seen by the Rails dyno. It's possible the Phoenix app could have more throughput available if our client network links had higher capacity. The standard deviation is particularly important here against a remote host. The Rails app struggled to maintain consistent response times, hitting 8+ second latency as a result. In real world terms, a Phoenix app should respond much more consistently under load than a Rails app.

  req/s: 2,691.03
  Stdev: 139.92ms
  Max latency: 1.39s

  req/s: 301.36
  Stdev: 2.06s
  Max latency: 8.36s

Phoenix (Cold)

$ ./wrk -t12 -c800 -d30S --timeout 2000 ""
Running 30s test @
  12 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   317.15ms  139.55ms 970.43ms   81.12%
    Req/Sec   231.43     66.07   382.00     63.92%
  83240 requests in 30.00s, 174.65MB read
  Socket errors: connect 0, read 1, write 0, timeout 0
Requests/sec:   2774.59
Transfer/sec:      5.82MB

Phoenix (Warm)

$ ./wrk -t12 -c800 -d180S --timeout 2000 ""
Running 3m test @
  12 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   318.52ms  139.92ms   1.39s    82.03%
    Req/Sec   224.42     57.23   368.00     68.50%
  484444 requests in 3.00m, 0.99GB read
  Socket errors: connect 0, read 9, write 0, timeout 0
Requests/sec:   2691.03
Transfer/sec:      5.65MB



sample#memory_pgpgin=204996pages sample#memory_pgpgout=196379pages

Rails (Cold)

$ ./wrk -t12 -c800 -d30S --timeout 2000 ""
Running 30s test @
  12 threads and 800 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.85s     1.33s    5.75s    65.73%
    Req/Sec    22.68      7.18    61.00     69.71%
  8276 requests in 30.03s, 18.70MB read
Requests/sec:    275.64
Transfer/sec:    637.86KB

Rails (Warm)

$ ./wrk -t12 -c800 -d180S --timeout 2000 ""
Running 3m test @
  12 threads and 800 connections
	  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.07s     2.06s    8.36s    70.39%
    Req/Sec    24.65      9.97    63.00     67.10%
  54256 requests in 3.00m, 122.50MB read
  Socket errors: connect 0, read 1, write 0, timeout 0
Requests/sec:    301.36
Transfer/sec:    696.77KB





Elixir provides the joy and productivity of Ruby with the concurrency and fault-tolerance of Erlang. We've shown we can have the best of both worlds with Elixir and I encourage you to get involved with Phoenix. There's much work to do for Phoenix to match the robust ecosystem of Rails, but we're just getting started and have very big plans this year.

Both applications are available on Github if want to recreate the benchmarks. We would love to see results on different hardware, particularly hardware that can put greater load on the Phoenix app.

Shoutout to Jason Stiebs for his help getting the Heroku applications setup and remotely benchmarked!

Elixir vs Ruby Showdown - Part One

Written by Chris McCord

We've taken a huge interest in Elixir here at the Littlelines office this year. I gave a 3.5 hour into to Elixir workshop at RailsConf in April, and have been busy building Phoenix, an Elixir Web Framework. Earlier this week, I put together Linguist, an Elixir internationalization library and was shocked at how little code it required after taking a look at the Ruby implementation. By using Elixir's metaprogramming facilities, I was able to define function heads that pattern match on each I18n key. This approach simply generates a function per I18n key, whose function body returns the translation with any required interpolation. Let's see it in action.

tl;dr The Elixir implementation is 73x faster than Ruby's i18n gem

edit Joel Vanderwerf put together a Ruby implementation in response to this post that runs in 3.5s, making the Elixir implementaiton 2.18x as fast. gist

defmodule I18n do
  use Linguist.Compiler, locales: [en: [
    foo: "bar",
    flash: [
      notice: [
        alert: "Alert!",
        hello: "hello %{first} %{last}",

iex> I18n.t("en", "flash.notice.alert")
iex> I18n.t("en", "flash.notice.hello", first: "chris", last: "mccord")
"hello chris mccord"

By calling use Linguist.Compiler, the above code would expand at compile time to:

defmodule I18n do
  def t("en", "foo") do
    t("en", "foo", [])
  def t("en", "foo", bindings) do

  def t("en", "flash.notice.alert") do
    t("en", "flash.notice.alert", [])
  def t("en", "flash.notice.alert", bindings) do

  def t("en", "flash.notice.hello") do
    t("en", "flash.notice.hello", [])
  def t("en", "flash.notice.hello", bindings) do
    ((("hello " <> Dict.fetch!(bindings, :first)) <> " ") <> Dict.fetch!(bindings, :last)) <> ""

Notice in the last function definition, the interpolation is handled entirely by string contcatenation instead of relying on regex splitting/replacement at runtime. This level of optimization isn't possible in our Ruby code.

Ruby's implementation requires a complex algorithm to split the I18n keys into a Hash to allow performant lookup at runtime. Since our Elixir implementation just produces function definitions, we let the Erlang Virtual Machine's highly optimized pattern matching engine take over to lookup the I18n value. The result is strikingly less code for equivalent functionality. Not only do we get less code, we also get a 77x speed improvement over Ruby 2.1.0. Here's a few benchmarks we ran to see how the Elixir implementation compared to Ruby:


defmodule Benchmark do

  defmodule I18n do
    use Linguist.Compiler, locales: [
      en: [
        foo: "bar",
        flash: [
          notice: [
            alert: "Alert!",
            hello: "hello %{first} %{last}",
            bye: "bye now, %{name}!"
        users: [
          title: "Users",
          profiles: [
            title: "Profiles",

  def measure(func) do
    |> elem(0)
    |> Kernel./(1_000_000)

  def run do
    measure fn ->
      Enum.each 1..1_000_000, fn _->
        I18n.t("en", "foo")
        I18n.t("en", "users.profiles.title")
        I18n.t("en", "flash.notice.hello", first: "chris", last: "mccord")
        I18n.t("en", "flash.notice.hello", first: "john", last: "doe")



  foo: "bar"
      alert: "Alert!"
      hello: "hello %{first} %{last}"
      bye: "bye now %{name}!"
    title: "Users"
      title: "Profiles"
class Benchmarker

    Benchmark.measure do
      1_000_000.times do |i|
        I18n.t("flash.notice.hello", first: "chris", last: "mccord")
        I18n.t("flash.notice.hello", first: "john", last: "doe")


Benchmark Results *

  • Elixir (0.14.1) Average across 10 runs: 1.63s
  • Ruby (MRI 2.1.0) Average across 10 runs: 118.62s

That's a 77x speed improvement for Elixir over Ruby, with the same top-level API! With careful use of metaprogramming, we were able to produce a clean implementation with compile-time optimized lookup. Elixir provides metapgramming abilities well beyond what we can dream up as Rubyists. Here's a few resources to learn more:

*never trust benchmark results, always measure yourself