To make that easier, here’s a list I’ve been meaning to put together for a long time, of books and other writings that have been essential in shaping my practice of programming. Well-read programmers will find few surprises here, but can be sure that any of these that they’ve missed will be worth their time.

Extreme Programming Explained, Kent Beck, 1999. (Although there’s a second edition, I’ve only read the first.) It’s only 1999 and nearly all of the essentials of agile development are already here: developer testing, refactoring, pairing, collective ownership, iterations. Twelve years later, it’s still hard to do much better. This may be the ideal first book for a professional programmer. For extra credit, the C2 wiki, where I first saw much of this material, is still a great place to see how some of XP’s ideas matured in public.

Test-Driven Development By Example, Kent Beck, 2003. This book supplies the single magic ingredient missing from Extreme Programming Explained, the driving force at the heart of agile development. In a mere 200 pages this book will bring you out of the desert of test-last in which you’ve been wandering and put you on the path to constant upward productivity. Sound cultish? Yes, it’s that essential. The RSpec Book (Chelimsky et al, 2010) covers similar ground but is muddled by details of tool use and the complex relation between unit and functional testing; read it if you write Ruby, but read Beck first.

Refactoring, Martin Fowler et al., 1999. The first programming book that made me realize that there was such a thing as a good programming book. Chapter 3, the best chapter (not by Martin Fowler but by Kent Beck), “Bad Smells in Code”, tells you in only a few pages so many of the reasons why bad code might be bad and what to do about it. The bulk of the book is a catalog of refactorings, not as inspiring reading as chapter 3 but as essential to good programming as (they tell me) standard openings are to playing chess. Practice them until they’re part of you.

Object-Oriented Software Construction, second edition, Bertrand Meyer, 1997. (Amazon’s page says 2000, but appears to describe the same second edition.) A rare exception to the rule that good books are short. This thousand-pager taught me how to think about object-oriented programs and how to design classes (Meyer codified the Open-Closed Principle) and methods (he also codified command-query separation and Design by Contract), and much more. Even agilists need to know good design when they back in to it, and this book goes a long way towards teaching us how. For those in a hurry, or who can’t find a copy of OOSC, Mitchell and McKim’s Design by Contract by Example (2001) covers many of Meyer’s high points and has the advantages of being short and in print, although it lacks OOSC’s sweep and impact.

I was lucky enough to have some smart people point out to me Robert C. Martin’s paper “Design Principles and Design Patterns” (2000) (linked to, along with other good stuff, here) at a time when I badly needed its explanation of software stability. It is easily the most important writing on object-oriented design since OOSC, introducing rationally derived principles not just of class-level design but of architecture. Agile Software Development (Martin et al., 2001) has a more thoroughly developed version of the same material, along with discussions of agile practices and some design patterns.

Code Complete, Steve McConnell, 1993 (!). This is the definitive guide to writing code line by line. It discusses in detail the best ways to declare variables, use control structures, name methods and even format code. Some details are language-specific but their rationales can be easily adapted to whatever language you’re using. Although my first edition barely seems dated, if you insist on something more recent there is a second edition (which I haven’t read, so caveat lector) and Robert C. Martin’s Clean Code (2009) (which I have) does a reasonable job of serving the same purpose.

Continuous Delivery, Jez Humble and David Farley, 2010. Covering source code control, automated testing, continuous integration and automated deployment, this book accomplishes the hat trick of summarizing many practical aspects of managing and delivering software which I had to learn the hard way over many years, doing so clearly and insightfully enough that I appreciate its treatment even of familiar material, and integrating enough current thinking on distributed version control and short-cycle delivery that there is plenty to learn here even for long-time CI devotees.

I’ll add just one book which strays a bit up the food chain from this list’s focus on books about writing code. Writing Effective Use Cases, Alistair Cockburn, 2001, is wonderfully clear and concise on the subject of writing down what an application needs to do to meet a user’s needs. Writing out use cases has gone with the agile wind, but anyone writing or implementing user stories needs to understand and keep in mind all of the concepts presented here.

Look at that — not one book on a language or framework. Not that there aren’t a few good ones, but it’s learning the fundamentals, independent of this year’s favorite technology, that really has lasting value.

Aside from technology-specific books, I’ve left off many that are worthwhile but not of the same exalted rank as those above (a line which I found surprisingly easy to draw), some that I know are well thought of but haven’t read yet, and some that I just dislike. If you’ve got a recommendation that you think is as valuable as the books on this list I would love to hear about it.

Getting rspec to run

I installed rvm on my development machine per the instructions and ran the specs. They refused to run, complaining about “invalid multibyte char”s in several files. The multibyte chars were there on purpose, to test that they made it in to and out of MySQL correctly. It turns out that while Ruby 1.8.7 obligingly handled UTF-8 in source files, 1.9.2 is strict and does not. The solution is to declare that the entire source file is UTF-8 with this comment on the first line:

# encoding: UTF-8


That fixed the encoding errors, but rspec still didn’t run:

> rake spec
(in /Users/dave/data/projects/gww/git)
/Users/dave/.rvm/gems/ruby-1.9.2-p180@global/gems/rake-0.8.7/lib/rake.rb:32:
  warning: already initialized constant RAKEVERSION
/Users/dave/.rvm/gems/ruby-1.9.2-p180@global/gems/rake-0.8.7/lib/rake/alt_system.rb:32:
  warning: already initialized constant WINDOWS
WARNING: Possible conflict with Rake extension: String#ext already exists
WARNING: Possible conflict with Rake extension: String#pathmap already exists
/Users/dave/.rvm/gems/ruby-1.9.2-p180@global/gems/rake-0.8.7/lib/rake.rb:404:
  warning: already initialized constant EMPTY_TASK_ARGS
[a screenful of already initialized constants later ...]
rake aborted!
stack level too deep


These errors came from confusion among the many copies of rake installed by Apple, rvm and GWW’s Gemfile. Fortunately I’d seen some recent blog posts on the rake 0.9 debacle which reminded bundler users that we should really have been running rake with

> bundle exec rake spec


(or whatever target) all along, and indeed using bundle exec fixed the errors and allowed rspec to run. Of course some individual specs were still failing.

Getting all of the specs to pass

The most common breakage came from code like this:

describe LocationParser
  it "finds a plain address" do
    text = '555 California'
    LocationParser.new([]).parse(text).should ==
      [ Address.new text, '555', 'California', nil ]
  end
end


The spec (this example happens to be a spec, but the issue is not rspec-specific) had passed in 1.8.7, but in 1.9.2 it failed like so:

/Users/dave/.rvm/gems/ruby-1.9.2-p180/gems/rspec-core-2.5.1/lib/rspec/core/configuration.rb:386:in `load':
  /Users/dave/data/projects/gww/git/spec/lib/location_parser_spec.rb:33:
  syntax error, unexpected tIDENTIFIER, expecting ']' (SyntaxError)


The fix was simply to add parentheses around the arguments to Address.new. This seems to be a bug in Ruby 1.9.2. We’ll see what the Ruby team thinks. Meanwhile it’s easy enough to work around.

A few specs that used RR (Double Ruby) had problems. One type of problem occurred when for example I chained query methods in a controller

photos = Photo.where(:person => @person).order(:dateadded)


and stubbed it with RR’s double graphs:

stub(Photo).where.stub!.order { [ photo ] }


Specs that did this failed with

NotImplementedError: super from singleton method that is defined to multiple classes is not supported;
  this will be fixed in 1.9.3 or later


Caught red-handed! Those chained query methods didn’t belong in controllers anyway. I extracted them into model methods which I could then stub in the usual way.

The other RR problem was in a spec that stubbed the value of 1.day.ago with this RR construct

any_instance_of(ActiveSupport::Duration) do |d|
   stub(d).ago { Time.local(2011) }
end


It failed with this error:

ActionView::Template::Error: uninitialized constant ActiveSupport::Duration::RR


This only happened one place in the application so I didn’t look in to it at all. I just replaced 1.day.ago with Time.now - 1.day, stubbed Time.now and moved on.

The very last class of spec failures was an odd one. A couple of controllers converted some model objects to JSON to be inserted into a page. Their specs tested that the JSON contained what it should like so:

json = {
  'id' => photo.id,
  'latitude' => photo.latitude,
  'longitude' => photo.longitude,
  'color' => 'FFFF00',
  'symbol' => '?'
}.to_json
assigns[:json].should == json


Under 1.9.2 this spec failed, with the expected and emitted JSON Strings containing the same data but in different order. A tour of to_json, which treats hashes and ActiveRecord objects even more differently than I expected, convinced me that I wasn’t able to control the order, so I settled for this somewhat uglier alternative:

json = {
  'id' => photo.id,
  'latitude' => photo.latitude.to_s, # because BigDecimals are strings in JSON
  'longitude' => photo.longitude.to_s,
  'color' => 'FFFF00',
  'symbol' => '?'
}
ActiveSupport::JSON.decode(assigns[:json]).should == json


With that, all of the specs passed. Amusingly, almost every change I’d needed to make had been in the specs; I’d barely needed to touch the actual code.

There was one thing left to fix before moving on to production: Ruby 1.9.2 warns about suboptimal regular expressions, and GWW had one. A regexp containing something like (?:\w+)? produced this warning:

nested repeat operator + and ? was replaced with '*'


In some cases of this warning the right thing to do would be to rewrite the regexp to use * instead of (?:)? and +. In this case, the (?:)? happened to be completely unnecessary and I just removed it.

Deploying to production

Switching the production instance to 1.9.2 was easy. I just installed rvm (multi-user this time) and followed the RVM/Passenger integration instructions. In my hands Passenger does need an .rvmrc in the project root and the config/setup_load_paths.rb given in those instructions.

RVM and TeamCity

I left GWW’s continuous integration for last. GWW uses TeamCity. It has a simple three-step build: it bundle installs, it copies database.yml into place and it does rake db:migrate spec:rcov.

• TeamCity doesn’t yet have good rvm support. To use an rvm Ruby in TeamCity, one needs to manually set in TeamCity all of the environment variables that rvm would have set. Fortunately Jonah at Carbon Five had already documented them in a nice blog post. With fake rvm in the environment the bundle install step ran without error.
• The other issue which showed up only in TeamCity was that rake db:migrate couldn’t log in to MySQL:

    Access denied for user 'root'@'localhost' (using password: NO)

Rails was finding config/database.yml, but it was only seeing the parameters in that file’s default section, the adapter and host. That’s this bug, and the workaround was just to inline the default section. With that done CI ran green.

Code coverage

rcov runs without error on 1.9.2, but it warns that its results are probably not correct, and indeed GWW was drenched in bogus red. Switching to simplecov was as easy as possible: add one line to spec_helper.rb and bundle exec rake spec instead of bundle exec rake spec:rcov. simplecov puts its results in the same coverage directory as rcov, so nothing else had to change. As a bonus, simplecov doesn’t have a bug that rcov did (it failed to mark complex hash initializers as covered) so GWW’s coverage increased to its rightful 100%.

The benefits

I upgraded GWW to Ruby 1.9.2 not because of any improvement in the new version, but to move to the place in the Ruby open-source ecosystem which will be the best supported for the foreseeable future. Nonetheless 1.9.2 is faster than 1.8.7; GWW’s specs now run in 13 seconds instead of 18, and longer-running processes in production saw a proportional speedup.

Let’s test this simple (non-RESTful) route (in routes.rb):

get 'about' => 'root#about', :as => root_about


There are two things about it to test: that the route connects to the correct controller and action, and that the name refers to the correct route. Stock RSpec tests the wiring like this:

describe RootController do
  describe '#about' do
    it "routes GET /about to root#about"
      { :get => '/about' }.should route_to :controller => 'root', :action => 'about'
    end
  end
end


Aside from the describe blocks (which we’re going to want in any case), that’s three lines of spec, which repeat both the URL and the route. Shoulda’s route matcher is much more succinct,

describe RootController do
  describe '#about' do
    it { should route(:get, '/about').to :action => 'about' }
  end
end


and it still produces a readable specification:

> rspec -f d spec/routing/root_routing_spec.rb
RootController
  #about
    should route GET /about to/from {:controller=>"root", :action=>"about"}


Now for the name. Stock RSpec would test a named route something like this:

describe RootController
  describe '#about'
    it "should have a route named root_about, where e.g. root_about_path == /about" do
      root_about_path.should == '/about'
    end
  end
end


which is three lines for each such spec, repeating the name twice and the URL once. That seems worth a custom matcher. In spec/support/matchers/routing.rb:

module MyApp
  module Matchers
    module Routing

      def have_named_route(name, *args)
        HaveNamedRoute.new(self, name, *args)
      end

      class HaveNamedRoute
        def initialize(context, name, *args)
          @context = context
          @name = name
          @path = "#{name}_path"
          @args = args
          if ! args.last
            raise ArgumentError, 'The last argument must be the expected uri'
          end
          @expected_uri = args.pop
        end

        def description
          "have a route named #{@name}, where e.g. #{example_call} == #{@expected_uri}"
        end

        def matches?(subject)
          @actual_uri = @context.send("#{@name}_path", *@args)
          @actual_uri == @expected_uri
        end

        def failure_message_for_should
          "expected #{example_call} to equal #{@expected_uri}, but got #{@actual_uri}"
        end

        def failure_message_for_should_not
          "expected #{example_call} to not equal #{@expected_uri}, but it did"
        end

        def example_call
          call = "#{@name}_path"
          if ! @args.empty?
            call << "(#{@args.map(&:to_s).join(', ')})"
          end
          call
        end

      end

    end
  end
end


and now it only takes one line to specify the name while still producing the same rspec documentation output as the spec written with stock RSpec:

describe RootController
  describe '#about'
    it { should have_named_route :root_about, '/about' }
  end
end


So the full specification of the route and its name, putting the specification of the name first for a more logical flow, is

describe RootController
  describe '#about'
    it { should have_named_route :root_about, '/about' }
    it { should route(:get, '/about').to :action => 'about' }
  end
end


Not too bad; only two lines, both readable, plus context. Hmm … next time around I might write a chained matcher to do it all in one line and eliminate a level of context, but this is doing the job for now.

Suppose your application’s home page needs to link to an about page. As with any other feature, we write the spec first (here using Webrat):

describe RootController do
  render_views
  describe '#index' do
    get :index
    response.should have_selector 'a', :href => root_about, :content => 'About'
  end
end


Next, implement the view. Perhaps we started with this easy specification and this is the entire contents of index.html.erb:

<%= link_to 'About', root_about %>


Not only is this much nicer than :controller and :action, it relieves us of having to specify twice (once in the template and once in the spec) what the route is to. Duplication eliminated before we’ve even implemented the route! To finish the thought, here’s the actual route, in routes.rb:

get 'about' => 'root#about', :as => root_about


Not a surprising choice, but we might have chosen any URL we liked without redoing any of the work we’d already done, as long as we kept the name.

Of course this applies to all routes, RESTful or not, and it need not apply only to view specs. Use named routes anywhere in your tests that you might have written out the URL. Except in routing specs, of course.

After a little more thought I realized that while it was true that I couldn’t have gotten that specific error past a compiler, and in fact I wouldn’t even have made it in a language that would only consider two methods of two different classes to be ‘the same’ if they implemented a method which appeared in a supertype, that kind of error is only a small subset of a much larger category of errors that can occur in any language. Instead of calling a nonexistent method, the controller might have called the wrong method, or it might have called the right method with the wrong parameters or expected a different range of return value or a different exception than the method actually had. In Java, the controller might have called an overloaded version of the same method. To catch such errors, in a perfect world every method call would be integration-tested regardless of language. What about in our imperfect world?

In an application that’s fully unit-tested (a reasonable and useful thing to achieve even in an imperfect world), every method will be unit-tested with all of the combinations of inputs and outputs that are needed for the application to function, and every method X that calls another class’s method Y will be unit-tested with a mock implementation of Y which accepts a subset of the inputs and returns a superset of the outputs of the real Y, guaranteeing that the real X and the real Y will work together. X and Y must be unit-tested separately to avoid the combinatorial explosion of tests that would be required to completely cover the code if all tests ran against only real implementations, and unit tests of Y must use a mock X to achieve all of the outputs of X that are necessary to test Y and, often, for acceptable test performance.

But in return for these benefits we pay the price of duplication. Tests of X fully specify its behavior, but tests of Y also contain at least a partial specification of X’s behavior. Where there is duplication, copies can diverge, leading to errors. These are integration errors, and only integration tests can catch them.

For an integration test suite to be complete it must cover every method call from one separately unit-tested module to another, not 100% line coverage but still a lot of testing. If the application is developed driven by integration tests, that level of coverage is automatically achieved. But integration tests are generally harder to write and set up and much slower to run than unit tests, so many projects test-drive with unit tests instead of integration tests at least some of the time and compromise on something less than full integration coverage.

Back to the effect of language: A program in any language can be fully unit-tested. What about integration tests? If you commit to full integration coverage, you’ll catch all integration errors, again regardless of language. If you don’t have complete integration coverage, language and framework matter, because different languages and frameworks allow different types of integration errors. How much they matter is a complex qualitative question that can only be answered by experience. Not having any statistics at hand, I’ll just say that I suspect that the answer to the question posed in the title is yes, static typing does reduce the chance of integration errors, but Ruby and Rails’ test-first culture more than makes up the difference, and its practitioners are happy with the tradeoff.

Migrating GWW to the new interface went smoothly, but when I was reviewing some pages for performance recently I noticed ActiveRecord issuing some extra queries that I didn’t expect, and that weren’t necessary. Fortunately, once noticed, they were easy to eliminate.

The extra queries came about because the new query interface issues SQL lazily. If you step through this method in a debugger while watching your test log,

class Photo
  def self.print_unfound_titles
    unfound_photos = where :game_status => 'unfound'
    puts (unfound_photos.map &:title).join ", "
  end
end

you’ll see that unfound_photos is an ActiveRecord::Relation rather than an Array, and it issues an SQL query only at the last possible moment, when map is called.

Lazy evaluation allows you to pass around queries and add to them without running them until you’re ready. But, as in real life, sometimes laziness results in doing more work in the long run. GWW had several situations like this

class PhotosController
  def map
    @photos = Photos.order :dateadded
    first_dateadded = @photos.first.dateadded
    @photos.each { |photo| photo[:color] = color_gradient first_dateadded, photo.dateadded }
    end
  end
end

in which it got a list of something from the database and made some use of the first or last item before iterating over the list. In Rails 3, this issued two queries:

  Photo Load (0.5ms)  SELECT `photos`.* FROM `photos` ORDER BY dateadded LIMIT 1
  Photo Load (0.3ms)  SELECT `photos`.* FROM `photos` ORDER BY dateadded

(Note that ActiveRecord finders don’t belong in controllers. I wrote that here to illustrate my point succinctly.)

The old Rails 2 query, Photos.all :order => :dateadded, had returned an Array, and .first simply returned the first element. In Rails 3, order returned a Relation and .first returned a second, refined Relation. Calling .dateadded caused the second Relation to issue a SQL query, and then iterating over @photos caused the first Relation to issue a second SQL query.

Having noticed the problem, the solution was simple: force the first Relation to run its SQL right away

class PhotosController
  def map
    @photos = order(:dateadded).to_a
    first_dateadded = @photos.first.dateadded
    @photos.each { |photo| photo[:color] = color_gradient first_dateadded, photo.dateadded }
    end
  end
end

thus turning .first into an array reference again and eliminating the LIMIT 1 query.

You can also fully load a Relation by calling .reload on it, and .reload returns self so you can chain method calls, but I find that .to_a better expresses that loading (not reloading) is taking place.

There are quite a few situations in which ActiveRecord loads a Relation only partially, and in which you might want to tell it to load fully. (I say “fully” rather than “eagerly” because the latter refers to the separate decision of whether to load associations.) Looking at ActiveRecord::FinderMethods, we see that some forms of .first load fully and some do not. .last generally works like .first, except of course for its lastness.

A not-yet-loaded Relation will also issue a reduced query when you call .size, .empty? or the blockless forms of .any? or .many?. These methods run a COUNT(*) query. (Oddly, .length always runs the full query; this seems like a bug.) As before, if you’re going to need the full query results anyway, you can eliminate the reduced query by calling .to_a on the Relation right away.

Although it’s good to know the kinds of methods that result in reduced queries, so that you have an intuition about when it might help to eliminate them, it’s probably counterproductive to try to learn all the details of the cases in which ActiveRecord might issue such a query. Furthermore, the implementation of any of those methods might change in future versions. One tactic would be to call .to_a in any case where an application first looks at some property of a query method’s return value and then uses the full query results, but that would be cargo cult programming. Instead, do what you were probably already doing: if the performance of a web page or other ActiveRecord-using process matters, periodically review the SQL that it issues and take whatever steps are necessary to eliminate redundant queries and address inefficient ones. There are plenty of reasons to do that besides the extra LIMIT 1 and COUNT(*) queries discussed here.

Although Rails 3 has been released for more than half a year, upgrading involved many hiccups and a couple of landslides. I’d have been happy to just update my gems and be done with it, but I had to hunt around enough that it seemed worth logging what I did in case it helped someone else. I’ve kept it as short and to-the-point as I could. Although I fixed all of the deprecations that Rails 3 reported as I encountered them, I won’t mention them further, since in every case they were completely clear and fixing them only required following instructions. I’ve also left out a handful of odd little incorrectnesses in GWW which worked in Rails 2 but broke in Rails 3, since the details probably won’t interest anyone else and the fixes were obvious. Otherwise, here’s how the upgrade went.

Background reading

The most helpful blog posts on this topic that I found included Simone Carletti’s excellent post on preparing for the upgrade,  Jeremy McAnally’s post on upgrading, Ryan Bigg’s Rails 3 upgrade log and Kurt Snyder’s summary of changes between RSpec 1 and RSpec 2. Thanks, all.

Before the upgrade

I made sure GWW’s test suite was thorough. Testing routes is critical, since routes.rb changes completely. For an app that uses ERB, tests of page content (GWW uses RSpec-Rails controller specs with, in RSpec 1-speak, integrate_views) are also especially valuable, since the move to Erubis breaks lots of things.

If you don’t already write tests the way GWW’s are written, you probably wouldn’t want to change for the sake of the upgrade alone, but it’s worth mentioning: GWW tests controllers and views in controller specs with (in RSpec 1 terminology) integrate_views. Although these tests had to change (see below), they had to change much less than if they’d been separate tests of controllers and views. That was not at all the point of writing tests that way (it was that I think the interface between skinny controllers and views is an uninteresting implementation detail, not test-worthy behavior), but it was a nice bonus. Also, GWW uses shoulda matchers in its routing specs; as well as being more succinct and readable than the RSpec matchers they replace, they work on Rails 3 unchanged.

I upgraded some components of GWW to what Rails 3 uses, testing and deploying each independently, before upgrading Rails, to make the actual upgrade as small and short as possible:

• I upgraded to Rails 2.3.11. This shook out some things that would otherwise have come up when upgrading to Rails 3.
• I upgraded the mysql gem to mysql2. I’ve lost track of whether it was a recent Rails 2.3.something or Rails 3 that required this.
• I migrated to Bundler. This is a good idea whether you’re planning to upgrade to Rails 3 or not.
• Immediately before upgrading to Rails 3, I upgraded GWW production to Phusion Passenger 3.0.5.

Some things that I didn’t do before the actual upgrade, but should have:

• Migrate to Erubis and the new default of escaping HTML in strings interpolated into templates.
• Migrate from RSpec’s have_tag and have_text matchers to Webrat’s have_selector and contains matchers or some other RSpec 2-compatible equivalent.

I don’t know of a simpler upgrade path from prototype to jQuery than rewriting all of the Javascript in mid-upgrade, which is what I ended up doing.

The upgrade

Install the rails_upgrade plugin

script/plugin install git://github.com/rails/rails_upgrade.git, then rake rails:upgrade:check, which said to do some of the things below and some other things that weren’t relevant or could wait. As mentioned above I didn’t worry about the deprecations until I ran into them.

Create application.rb

rake rails:upgrade:configuration > someplace-out-of-the-way/application.rb

Create the new routes.rb

rake rails:upgrade:routes > someplace-out-of-the-way/routes.rb

The generated routes.rb had several classes of errors which I fixed manually:

• It sometimes said :via => get when it meant :via => :get
• It ignored :only options to resources routes. Fortunately that syntax is unchanged in Rails 3 and I could just copy them over.
• It ignored :conditions options on with_options blocks. I added more :via => whatever as needed.

The generated routes.rb wasn’t well factored, either, but fixing that could wait.

Back up customized files

rake rails:upgrade:backup

Install Rails 3

I edited GWW’s Gemfile to use Rails 3.0.5, rspec 2.5.0, shoulda-matchers 1.0.0.beta2 (it had previously been using shoulda 2.11.3), webrat 0.7.3 and will_paginate 3.0.pre2 and ran bundle install.

Generate a Rails 3 application on top of the existing Rails 2 application

rails new .

I let it overwrite anything it wanted, backing it up manually first if it hadn’t already been backed up by rake rails:upgrade:backup.

Sort out the damage

• I went through all of the files that were modified or not in version control, merging changes from the app’s Rails 2 version and then removing the backup.
• I moved the application.rb and routes.rb created above to config.
• The directory I was in didn’t have the name that I wanted to use for the module containing my Rack application object, so I searched for all of the places where rails_upgrade or rails new . had used that directory name for the module name and corrected them.
• I removed Rails 2′s config/initializers/new_rails_defaults.rb, since the defaults that it sets are now actually defaults in Rails 3.
• I removed config/preinitializer.rb, which integrated Bundler with Rails 2 and is obsoleted by Rails 3′s built-in Bundler integration.
• The only script in the script directory in Rails 3 is rails; I removed all of the others, leftovers from Rails 2.
• Finally, I removed the rails_upgrade plugin, since its job was done.

Start the Rails console

rails c reported some errors, mostly uninteresting. One issue that caused a few minutes of head-scratching at this stage: A bug makes Rails say “Missing helper file” when it should say “required file not found” when loading a helper module with an incorrect require. Having figured out the error message, I didn’t try to figure out why requires that worked under Rails 2 failed under Rails 3; I just fixed them.

After a few rounds of minor fixes, rails c started without errors. I added a temporary fake production database config to config/database.yml and went another few rounds with rails c production, after which it too started without errors.

Configure RSpec 2 and RSpec-Rails 2

I ran rails g rspec:install and moved customizations from the old spec/spec_helper.rb to the new one. Per the instructions, I

• made sure that rspec-rails was in GWW’s Gemfile’s :development group (which it already was for a reason not related to Rails 3 or RSpec 2)
• removed lib/tasks/rspec.rake, spec/spec.opts and spec/rcov.opts
• copied the definition of the spec:rcov task from /opt/local/lib/ruby/gems/1.8/gems/rspec-rails-2.5.0/lib/rspec/rails/tasks/rspec.rake to a new task file, lib/tasks/rcov.rake, and modified it to unload the old spec:rcov task and to exclude some additional files from coverage:
  


  require 'rspec/core'
  require 'rspec/core/rake_task'
  
  Rake.application.instance_variable_get('@tasks').delete 'spec:rcov'
  
  namespace :spec do
    desc "Run all specs with rcov"
    RSpec::Core::RakeTask.new(:rcov => 'db:test:prepare') do |t|
      t.rcov = true
      t.pattern = "./spec/**/*_spec.rb"
      t.rcov_opts = '--exclude /gems/,/Library/,/usr/,lib/tasks,.bundle,config,/lib/rspec/,/lib/rspec-,spec/,buildAgent/plugins'
    end
  end
  

Run the specs, one layer at a time, and fix issues

rake spec:lib reported errors related to a use of ActiveSupport::TestCase.use_transactional_fixtures. It wasn’t essential (GWW’s specs other than model specs turn off transactions just to reduce noise in the RSpec log) so I commented it out and rake spec:lib passed.

rake spec:routing failed, complaining of “undefined method `route'“. It seems that current versions of shoulda don’t integrate quite correctly with RSpec; the solution was to add config.include Shoulda::Matchers::ActionController to spec/spec_helper.rb. That fixed most of the routing tests. The remaining handful of failing routing tests were due to errors in the new routes.rb. (Was I glad I’d tested every route!) After fixing those rake spec::routing passed.

rake spec:models failed with the error “undefined method `add_limit!'” in methods using will_paginate’s paginate_by_sql. That was easily fixed by upgrading will_paginate to 3.0.pre2. (I lied above; I didn’t figure out that I needed that upgrade until this point.) That was all it took to get rake spec::models to pass.

rake spec:helpers took only a little more effort:

• Helper specs that set request parameters through the magic params variable had to be changed to use the new magic controller.params variable.
• Different helpers with methods with the same name suddenly found themselves in conflict. GWW had only a few such methods, so the easy solution was to rename some of them to eliminate ambiguities.
• Custom matchers and other modules that were included in spec/spec_helper.rb with the :type option failed to load when :type was an array of spec types. Breaking each such include into one line per spec type fixed that. For example,

config.include Photos, :type => [ :helper, :controller ]

became

config.include Photos, :type => :helper
config.include Photos, :type => :controller

Making rake spec:controllers pass took by far the most effort of any step so far. First, the easy parts:

• RSpec 1′s integrate_views is render_views in RSpec 2, a quick search-and-replace.
• Each partial called with :object (which defines a local with the name of the partial) no longer saw that local. I’m not sure whether this was an intentional change in Rails 3 or not, but it was easy to name the local explicitly.

And then the big part, which was to config.include Webrat::HaveTagMatcher and rewrite all of the specs that used RSpec’s have_tag matcher to use Webrat’s have_selector and contains matchers. I referred to this nice cheat sheet by Wincent Colaiuta.

The updated specs found one more problem: Since Rails 3/Erubis now escapes all expressions in templates by default, expressions which returned HTML needed to be wrapped in raw().

With the controller specs rewritten, rake spec:controllers and rake:spec passed. Weirdly, rake spec:rcov failed; although it had not been necessary for rake spec:models to pass, rake spec:rcov needed config.include Shoulda::Matchers::ActiveRecord in spec/spec_helper.rb.

Manual testing

Now that all of the specs passed it was finally time to rails s GWW and look at it in a browser. It came out from under the bandages in great shape, with only a few blemishes that needed correcting:

• A few template expressions that didn’t want to be escaped and hadn’t been caught in tests needed to be wrapped in raw().
• The ID of the form created by the button_to helper had changed from button-to with a hyphen to button_to with an underscore, so some CSS and Javascript had to be updated.
• A popup that issued an AJAX request no longer worked. Rails complained that it didn’t know how to respond to a request for application/json. Amusingly, the action emitted HTML, not JSON; a copy-and-paste error had made the requesting Javascript Accept: application/json. Rails 2 never noticed, but Rails 3 did, and refused. The fix was just to remove the bogus parameter.
• Some of GWW’s HTML needs to be exact about its newlines, since it’s pasted into Flickr discussions where newlines are significant. Erubis ignores whitespace around <% %>. Where GWW’s ERB templates intentionally used <% %>, not <% -%>, to insert a newline, I needed to add a blank line.
• The other side of the coin: Putting <% -%> at the end of a line in an ERB template suppresses the newline, but that doesn’t work in Erubis. Fortunately <%= -%> does work, so I just needed to replace each <% -%> with <%= -%>.

Just one more thing …

The very last thing that had broken was autocompletion in two text fields. Autocompletion had been done by a venerable plugin which was last updated in 2007 and didn’t work with Rails 3. There seemed to be no replacement which used prototype; the leading autocomplete plugin uses jQuery. Well, the Rails team is backing jQuery these days (and who isn’t?) and I’d have migrated soon enough anyway. I installed jquery-rails and rails3-jquery-autocomplete, installed jquery-rails per the instructions (with --ui), picked a jQuery UI CSS file from jqueryui.com, used the new autocompleter on the autocompleted fields, and rewrote all of GWW’s custom Javascript to use jQuery instead of prototype.

After 2 1/2 days of work, production GWW is on Rails 3. There’s still some followup to do: refactor routes.rb, make use of Arel, take all of the now-redundant h‘s out of the templates, etc. And then back to features!

Another reason I kept the Treo as long as I did is that I wasn’t looking forward to the migration. I’ve used Palms for twelve years, and I intend to keep my data. I expected it would be a significant project to move it all over. It wasn’t as bad as I feared –  Palm Desktop was prepared to bow out gracefully, in that it could export all of its data into standard or at least reasonable formats — but it took enough experimentation and fiddling that I’ve written it down here on the off chance that it will help someone else who still has this job ahead of them.

Source: Palm Treo 650 running Palm OS 5.4.8, synced to a Mac OS 10.4 laptop running Palm Desktop 4.2.1. A Mac OS 10.6 laptop pitched in too.

Target: Droid X running Android 2.2.1.

Contacts

Android’s stock Contacts application syncs with GMail contacts, so I saw no reason to look further. Palm Desktop can export contacts as vCards and GMail can import that format, so migrating from one to the other was relatively easy.

1. Export contacts from Palm Desktop into vCard format. To preserve groups, export one group at a time, each into its own file. For example, save the Home group as home.vcf.
2. Comments on Palm contacts become X-Palm-Custom# lines in vCards, where # is 1, 2, 3 etc. GMail ignores these. Convert them to NOTEs, which GMail does understand, by running them through the following script. For example, palm-vcf2gmail-vcf home.vcf > home-gmail.vcf.
3. Import each file into GMail, creating a new group for each file.
4. Go to Android Contacts and there they are, in the Google category. Sadly, the groups which I was so careful to create don’t appear on the Droid. Oh well; I can maintain them in GMail and if Android Contacts ever adds that feature I’ll be ready.

The script, palm-vcf2gmail-vcf:

#!/opt/local/bin/ruby
ARGF.each do |line|
  puts line.gsub /^X-Palm-Custom\d/, 'NOTE'
end


Hmmm; I could have done that with ruby -n. Next time.

Calendar

As with contacts, Android Calendar is probably adequate for me and syncs with Google Calendar. Google Calendar imports iCal and and Outlook CSV. Palm Desktop doesn’t export either of those, but it does export vCal, and the standard Mac OS iCal application imports vCal and exports iCal.

1. Export the calendar from Palm Desktop, saving it as e.g. calendar.vcs.
2. Import calendar.vcs into iCal, telling iCal that it’s in the vCal format. I imported into the newest version of iCal I had, 4.0.4, running on a different, newer laptop than the one running Palm Desktop, which had iCal 2.0. The two iCals export quite different iCal files, but I don’t know whether using the newer iCal made any difference.
3. Examine the results in iCal. Fix any problems in the Palm’s Calendar app, resync, re-export from Palm Desktop and re-import into iCal (in a new calendar, to avoid confusion with previous attempts). Repeat until everything that can be fixed on the Palm has been. Here’s what I had to fix:
   • Who knows why, but I had a handful of events which repeated, but for only one day. On the Palm these look like non-repeating events; in iCal they repeat forever. I just found each of them in the Palm and set them to not repeat.
   • I didn’t have many events with notes, because it was usually easier to just put all of the text in the event, but I had a few notes on events from before I settled in to my current usage. I just appended each note that I wanted to keep to the text of its event and deleted the note. Fortunately there weren’t many and none were long. If you need to do something different, you’ll find event notes in your memo export file (see below). No idea how to keep them attached to their events.
4. Fix any problems in iCal that can’t be fixed on the Palm. I had one such problem: repeating no-time events that began between early November and early May turned into series of two-day events in iCal, each beginning one day too early. It didn’t take too long to fix these manually.
5. Export the calendar from iCal to a file, e.g. Calendar.ics.
6. Import the file into Google Calendar. The only trick here is that Google Calendar allows you to import only 5000 events per day. I had just under 5000, so I could do it once, but when I had to fix something and do it again I had to wait a day!

I immediately missed Palm Calendar’s easy navigation. Google Calendar is better with two Labs add-ons: Year View and Jump to Date.

Afterthought: I might also have synced iCal to Google Calendar. Export and import was the first thing I tried and it worked, so I didn’t look further.

Memos

Android has no standard to-do application (curse the slack-jawed target user, consuming and never producing!). I went with Note Everything, which is free, widely used and not too much more ambitious than Memos, which is all I want for now. Note Everything imports a CSV format exported by Palm Desktop for Windows, but Palm Desktop for Mac OS doesn’t export that CSV format, so I needed to convert to it myself.

1. Export all memos from Palm Desktop to the “tab and return” format. Export only the title and body fields.
2. Convert tab-and-return to CSV using the following script, e.g. tab-and-return2note-everything all.tab-and-return > all.csv.
3. Connect the Droid to the Mac with its USB cable and copy all.csv to the Droid.
4. In Note Everything, go to Menu -> More -> Import -> Import from Palm memos and import the file.
5. The script puts all memos in the Unfiled group, since that’s where I kept all of my memos in the Palm Memo app. Now that group is useless, so move all of the memos manually from Unfiled to “Main folder” and delete Unfiled.

The script, tab-and-return2note-everything:

#!/opt/local/bin/ruby
# Converts Palm Desktop's "tab and return" export format to the CSV format that
# Note Everything expects when "importing from Palm memos".
tar = ARGF.read
lines = tar.split "\r"
lines.each do |line|
  # Skip notes attached to calendar entries
  next if line =~ /Handheld Note: Date Book/
  line.gsub! "\246", "\n"
  line.gsub! "\t", "\n"
  line.gsub! '"', '""'
  puts "\"Unfiled\", \"#{line}\""
end

Tasks

As with Tasks, Memos has no standard Android replacement. Astrid seems to be the only third-party to-do app which is relatively mature and can import from a file, and then only with the not-free Power Pack add-on. Like other third-party to-do apps, Astrid can sync with Google Tasks and other task managment web applications, but neither Google Tasks nor the other web apps I looked at can import tasks from a file, so that doesn’t help.  The only recourse was to reverse-engineer Astrid’s backup format, which, fortunately, is just simple XML.

1. Export To Dos from Palm Desktop in Tab & Return format. Include only the Category 1 and To Do columns, in that order. Save as, for example, to-do.tab-and-return.
2. Convert to Astrid backup format with the following script, e.g. palm2astrid to-do.tab-and-return > user-110314-1600.xml.
3. Copy the fake backup to the astrid directory on the Droid.
4. In Astrid, go to Menu -> Backups -> Import Tasks and import the fake backup.

The script, palm2astrid:

#!/opt/local/bin/ruby

def escapeHTML(string)
string.gsub! '&', '&'
string.gsub! '<', '&lt;'
string.gsub! '>', '&gt;'
string.gsub! '"', '&quot;'
end

tar = ARGF.read

puts <<EOS
<?xml version='1.0' encoding='utf-8' ?>
<astrid version="174" format="2">
EOS

created = Time.now.to_i

tar.split("\r").each do |line|
line.gsub! "\255", '\''
category, text = line.split "\t"
escapeHTML category
escapeHTML text
puts <<EOS
<task calendarUri="" completed="0" created="#{created}" deleted="0" details=" |  | &lt;img src='silk_tag_pink'/&gt; #{category}" detailsDate="#{created}" dueDate="0" elapsedSeconds="0" estimatedSeconds="0" flags="0" hideUntil="0" importance="2" modified="#created" notes="" postponeCount="0" recurrence="" notificationFlags="6" lastNotified="0" notifications="0" snoozeTime="0" timerStart="0" title="#{text}">
<metadata created="#{created}" key="tags-tag" value="#{category}" />
</task>
EOS
end

puts <<EOS
</astrid>
EOS



Note that this script will probably not work with versions of Astrid other than 3.7.2, which is what I copied the backup format from.

Later I ran across the Got To Do Android app, which syncs with the ToodleDo web app, which can import from a tab-and-return export of Tasks from Palm Desktop. The job is done and I’m happy with Astrid for now so this will remain the road not taken.

PalmReader documents

Palm Memos only handled memos up to some not very large size, and it wasn’t able to search within a memo, so I had a collection of larger text documents which I’d converted into Palm Reader format and read in Palm Reader. The situation on Android is the same: by experiment, Note Everything can handle documents up to a size of about 341Kb; it truncates larger documents on import. For documents below that size, I just copied the plain-text originals to the Droid and imported them into Note Everything using its file-at-a-time and directory-at-a-time import.

That left a couple of larger documents. I copied the plain-text originals of those to the Droid, too, but I read them with iReader, which handles documents up to at least 3-something Mb.

Ringtone

I last added a ringtone to the Treo in 2006 using phonezoo.com, and it was still there for me. I just told Phonezoo what kind of phone I had now and sent myself “Huxley Beagle Howling”.

I recently added a new model type and set of page to GWW — it now explicitly tracks ScoreReports — and took the opportunity to begin using RESTful routes. It worked beautifully; I really appreciate being able to lay down a whole set of standard, named routes with so little code. But when I pushed the score reports tracking feature to production, it broke: creating a new score report (done by pushing a button to submit a form) did nothing, only refreshed to the index view with no new score report at the top. After a little head-scratching and watching the form submission with Live HTTP Headers, I realized that, like seemingly every only-in-production bug, the problem was due to caching.

In production, GWW runs under Apache httpd and Phusion Passenger, which despite the bug I mentioned have been rock-solid. Many of GWW’s pages take a little time to generate, so to speed things up most of them are cached using Rails’ page-caching mechanism. So that cached pages don’t litter the public directory as they would in the default configuration, GWW puts them in public/cache with this line in config/environments/production.rb:

config.action_controller.page_cache_directory = File.join(RAILS_ROOT, "public/cache")

Passenger needs no configuration to serve cached pages from the default location in public, but having moved them they must be redirected to manually in public/.htaccess:

RewriteEngine On

# If REDIRECT_STATUS is empty (not set), this request comes directly from a
# client, not from an internal redirect following one of the rewrites below.
# In that case, forbid URIs which begin with "cache" so clients can't request
# cached pages directly from the cache directory.
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^cache - [F,L]

RewriteCond %{DOCUMENT_ROOT}/cache/index.html -f
RewriteRule ^$ cache/index.html [L]

RewriteCond %{DOCUMENT_ROOT}/cache%{REQUEST_URI}.html -f
RewriteRule ^(.+)$ cache/$1.html [L]

This worked fine until GWW acquired its first RESTful route.

Under the old Rails routing conventions (or at least GWW’s interpretation of them), the URI from which one would get a list of score reports would be something like /score_reports/list and the URI to which one would post to create a ScoreReport would be something like /score_reports/create. With RESTful routes, both actions use the same URI, /score_reports, one with HTTP method GET (exercised by clicking on a link in a browser) and one with HTTP method POST (exercised by submitting a form).

The bug had been in the caching configuration all along; it was just exposed for the first time by adding to the application two routes that differed only in HTTP method. The redirect to a cached page wasn’t considering the HTTP method. GETting /score_reports in the browser cached the page, then POSTing to /score_reports caused httpd to return the cached page before the request ever reached Rails.

Once understood the problem was easily fixed by adding a condition to rewrite only GETs:

RewriteEngine On

# If REDIRECT_STATUS is empty (not set), this request comes directly from a
# client, not from an internal redirect following one of the rewrites below.
# In that case, forbid URIs which begin with "cache" so clients can't request
# cached pages directly from the cache directory.
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule ^cache - [F,L]

RewriteCond %{REQUEST_METHOD} GET
RewriteCond %{DOCUMENT_ROOT}/cache/index.html -f
RewriteRule ^$ cache/index.html [L]

RewriteCond %{REQUEST_METHOD} GET
RewriteCond %{DOCUMENT_ROOT}/cache%{REQUEST_URI}.html -f
RewriteRule ^(.+)$ cache/$1.html [L]

With this configuration in place POSTing to /score_reports routes to ScoreReportsController.create as intended. In web applications in general it never makes sense to cache POSTs, PUTs or DELETEs, so GWW’s caching configuration would certainly have been this way in the first place if it had occurred to me. Oh well; better late than never.

Now that the first set of RESTful routes is working, I’ll look forward to converting more of GWW to use them. The tighter routes.rb is, the less work there will be to do when GWW moves to Rails 3.