MOVED

FYI – I’ve been posting new content at spraints.github.com. I was holding off on posting this here until I customized the UI some and maybe put it on a custom domain. That’s not happening in the near future, so, for now, head on over and enjoy my latest posts!

 

The Drunken Wave

If Ruby5 can have tales of drunkenness, so can my ruby blog.

On Sunday, on the way back home from town, we were driving down a four-lane city street. The guy just ahead of us in the other lane was swerving towards oncoming traffic. Every time a car would come the other way, he would drift across the yellow line. Anticipating a collision that would send his truck rebounding into my lane, I slowed and left plenty of space. This foresight proved prudent when the road narrowed to two lanes, as he swerved at the last second into my lane.

I continued to leave plenty of space, about 150 yards, which brought out the impatience in the drivers behind me. Two such drivers passed on the now dark country highway, only to find obviously-inebriated-dude just ahead. The driver-who-shouldn’t-have-been pulled part way off the road and stopped (on a state highway!) to let the rest of us pass. I tried to slow down enough that he’d go on without me passing, but he waved me on. (This was the wave that is featured in the title of this post.)

I reluctantly passed, and traffic ahead of me slowed. I slowed, and saw the lights behind me accelerate. Great, I thought, now I’m going to get rear-ended. I pulsed my brake lights, and that got his attention so he slowed. The cars ahead sped up, then slowed again. Repeat the pulsing lights to encourage Mr. Tipsy to slow down. At the first side street, I turned off the highway, turned around, and got back on the highway. This was plenty of time for the subject of the story to pass.

At the next major intersection, I was committed to turning a different direction from the other traveller. He went straight, down a gravel county road (county road 50 S), which suited me fine, as I was turning right (onto highway 29).

The rest of the story made the paper. Fortunately, nobody suffered permanent injuries.

 

Add a primary key to a has_and_belongs_to_many join table

This is a not that common of a problem, though there are other posts on the topic. But my solution’s a little different from what I found, so I figured it was blog-worthy. I used this in a rails 3.0 app. I’ve also only tried it with sqlite, so it may totally blow up when I push to heroku. YMMV.

Take, for example, some initial tables:

users – id:primary_key
replies – id:primary_key
liked_replies – user_id:integer, reply_id:integer

and some initial models:

class User < ActiveRecord::Base
  has_and_belongs_to_many :liked_replies, :class_name => 'Reply', :join_table => 'liked_replies'
end

class Reply < ActiveRecord::Base
  has_and_belongs_to_many :likers, :class_name => 'User', :join_table => 'liked_replies'
end

And, of course, there’s existing data that you don’t want to destroy.

So, I generated a migration:

bundle exec rails g migration add_id_to_liked_replies id:primary_key

If you’re curious, the migration looks like this:

class AddIdToLikedReplies < ActiveRecord::Migration
  def self.up
    add_column :liked_replies, :id, :primary_key
  end
  def self.down
    remove_column :liked_replies, :id
  end
end

Migrate as usual, and then the app stops running, because routes can’t be built, because User is a devise model, and devise_for tries to load User which has an association that generates one of these:

Primary key is not allowed in a has_and_belongs_to_many join table (liked_replies). (ActiveRecord::HasAndBelongsToManyAssociationWithPrimaryKeyError)

Hm. This will probably happen when we deploy, too. And this makes a db:rollback fail, too. !!!! So, how can we make the code work with or without a primary key? I’ve seen lots of people talk about making your code work with the before & after version of your db, and that seems like the right goal here, too.

Here are my new model classes that work with either the new or old liked_replies table:

class User < ActiveRecord::Base
  begin
    has_and_belongs_to_many :liked_replies, :class_name => 'Reply', :join_table => 'liked_replies'
  rescue
    has_many :liked_reply_records, :class_name => 'LikedReply', :dependent => :destroy
    has_many :liked_replies, :through => :liked_reply_records, :source => :reply
  end
end

class Reply < ActiveRecord::Base
  begin
    has_and_belongs_to_many :likers, :class_name => 'User', :join_table => 'liked_replies'
  rescue
    has_many :liked_replies
    has_many :likers, :through => :liked_replies, :source => :user
  end
end

class LikedReply < ActiveRecord::Base
  belongs_to :user
  belongs_to :reply
end

 

Polymorphic rabl children

So, say you’re in a fantasy world, and the rif-raf is beating down your doors…

Or, maybe, you’re trying to write a sane json rendering of your model, and the model has a polymorphic belongs_to.

As a bit of review, let’s start with the easy case. You have some models with a simple association.

class Child < ActiveRecord::Base
  belongs_to :parent
end

class Parent < ActiveRecord::Base
  has_many :children
end

The rabl for a child is pretty easy. In app/views/children/show.rabl you might have:

object @child
attributes :id, :name
child(:parent) { partial "children/parent" }

As it turns out, you’re actually writing an app that’s used to track orphans left behind after UFO visits, so you need to support children of aliens, too. So your models look more like this:

class Child < ActiveRecord::Base
  belongs_to :parent, :polymorphic => true
end

class HumanParent < ActiveRecord::Base
  has_many :children, :as => :parent
end

class AlienParent < ActiveRecord::Base
  has_many :children, :as => :parent
end

And of course you need to output different information for alien parents (app/views/children/alien_parent.rabl) than for human parents (app/views/children/human_parent.rabl). So your rabl in app/views/children/show.rabl will look more like this:

object @child
attributes :id, :name
code(:parent) { |child|
  partial "children/#{child.parent.class.name.underscore}",
   :object => child.parent
}

 

Faster rubygems with squid

I’m giving squid a spin as a local cache for gems. My goal is to speed up the time git clean -whatever ; bundle install --path .bundle takes.

This solution isn’t perfect. It doesn’t make the bundle install as fast as I’d like. Also, squid needs to be restarted (sudo launchctl stop squid) when I change locations and/or sleep my laptop, because it forgets how to look up DNS (it seems to be caching the DNS server list).

I’m using homebrew and on a Mac running Snow Leopard, so this will be most helpful for anyone running the same.

Install squid

brew install squid

This installed squid 3.1.9.

Configure squid

Open /usr/local/etc/squid.conf, and uncomment the cache_dir line.

Start squid

First, you need to initialize squid. Some of this stuff may be equivalent to throwing your machine under a bus, so please be aware of what you’re copying.

chown nobody /usr/local/var/cache
chgrp everyone /usr/local/var/logs
chmod g+w /usr/local/var/logs
sudo squid -z

I wanted squid to run any time the machine is running. So, I created /Library/LaunchDaemons/squid.plist to look like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>squid</string>
    <key>OnDemand</key>
    <false/>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/sbin/squid</string>
      <string>-N</string>
      <string>-d 1</string>
      <string>-D</string>
    </array>
    <key>ServiceIPC</key>
    <false/>
    <key>RunAtLoad</key>
    <true/>
  </dict>
</plist>

Start it by doing sudo launchctl -w /Library/LaunchDaemons/squid.plist

Configure bundler/rubygems

Open ~/.gemrc and add:

http_proxy: http://localhost:3128

My .gemrc looks like this:

---
gem: --no-ri --no-rdoc
http_proxy: http://localhost:3128/

 

Bootstrap a rails app without installing the gem

For a while, I’ve been keeping bundled gems local to each project. (It’s pretty easy to do: bundle --path .bundle/gems the first time you bundle a project. I like putting it in .bundle because it’s already gitignored.) This results in a lot of duplicate gems across my system, but it has the advantage of keeping each project very self-contained, and I avoid undesirable side-effects (like rake getting upgraded) just because I bundle some code from github.

This leads to a pretty sparse set of system gems. But it also makes it a little challenging to bootstrap a new rails project, since there isn’t a current rails executable! How to avoid installing the latest rails to start a new project?

It’s pretty easy, when you think about it, but it is a little less straightforward.

mkdir -p ~/my/new/awesome_project
cd ~/my/new/awesome_project
echo source :rubygems            > Gemfile
echo gem 'rails', '~>3.1.0.rc4' >> Gemfile
bundle --path .bundle
bundle exec rails .

It’ll ask if you want to overwrite Gemfile, which is, of course, fine, because it’s just going to add more gems.

 

rake db:schema:load test

Don’t do it.

And not just because you should be using rspec.

rake db:schema:load test does this:

** Invoke db:schema:load (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:schema:load
** Invoke test (first_time)
** Execute test
** Invoke test:units (first_time)
** Invoke test:prepare (first_time)
** Invoke db:test:prepare (first_time)
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment 
** Execute db:abort_if_pending_migrations
** Execute db:test:prepare
** Invoke db:test:load (first_time)
** Invoke db:test:purge (first_time)
** Invoke environment 
** Execute db:test:purge
** Execute db:test:load
** Invoke db:schema:load 
** Execute test:prepare
** Execute test:units
** Invoke test:functionals (first_time)
** Invoke test:prepare 
** Execute test:functionals

If you look closely, you’ll see that db:schema:load is in there twice. The first time, it’s loading into your dev environment. The second time, it’s loading into your test environment. Or, tries to: rake notices that db:schema:load was already run, so it skips it. You know, unnecessary extra work.

I am most likely to rake db:schema:load test (or spec) on a CI server, sometimes with a db:migrate mixed in. One-liner ci scripts are nice. But keep db:schema:load and test separated.

If you have db/schema.rb in source control like you should, and you keep it up to date, you still need to db:schema:load to avoid the ‘abort if pending migrations’ thing: rake db:schema:load ; rake test

If your db/schema.rb might be out of date, rake db:schema:load db:migrate ; rake test

If you’re a glutton for punishment: rake db:migrate test

 

Running jenkins on a mac, from homebrew, as a daemon

So, what if you want to run jenkins locally on a mac? What if you want it to build from a private github repository, and you want it to use its own private key? And you’re on a mac?

Get Jenkins

First, you need to get jenkins. On my homebrew-enabled mac, I ran brew install jenkins. Without homebrew, you can just download the latest war file. (You will need java 6 to run jenkins.) Homebrew provides a plist for launching jenkins when you log in. If that’s adequate, then you can stop reading.

Create a service account

Next you need to create a user to run jenkins. This is the most challenging part of the process. (I used a script from a pastebin as reference.) Here’s what I did:

sudo mkdir /var/jenkins
sudo /usr/sbin/dseditgroup -o create -r 'Jenkins CI Group' -i 600 _jenkins
sudo dscl . -append /Groups/_jenkins passwd "*"
sudo dscl . -create /Users/_jenkins
sudo dscl . -append /Users/_jenkins RecordName jenkins
sudo dscl . -append /Users/_jenkins RealName "Jenkins CI Server"
sudo dscl . -append /Users/_jenkins uid 600
sudo dscl . -append /Users/_jenkins gid 600
sudo dscl . -append /Users/_jenkins shell /usr/bin/false
sudo dscl . -append /Users/_jenkins home /var/jenkins
sudo dscl . -append /Users/_jenkins passwd "*"
sudo dscl . -append /Groups/_jenkins GroupMembership _jenkins
sudo chown -R jenkins /var/jenkins

Before I ran through that, I did check that the ids were available. You can search for users with dscl . -search /Users uid 600 and groups with dscl . -search /Groups gid 600.

Also, I don’t think the uid and gid need to be the same, but most of the builtin service accounts (e.g. jabber) are, so I just went along with it.

Create the daemon

Mac OS uses launchd to control daemons and agents. It’s pretty easy to create a launch daemon. Create the file /Library/LaunchDaemons/org.jenkins-ci.plist with the following content, based on the plist from the homebrew jenkins formula. You may need to update the version number in the ProgramArguments.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>Jenkins</string>
  <key>ProgramArguments</key>
  <array>
    <string>/usr/bin/java</string>
    <string>-jar</string>
    <string>/usr/local/Cellar/jenkins/1.414/lib/jenkins.war</string>
  </array>
  <key>OnDemand</key>
  <false/>
  <key>RunAtLoad</key>
  <true/>
  <key>UserName</key>
  <string>jenkins</string>
</dict>
</plist>

I think I had to load the daemon (sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist). Rebooting should work, too.

Create an ssh key

Like I said, I wanted jenkins to have its own ssh identity. This is fairly easy: sudo -u jenkins ssh-keygen

The new key is in /var/jenkins/.ssh/id_rsa.pub and can be copied to github, or wherever you have your source code.

Set up a build

Now you need to configure jenkins. Open http://localhost:8080/, click “Manage Jenkins”, “Manage Plugins”, then “Available”. I installed the git and github plugins. The git plugin gives you basic git functionality. The github plugin gives you links from the build info pages to github commit pages. After the install is complete, click the “Schedule a restart” button.

After jenkins restarts, configure git. From the “Manage Jenkins” page, “Configure System” and make sure the path for git is right. (It wasn’t for me: /usr/bin/git is the default, but homebrew put it in /usr/local/bin/git.)

Now, you can create your project. The rest of this post is a description of how I set up the build for my rails 3.0 project.

Click “New Job” in the menu on the left. Choose the “free-style” option. On the configuration page, set the github url prefix (e.g. https://github.com/user/project). Set the source code as git, with the repo url (e.g. git@github.com:user/project.git; if your url starts with git:// or http://, then you probably don’t need to do the ssh key stuff earlier). I set branches to build to “master”, but leaving it blank is sometimes useful. I chose to Poll SCM, with a schedule of “* * * * *”, though I didn’t set that up until after I had the build mechanics set up right.

The only build step is to execute a shell. Here’s what I’m using. There are a few things to note here. I’m using a system-installed rvm, so all the ruby commands are run through it. Jenkins can collect JUnit test results and produce some trend graphs, so I use the ci_reporter gem’s ci:setup:testunit to format the results JUnit-style.

/usr/local/rvm/bin/rvm 1.8.7 exec bundle --path .bundle/gems

cat <config/database.yml
development:
  adapter: sqlite3
  database: db/development.sqlite3
test:
  adapter: sqlite3
  database: db/test.sqlite3
END_CONFIG

/usr/local/rvm/bin/rvm 1.8.7 exec bundle exec rake db:bootstrap db:migrate ci:setup:testunit test --trace

 

git-tfs 0.11.0 Release Notes

I just released git-tfs v0.11.0.

There are two new commands in this release: bootstrap and verify. bootstrap will find TFS commits and configure git-tfs remotes for you. verify checks that the latest TFS-related commit has the same content as the corresponding version in TFS.

There are a number of bug fixes:

• Correctly handle the “no newer checkins on the server” case for VS2010. (commit)
• Work on x86 and x64 more often (commit)
• Allow checkin policy overrides (commit)
• Generate a default checkin comment (commit, commit)
• Ensure consistent casing in the new repository (commit)

You can see the full diff on github.

 

bootstrap, checkin policies

Today I pushed two new git-tfs goodies: bootstrap, and checkin policy override.

checkin policy override

The checkin policy override support is pretty straightforward. If you try to check in to TFS…

git tfs checkin

…and it fails with some messages about checkin policies (e.g. no associated work items, or the code analysis policy can’t run), you can now override the policy failures. Of course, the best fix is to comply with the policy. For example, if you need to specify an associated work item and provide a checkin comment:

git tfs checkin -w 12345 -m "My awesome code has an awesome checkin comment."

But, if you really need to override the policy failures, you can now do it:

git tfs checkin -f "Policy override because of X" -m "Normal checkin comment."

Of course, you can use checkintool to do all this in a GUI.

bootstrap

The other change was the addition of a bootstrap command. This is useful if you create a TFS clone and share it with a colleague who then needs to interact with TFS. While two identical invocations of git tfs clone will produce identical repositories, git clone is always going to be faster than git tfs clone. So, I would guess that most people who want to collaborate on a TFS project using git will benefit from this command.

The old workflow for this was:


[user1] git tfs clone http://blah/blah/blah $/blah
[user1] cd blah
[user1] git remote add shared git@someplace:shared/repo.git
[user1] git push shared master
[user2] git clone git@somplace:shared/repo.git
[user2] cd repo
[user2] git tfs init http://blah/blah/blah $/blah


At this point, the users can collaborate with each other using git, and they can both do TFS checkin or fetch. For the best workflow, both users need to type in the exact same path. An extra ‘/’ or a capitalization change will keep git-tfs from matching up the TFS remotes, and it will refetch things it doesn’t need to.

So, the bootstrap command replaces the last ‘git tfs init’:

[user2] git tfs bootstrap

This will scan HEAD’s history for checkins with TFS metadata, and configure one or more TFS remotes to match. If you already have the remotes configured, it will just tell you what it found.