Add release integration test

[dsherret]

I broke the last release on mac. There should be some way to test this in integration. No, I didn't break the build. I was just accidentally running an old release that was broken.