Alieniloquent

I just wanted to poke my head in and say that I’m still around, and still working on/thinking about the erlang project. I am, after all, still giving the talk next month.

I am, however, getting super busy with school. So, I’m having to sit down and schedule all of my non-work programming time. It’s looking like the erlang project isn’t going to get much love until after the end of September.

That said, when I get a few free moments this week, I’ll post some code I’ve already written that makes a nice interface around shelling out from erlang.

It was suggested that I use CouchDB as the storage engine for blogerl. I’m not going to be using CouchDB for this project, and I would like to explain why.

My reason for choosing Git over CouchDB is really just a preference. I think Git is neat, and I want to use it in a novel way. I’d like to demonstrate how it is a very powerful tool and can be used to do more that just version software. I think CouchDB is cool, and at some point, I’ll probably do a project with it. But, for this project, it’s not what I want.

In my previous post, I discussed the goals and priorities for blogerl. This time I am going to brainstorm some stories and prioritize them. Then I’ll select the first few from the prioritized list to be my first iteration. The goal is to have something I can deploy so I can get the blog started as soon as possible.

My philosophy for stories is to give them titles that are short. The story names are simply mnemonics to assist with recalling conversations and other details. Since I’m a team of one, the only conversations will be those I have here with you. But after this post, I’ll probably just refer to the stories by their title.

Here are the stories in priority order:

Storage

I need some way to store my posts. I’m not very choosy about how this is done. But, because I am in an experimental mood, I’m going to opt for using Git. I’m probably going to end up using Grit via erlectricity. There’s already been a wiki built on top of Git, why not a blog?

RESTful interface

Like I said before, I don’t want to compose my posts in a <textarea> but I do need to be able to get content into my application somehow. So, I’m going to design a RESTful interface to my data store. This is going to have to include an authentication mechanism, since I don’t want just anybody to be able to update or edit my blog.

Emacs mode

My preferred program for editing anything is Emacs. So it seems only natural that I’d choose to write my first REST client as an emacs mode.

Index

A front page that shows the posts in reverse chronological order. Should be paginated with 10 posts to a page.

Single

Permalinked pages for each individual post.

RSS

Add an RSS feed for the main index.

Atom

Add an Atom feed for the main index.

Archives

Provide archives based on year and month.

Templates

Templates for each of the views. Not YAWS pages. This will involve creating my own template language. I don’t like ErlTL for this as it is too programmery.

Caching

Keep rendered content in a cache with a TTL. Expire the cache for the index page if a post is added, edited or deleted. Expire the cache for a single post if it is edited or deleted.

Tags

Be able to specify an arbitrary number of tags to associate with a post. Provide archive pages for each tag.

Comments

Provide comments. They do not need to be threaded. Plain, flat comments are sufficient.

Trackbacks

Provide the ability for other blogs to post trackbacks.

Post-dating

Give the ability to submit a post that will be published at a later date.

Markdown

Add support for formatting a post in the Markdown formatting language.

Textile

Add support for formatting a post in the textile formatting language.

Formatting default

Add a configuration option specifying the default formatting (which starts out as plain HTML).

The first six are going to be what I aim to complete before I tag an 0.1 and deploy to my webserver. That’ll give me a means to share my blog posts with people as a stream or individually via their browsers or via RSS feeds.

First off, I’ve decided on a name for my project: blogerl. I will be hosting the source code here on GitHub.

With that bookkeeping out of the way, I’ll get to the meat of the post. I want to include you in my brainstorming process as I figure out what my goals are with this project. In my next post I’ll brainstorm features that will help me meet these goals and select a subset of those features to implement initially.

Before I can start brainstorming features, I need to figure out an overall vision for what I am trying to build. So the first thing I want to describe is the primary goal of this system, and possibly some secondary goals as well.

I want a system that will manage the storage and presentation of my blog content. That content will be primarily textual, and may be annotated with various pieces of meta-data (e.g. date, title, or tags).

That’s pretty vague, and I can design several extremely different systems that will deliver on that, so I’m going to provide some additional goals, in priority order, to narrow the design down.

Adding or editing content should be easy.

So many blogging tools are a pain to use. I’ll be honest, I’m not the biggest fan of the web application. Especially not for things like writing. I intend for Omniloquent to frequently have essays, and I don’t really like doing massive amounts of writing in a <textarea>.

Viewing the website should be fast.

This isn’t usually a problem in most modern blogging tools. However, I’m writing this one from scratch, so it’s going to be a little less modern at first. I don’t want people to be sitting around waiting for my content to load. I want it to be snappy. The closer to static content it feels, the happier I’ll be.

Adding or editing content should be fast.

The converse of the previous goal: I don’t want posting to take forever. This is why I hate Movable Type. The idea that I should have to rebuild dozens of pages when I just update a single post is ridiculous. However, I am willing to suffer a little, as I recognize that I’d rather it take me a second or two to post but have the blog be lightning fast for my readers. That said, the closer to instantaneous posting I can get, the happier I’ll be.

On October 14th, I will be giving a talk at ODYNUG about advanced erlang. Unlike my previous talks, this one is going to delve into the code of a real application. I’ll be discussing the architecture of the application and the reasons behind the design decisions.

Of course, in order to do that, I need an application. More specifically, I need an application for which I know all the reasons behind the design decisions. So, I’m going to write one, and I’m not a person to buck a tradition, so I’m going to write a blog.

I’ll be documenting the process here, and you can see the blog live here.

Hi. I know it’s been a long time. Things have been crazy-busy for me all summer, and as usual, the blog fell by the wayside.

One of the things keeping me busy has been my new job. Yes, I got another new job. Since June, I’ve been working at Engine Yard as a developer on the Vertebra project, and recently I’ve taken over as development lead. It’s very exciting and fulfilling to be working on such an interesting problem.

I’ve also been speaking and traveling quite a bit. I spoke at both BarCampKC and BarCampOmaha, and my usual user group talks.

But, the real reason I haven’t written since March, is that I just haven’t had ideas about which I wanted to write. I finally decided that I needed to start writing even if I couldn’t think of anything to write about. Of course, at that moment, an idea occurred to me.

I’m going to be starting a series of posts over the next month. I’m working on a project for my Advanced Erlang talk next month at ODYNUG, and I’m going to blog about it.

I came across this amusing (fictional) story about a broken watch, and it’s impact on the author’s development process.

I particularly liked one of the author’s comments in the comment thread:

Velocity is a measurement that comes out of development, not an input that goes into it.

[Uncle Bob] said . . . that when you are trying to measure something, it’s counter-productive to put pressure on it.

I’ve really fallen head over heels in love with Git. But my original solution was really a hack. There is a better way to do it. In fact, it’s so much better, it comes with Git.

I took a look at git-svn when I was researching this a couple weeks ago, and the trouble that I had was that it didn’t fetch externals. Rather than figure out the problem, I just moved on with the working solution that I had. But, it bothered me. So I continued to research, and sure enough, Git has a way to do it just fine.

Nazar Aziz over at Panther Software posted an excellent guide for setting up a Rails app with plugins using git-svn and Git submodules. I am going to distill it and add a few notes about how to make your use of Git as unnoticeable to the other SVN users as possible.

Step One: Clone your externals

Git submodules are fantastic, but to use them you need Git repositories for each of your externals. Fortunately, you can easily clone them with git-svn. First, to list your externals:

$ svn propget svn:externals http://example.com/svn/app/vendor/plugins
foo_plugin  http://example.com/svn/foo_plugin/trunk

Now you should make a directory to put your clones of these in.

$ mkdir ~/Projects/plugins

And then cloning them is as simple as this:

$ git svn clone http://example.com/svn/app/foo_plugin/trunk ~/Projects/plugins/foo_plugin

Step Two: Clone your SVN repository

The next step is to clone your repository sans-externals. We’ll use git-svn to do that, but we’ll use it in a slightly different manner. The Git folks recognize that there is a standard layout for SVN repositories. If you tell it where the trunk, branches and tags are kept relative the the URI you provide, it will try to preserve that information. It makes branches for each of the SVN branches, and it makes branches for the tags as well.

Just do this wherever you want your project to live. You may want to rename any SVN working copies you have so that there aren’t any naming conflicts.

$ git svn clone http://example.com/svn/app -T trunk -t tags -b branches

That’ll give you a git repository named “app” in the current directory. The master branch will be a remote tracking branch that is set up to track trunk, and other branches are set up for any tags and branches.

Step Three: Hook up the submodules

Now that we’ve got Git repositories for all of the plugins, and a Git repository for our project, we can hook everything up. We will set up a submodule for the external we cloned above.

From within the top-level of your project repository do this:

$ git submodule add ~/Projects/plugins/foo_plugin vendor/plugins/foo_plugin

After you’ve added the submodule do this:

$ git submodule init
$ git submodule update

That should get the code from the plugin repository and into your project repository just like the external did.

Step Four: Cover your tracks

When you are using git-svn it commits all of your Git commits into SVN, and you don’t really want to commit anything into your Git that you don’t want finding it’s way into SVN (at least not on the branch that you commit to SVN from). But it is easy to set Git up to ignore all of the files.

First, let’s make sure Git ignores all the same things SVN was ignoring:

$ git svn show-ignore >> .git/info/exclude

Then open up .git/info/exclude and add these lines to it:

# .git/info/exclude
.gitignore
.gitmodules
/vendor/plugins/foo_plugin

That should prevent you from committing anything into SVN that is git-specific.

Step Five: Using this thing

So once you’re all set up, you’ll want to be able to interact with the SVN repository. Here are your two basic operations.

Update from SVN

$ git svn rebase

This works just like git-rebase, except it pulls from SVN instead of some other Git branch. It will not work if there are changes that have not been committed to Git. What it does is roll back all of the changes since the last time, and then update from SVN, then reapply the changes in order. If there are conflicts, you resolve them as you would if you were using git-rebase.

Commit to SVN

$ git svn dcommit

This will take all of the commits since your last time and commit them one at a time to SVN. This allows all those people still using SVN to see each individual commit instead of one monster commit.

I recommend using SVN to do anything more involved than simple adds, removes, renames and edits.

Something to be aware of with this set up is that your submodules are effectively frozen at whichever revision you cloned. If you want to update them, you’ll need to first update the cloned repository, and then run this command at the root of your repository:

$ git submodule update

Another caveat is that you need to keep your development as linear as you can. Don’t try to do anything crazy with lots of branches and merges between them. SVN can’t really make sense of it. The big deal here is you want to use git-rebase to pull in changes from SVN.

Here’s my workflow. I use a branch named work to do all of my work in. I will sync it up with SVN several times a day, just so it isn’t too stale. This is how I do that:

$ git checkout master
$ git svn rebase
$ git checkout work
$ git rebase master

Then, when I’ve commited all of my changes to my work branch, and I’m ready to commit to SVN:

$ git checkout master
$ git merge work
$ git svn rebase # Just to be safe
$ git svn dcommit

It works well, and it allows me to do my work disconnected from the network.

I love ruby, and I love Rails, but in some ways it really is a ghetto. It has a lot of broken windows that only serve to encourage bad coding from developers who should know better. Today I ran into an example of one of those broken windows and I was beside myself. I could not believe what I was reading.

One of the projects I work on for my employer is an import process that takes a long time. In order to make it resilient to database fail-overs, I wanted to catch the exception that is raised when the connection dies, wait a few seconds, and then try to reconnect. The idea is simple, and it works once I account for the broken window, but I am not pleased with the code I had to write.

When the database connection disappears, the database driver throws an exception. ActiveRecord::Base catches that exception and does this:

# Find this in Rails 2.0.2
# active_record/connection_adapter/abstract_adapter.rb:121

rescue Exception => e
  # Log message and raise exception.
  # Set last_verfication to 0, so that connection gets verified
  # upon reentering the request loop
  @last_verification = 0
  message = "#{e.class.name}: #{e.message}: #{sql}"
  log_info(message, name, 0)
  raise ActiveRecord::StatementInvalid, message
end

This is the exception handler that catches all exceptions raised during a query run by ActiveRecord. As you can see, it snags the class name, and the exception message off of the exception, and then throws the object away, reraising with ActiveRecord::StatementInvalid. So, if your database driver has hundreds of error codes which are provided in order for you to tell specifically what error occurred, such as Mysql::Error, you lost them.

So ActiveRecord provides one exception that covers everything from primary key violations to database connection errors, and the only way to distinguish them is by inspecting the message. Surely, that can’t be true, right? I dig further and find this:

# Find this in Rails 2.0.2
# active_record/connection_adapters/mysql_adapter.rb:244
#
# Note: I snipped the error message because it is very long

rescue ActiveRecord::StatementInvalid => exception
  if exception.message.split(":").first =~ /Packets out of order/
    raise ActiveRecord::StatementInvalid, snipped_error_message
  else
    raise
  end
end

That is just completely unacceptable. I can find it in my heart to forgive the abstract adapter for doing something that throws away implementation-specific information, but the Mysql adapter should remedy that. It willingly lets it’s exception information be cast aside and goes about inspecting what the abstract adapter had the decency to keep around.

“But that information is good enough to tell what the exception is,” you might say.

Until the Mysql folks change the error message. The Mysql API exposes numeric constants, and I’m sure they’re very careful to keep them the same, but do you think they take the same approach to error messages? I doubt it. They provide a function that will give you an error message given the numeric constant, and encourage you to use it. That’s what the Mysql bindings for ruby do.

Expecting developers to inspect the exception message is essentially promoting programming with magic numbers. Sure, they’re string literals, but they’re still duplicated information, and extremely brittle.

All I’d want is an inner_exception attribute available on ActiveRecord::StatementInvalid or maybe its parent, and then assign it when doing reraises. Is that too much to ask for?

Yesterday I wrote about using SVN and Git together to have version control away from the network your SVN server is on. Now, I’ll admit, I wrote that shortly after figuring it out and doing it. So, at that time, I hadn’t actually come back into the office and merged my changes with the repository and committed them. Having done that a couple of times now, I’m here to say that this setup is fantastic.

So using this system you’re either at the office, so you can use SVN, or you’re not, so you have to use Git. I’ll give steps to follow for each. You should probably read the documentation available from the Git website to familiarize yourself further with these commands.

These steps assume that you’ve made a Git branch using the following command:

$ git branch home

Taking your work home: SVN -> Git

Make sure your SVN working copy is as up to date as you want it. Ideally, commit any changes. But, if you’re in the middle of a change set, that’s fine.

$ git commit -a -m "Merging in changes from SVN since last commit"
$ git checkout home
$ git merge master

Now you’re ready to use Git to continue making your changes while you’re away from the office.

Getting back to work: Git -> SVN

Make sure your Git changes are all committed to the home branch.
$ git checkout master
$ svn up

Resolve any conflicts from SVN.

$ git merge home

Resolve any conflicts from Git.

Now you can continue with your changes using SVN, or commit them right away if they’re already perfect.