A Bazaar branch of Drupal HEAD with all history

I originally created a Bazaar branch of Drupal HEAD with hourly snapshots of upstream updates (commits to CVS HEAD) to streamline work on patches to Drupal HEAD.

This snapshot method had a few advantages over using Launchpad’s Drupal branch:

  • Because of the shallow history, branching from the Four Kitchens server was relatively fast compared to branching from Launchpad.
  • The Four Kitchens server has more reserve bandwidth and capacity, making initial branching and updates faster.
  • Launchpad goes down more often than the Four Kitchens server.

The snapshot-based branch also had a big disadvantage: it didn’t keep a true commit-by-commit history that mapped to CVS HEAD commits, making it hard to understand ongoing changes. After some discussions with chx, I’ve decided to combine the best of both worlds: a mirror of Launchpad’s Drupal HEAD branch on the Four Kitchens server with instructions (provided below) to avoid the trouble of pulling thousands of revisions every time you branch.

Why not branch directly from Launchpad? Aside from occasional Launchpad reliability issues, downloading all Drupal revisions takes takes 2m37s from Launchpad versus 1m15s from Four Kitchens (both tested from my home connection).

I will also continue to maintain the old snapshot-based branch until Drupal 7 is released.

Method 0: Plain, old branching

  • Pros
    • Runs with older Bazaar versions
    • Performs all post-branch operations locally with no network access
    • Easy to set up
  • Cons
    • Downloads all upstream revisions each time you branch (takes 1m25s)
    • Stores all upstream revisions each time you branch (takes 51M each branch)

Run this: bzr branch bzr://vcs.fourkitchens.com/drupal/7-all-history

Method 1: Stacked branches

  • Pros
    • Fast initial branching: only downloads basic branch data (takes 42s)
    • Most space efficient: only stores basic branch data (takes 6.6M total)
    • Easy to set up
  • Cons
    • Requires Bazaar 1.6 or later
    • Requires internet access to perform most history-based operations

Run this: bzr branch --stacked bzr://vcs.fourkitchens.com/drupal/7-all-history

Method 2: Shared branch storage

  • Pros
    • Runs with older Bazaar versions
    • Downloads upstream revisions once for all branches
    • Stores upstream revisions once for all branches (takes 51M once)
    • Performs all post-branch operations locally with no network access
  • Cons
    • Still stores upstream revisions once
    • Still downloads all upstream revisions once
    • You have to run a few more commands to create your first local branch

From a new directory that will be a parent of your future branch directories, run this:
bzr init-repository .

From within the parent directory you just created, run this to create your local branch:
bzr branch bzr://vcs.fourkitchens.com/drupal/7-all-history

Using your branch

Presumably, you’ve gone through all this to work on patches for Drupal HEAD. I’ve updated my earlier instructions for how to use your new branch.

Commenting on this Blog post is closed.

Comments

Just a quick post to tell you that your articles on using Bazaar with Drupal are greatly apreciated.

I’m from a designer’s background and have been trying to teach myself how to handle versioning of my “solo” projects and bazaar has, for now, been the easiest to grasp (I’ve looked at subversion and a bit at git).

Although I do a fair bit of coding, versioning has always been somewhat obscure.

I hope you continue to post simple worflow ideas and best practice articles.

Thank you.

I’m glad to hear my posts have been helpful. :-)

I’d also appreciate any questions or suggestions you have for future posts. I’ve spent a lot of time figuring out the best ways to do things with Bazaar, and I’d be delighted to share more of them.

I have many questions, but I guess it all relates to best practice and common workflows.

I basicaly use bazaar and it works for my needs as of now, but I have no one or no other source (except your posts in the past weeks) to compare what I do and validate my ways of doing.

Since I’m not a “programmer” per say (I’m more of a montrous hybrid between a designer and a programmer) and I think I’d like to see concepts/workflows explained in the context of an actual or fictional project case study. Its always trying to grasp how the concepts lead to actual usage that is the hardest to bridge.

Another, more to the point question, is the handling of database data for versioning purpose. I do not do any as of now and wonder if I should and if so, how do I include it in my versioning workflow.

Thank you again for taking the time to write these.

I think following the version-control process for an example project would be a great series of posts. But presenting the story in coherent order would be important, so I’m going to wait until I’ve “solved” a few more Drupal workflow problems in Bazaar and set up a bit more Bazaar infrastructure on the Four Kitchens servers for things like modules and themes.

Versioning the database is typically not done using standard version-control tools because database dumps are often large. Moreover, existing version-control tools are not designed to intelligently detect the way changes are reflected in SQL dumps. For example, row order for INSERTs in SQL dumps is typically insignificant, but version-control “diff” tools will consider them significant, causing an absurd number of spurious “changes” in the revision logs.

I have also enjoyed reading your recent Bazzar posts. One thing I haven’t seen covered is an explanation of why you choose Bazzar over the other distributed version control systems available (eg. Mercurial, Git, etc.). I currently use Subversion for all of my version control (with the exception of CVS for drupal.org related stuff) and I suspect that my workflow could be improved if I learned to use one of the newer systems, but since I don’t have a good feeling of which one is “better”, I haven’t used any of them. I’ve seen feature comparisons on Wikipedia, for example, but for the most part those aren’t that helpful to me because I’m not that familiar with the distributed VCS systems work and how some of those features would be useful.

I realize that “better” is relative to the individual, but it seems that you must have decided that Bazzar was the best for you, and I think it would be interesting to find out why.

After trying a few things using the branch you provide on your server, I’ve come up with a question I’d like to ask.

First, let’s say I branch from the drupal 6 branch you provide and then make a local branch of that branch for a, say, projectX.

In projectX’s branch, I would like to remove some of Drupal’s .txt files and some themes I do not want to use before before publishing the said project (keeping only garland). Once I’ve done that and try to merge projectX with drupal 6 branch, I get conflicts for having missing files/folders. I guess the answer is do not remove those files, but I was wondering if there’s any way to have this kind of workflow and have bazaar know it doesn’t need to care about those folder’s when merging, merging only the change in the remaining folders.

I wonder if this make sense…

Thank you.

Getting a stacked branch does not work. Do you know of this already?

bzr branch --stacked bzr://vcs.fourkitchens.com/drupal/7-all-history 7s
_bzr: ERROR: Received bad protocol version marker: 'Source repository format does not support stacking, using format:\n Packs 5 (adds stacking support, requires bzr 1.6)\n'_

I’m 99% sure I got the same error on the ‘6’ branch yesterday, but right now that error is gone. So you might be upgrading your bzr repositories/server, and I’ll keep it short :-)

(I made the mistake of longevity already at https://bugs.launchpad.net/bzr/+bug/196080 - but only discovered later that wat I reported, wasn’t generally true…)

We almost always run the latest stable Bazaar on our server with the default repository format. What client version are you running? Several stacked branching bugs were fixed in 1.15 and 1.16.

I’m using version 1.16.1.

Since I totally did not understand why this bug would occur (and not be reported on Launchpad) I dug around a lot, and I ended up testing 1.16.1, 1.13,
1.6.1 and 1.5.

What’s even weirder is that getting a stacked branch from the ‘6’ repository now does not work anymore, and did not work earlier, but it did work when I was typing my reply here about 16 hours ago, multiple times!?

bzr branch --stacked bzr://vcs.fourkitchens.com/drupal/6
_bzr: ERROR: Received bad protocol version marker: 'Source repository format does not support stacking, using format:\n Packs 5 (adds stacking support, requires bzr 1.6)\n'_

Feels like I’m going mad.

I’ve moved to using non-stacked branches so I’m fine. But if there is something which we can do to debug this, I’m up for it. Unfortunately I can’t reproduce this on my own server (running bzr serve 1.16.1)

Indeed, our source repository is not using Packs 5. (We’re using the default, pack-0.92.)

I’m going to check with the other Bazaar developers about this issue. I don’t think it should require an advanced server repository format to use stacking on the client.

I’m getting the same error message with a plain bzr 1.17 as well.

I think I’ll just upgrade the storage version on the server. There aren’t many folks out there with Bazaar 0.92 anymore.

Actually the problem is mentioned here:
 https://bugs.launchpad.net/bzr/+bug/196080/comments/2

We are running a Bazaar “smart server” via xinitd. Upgrading our repository version may not solve the issue., but I will do so anyway and upgrade Bazaar, too.

Thanks for your efforts; but it’s still reporting the same error message.

Here’s me again. While merging the ‘6’ tree (with the recent security release) I got this message.


bzr@d01:~/drupal/6-downloads$ bzr merge
Merging from remembered parent location bzr://vcs.fourkitchens.com/drupal/6/
bzr: ERROR: Received bad protocol version marker: 'Doing on-the-fly conversion from to (remote).\nThis may take some time. Upgrade the repositories to the same format for better performance.\n\n'

I already worked around it - but still wanted to notify.

Because together with the weird error message (and resulting discussion) above, it occurred to me… Maybe this does not signify bzr bugs (or incompatible bzr formats) at all. Our bzr clients just freak out because they get in-itself-harmless text messages.

Maybe your bzr+xinetd configuration is sending stuff that should go to STDERR, to STDOUT instead? (i.e. over the line where bzr is negotiating a ‘protocol version’, but instead gets some warning/notification text. And therefore panics & dies.)

(edit: my own branch is now 2a, and I tried this with bzr 2.0.0 and 2.1.0.)

Indeed, it’s nothing to worry about. It’s just Bazaar warning you that it’s slow. We keep the existing Pressflow branches (5 and 6) in a format compatible with Bazaar 0.92 and newer. This older format is slower for Bazaar 2.0+ users. Pressflow 7 will probably use the 2.0/2a format.

Indeed, it’s nothing to worry about.

I’m not good at stressing the important point, apparently

It’s not ‘just a warning’. Bzr quits with this “ERROR: …” message. It does not work. The bzr client just dies, because it gets info/warning strings on the channel where it expects ‘protocol negotiation feedback’.

And this is why I, and everyone else whose configuration causes info/warning messages from your server, is unable to use your repository for updating/merging. Since july or before. (As also apparent from Konstantin’s messages.)

Bazaar 2.0 has been out for long enough at this point. I’m upgrading the branches hosted on the Four Kitchens server right now.

Just an FYI, I am still receiving the ‘bzr: ERROR: Received bad protocol version marker’ error when trying to pull recent updates from your repo.