Last modified 6 years ago Last modified on 26.10.2011 22:55:12

cronbuilder

Version 0.7 is out

Minor release, incorporating new command line option helping in source repository migration.

Version 0.6 is out

From now on, cronbuilder supports multiple instances on the same destination repository.

Additionally, Subversion support (1.6 and 1.4 versions) is fixed, and SSL support is improved by adding config options to specify X.509 CA certificate for both SRC and DEST repositories.

Version 0.5 is out

New release incorporates working copy locking, to prevent overlapping jobs in case the build takes longer than cronbuilder's run interval.

Version 0.4 is out

Initial public release 0.4 is ready for download and build.

Description

In sysadmin's life there is often a need for fetch-generate-save task running from crontab. Such task is composed of three steps:

  1. fetch

Sources (templates of some kind, possibly with parameters) are taken out of some repository. Generally it's a good idea to have sources in a repository of versioning system of some kind, e.g. svn(1) or git(1).

You can think of this as of configuration templates for a set of servers.

  1. generate

Sources are used for producing some other files. You can think of this step as of generating configuration files for specific server from templates.

  1. save

Files produced in step 2. should be saved somewhere. Again, it's a good idea to have them in a repository (not necessarily in the same as in step 1., though).

cronbuilder is designed for making tasks like this easier to set up. Not that the task is difficult. It's just annoying to re-implement fetching from one repository and saving in another repository, especially generating useful commit messages, handling repository access credentials or not sending an e-mail when everything went fine, but nothing was generated.

cronbuilder takes care of pretty much everything. The only things you need to specify, are a) where to get sources (URL + credentials), b) where to save generator output (URL + credentials) and c) how to produce output from sources (build command).

You don't need to bother with not producing output in build command -- quite the opposite! This output will be useful in case of build error.

cronbuilder will send you an e-mail (well, cron(8) will) each time when a problem appears and each time when changes were saved in destination repository (this one is configurable). If nothing new was generated, you won't get an e-mail. As quiet as possible, but not more.

How to download

git clone http://dozzie.jarowit.net/code/cronbuilder.git

How to install

NOTE: Installation process is tested for building packages. If you want to make mess in your system by omitting package system, you are on your own.

You need casual package building tools for your distribution, either rpm-build (RPMs) or dpkg-dev with fakeroot (DEBs).

Red Hat

  1. Prepare source RPM (non-root privileges are fine here)
    make srpm
    
  2. Build binary RPM (unless you've taken care of building as non-root, you need to be root here)
    rpmbuild --rebuild cronbuilder-*.src.rpm
    
  3. Install package with its dependencies (exact path should be printed by previous step); most probably it will be located in /usr/src/redhat/RPMS/noarch/cronbuilder-*.rpm

Debian

  1. Build binary package
    dpkg-buildpackage -b -uc
    
  2. Install package (../cronbuilder_*.deb) with its dependencies

How to use

We'll need two repositories: for sources and for generated files. I recommend to use git for the former and Subversion for the latter, but that's just my preference. For now, let's stick with two Subversion repos:

svnadmin create /somewhere/subversion/sources
svnadmin create /somewhere/subversion/generated

Now, prepare environment for cronbuilder.

mkdir build-env
cat > build-env/cronbuilder.yaml <<EOF
source:
  type: svn
  url: /somewhere/subversion/sources
destination:
  type: svn
  url: /somewhere/subversion/generated

build:
  command: /bin/true
  rebuild: false
  inform: true
EOF

We don't want to specify build command yet.

If your source (or destination) repository is accessible via HTTP(s), you need to add source.user with source.password in build-env/cronbuilder.yaml.

Now try if cronbuilder can actually check out repositories:

cronbuilder --no-build build-env

No build is performed, so nothing is saved in destination repository, but source and destination working copies are prepared.

It's time to develop some build method. Personally I recommend makefile, and this is what we'll do now.

cat > build-env/source/Makefile <<'EOF'
DESTDIR = /tmp

.PHONY: build
build: lorem-ipsum.txt
	cp $^ $(DESTDIR)

lorem-ipsum.txt:
	echo "Lorem ipsum dolor sit amet, consectetur adipiscing elit." > $@
EOF

Remember to indent rules with tabs instead of spaces (it's a make's quirk).

Our build command will be "make -C $SOURCE build DESTDIR=$DESTINATION". Update build.command option in build-env/cronbuilder.yaml.

Check if the command actually works:

cronbuilder --debug --no-checkout build-env

--no-checkout is to keep your local changes (new file Makefile) in source working copy.

You should see something like this:

Build successful (revision <uncommitted>).
Source revisions <none>..0

Change summary:
  - No previous build.

Files affected:
  file-list-unknown: ~

##############################################################################
# build log

make: Entering directory `.../build-env/source'
cp lorem-ipsum.txt .../build-env/dest
make: Leaving directory `.../build-env/source'

##############################################################################
# commit log

<Commit was skipped due to command line option>

Save Makefile in source repository (note: just Makefile; lorem-ipsum.txt is generated, and thus unnecessary in sources) and remove directories build-env/dest and build-env/source (cronbuilder will recreate them).

Last thing you should do is to run cronbuilder as it would be in crontab (".../build-env" means absolute path to your build environment):

cronbuilder .../build-env

This will generate first automatic commit in destination repository. You can put this command to crontab.

Actually, you don't need to run this command manually just to see the first autocommit. You can wait for cron to do that for you as well.

From now on, you can add new files and rules to the sources as you need.