What’s on our mind: Further automation of Open Fifth Koha system upgrades

IN THE BEGINNING, THERE WAS THE COMMAND LINE

(With apologies to Neal Stephenson)

“Back in the day”, before we started looking at automation, we had three ways of building Koha systems: Git Standard (the Git installation, tuned for production), Git Development (the Git installation, tuned for development) and the Debian Package (a package for production, built in the same way as the underlying operating system expects – which is Debian)

This was fine for a while, but when we had The Move from a hosting partner we no longer use. We took the opportunity to move everyone to Debian Packages – by that time, we had better ways of doing development testing (Koha Testing Docker!) and it allowed us to standardise things, which was important as The Move showed us just how slightly different most Koha installations had been in the past; typically down to the whims of whomever had built the server at the time. They were almost, but not entirely, the same. Just ‘not entirely enough’ to be a problem for migrating. So we cleaned that up as we went along…

This by its nature helped with systems stability and supportability a lot, as with all servers built the same way, with customisations in the same locations, and so on, everything got nice and consistent!

While doing this, we also learned how to use Ansible, which is a method of ‘roboticising’ human interactions on the server. So instead of us typing in a series of esoteric commands to move a Koha installation from one location to another – synchronising config files and database backups, installing Koha, configuring it to suit etc., Ansible does all that for us. 

We write in the commands we want it to use, we give it the responses we want it to give to those commands, and we let it run. Ansible ‘types’ faster than us, it doesn’t have fat finger moments, it doesn’t get distracted by phone calls, etc.; all realistic potential hangups and slowdowns when working. So we didn’t do it, we let Ansible do it.

The Revelation

We realised quite early into The Move that we could apply this to the upgrades process as well – when we drop you an email and schedule that once a year, four-hour window when we (typically) upgrade your operating system, and upgrade the major Koha version (i.e. 23.11 to 24.11) which you need to schedule downtime for. 

Although it would take some time to work that part out. So once we were happy The Move was going as well as could be expected and things were stabilised process-wise, we took some time; in a hotel in the Wirral, three nerds loaded up on coffee and bar snacks got that working at a basic level, then started refining and refining and refining.

The final refinement has been using it in production – as is always the case, to some degree! This process has now been ‘battle hardened’ for the last year; almost every 23.11 to 24.11 and Debian 11 to Debian 12 upgrade in the last year has been done by what is basically a very fancy, very flexible script that we run by hand, but that once it starts, it carries on by itself. Yes, there is more to Ansible than that, but for the purposes of non-systems admin types, that’s basically how that works – it does the ‘boring’ legwork for us.

So, now that we know it works, and works reliably, rather than running it by hand at 8am on a given day, we’re scheduling it to run at 11pm, so that your servers are ready to be used at 8am the next day. This significantly minimises downtime for you, the actual user of the system.

We’re also refining our comms on the subject a little, too – we’ve previously been perhaps a bit too communication-y, with multiple emails and chains throughout the process, documents to read, and so on. We’re now (and this is still being refined, and so is subject to change from feedback, which we’ve already had!) using a slightly lighter set of comms, we’ve moved the testing suggestions and new features to the Open Fifth website and feedback we’d like through the helpdesk with a ticket subject that starts with ‘Upgrade Feedback’ – basically we’re trying to make things just less of a chore overall, and bring the upgrades process from a procedural standpoint to be less ‘special’ and more ‘everyday support’.