Captain Kerbal

Category: Coding

SSS mk.2 – Possible architecture

The aims of the proposed successor of SSS, SSS mk.2 (or another name…), are detailed in this post.

Currently, I visualise it something like this:

Applicator:

Generates a state from a set of plaintext files, checks for deviations from the state, and fixes them as required. Kind of like Make, but more specialised.

Syncer:

Syncs the state files across computers. The best method would see to be to use Git over SSH – the advantage over immix is that I can have multiple branches.

Timer:

Runs the syncer and applicator every few hours/minutes/seconds. This could be as simple as a script hooked up to a systemd/launchd/init service file, or a cron job.


The most complex part is the applicator. Generating the state from a set of plaintext files would be challenging, so would checking for deviations and applying them.

The state would probably be defined in a set of plaintext files, with the target state declared within.

For instance, to make sure some packages were installed, the file would include:

installed_packages = lynx,vim,fbpdf,other

Or something like that. Add in if statements, include statements, variables, and some commands, and the resulting basic declarative language should be able to define the target state, including understanding the host OS and changing accordingly (launchd vs. systemd service files, for instance), the current architecture, the specifics of the host system, and the hostname (in essence, the computer ID). I will look to see whether my chosen language has a library that could handle this part for me, but otherwise I think it should be within my grasp to create something that worked.

One additional challenge would be some sanity checking – checking that a package was not both required and forbidden, for instance. I’m not sure where that would fit, but it would probably be the most challenging task.

Checking for deviations would be hard, but not insurmountable. For instance, depending on the package manager, the command for checking for a program being installed would vary. On Mac OSX, the simplest method would be to try to find the program binary, since there is not native package manager, and resort to macports or homebrew as required for installing. Being able to define behaviour – like how to check that a package is installed – in the declarative language would be a must. Actually checking the state should be reasonably easy – run through each of the proposed state variables and check with the method defined for that particular system/architecture/OS whether they are satisfied. After creating a list of unsatisfied substates, the checker could pass the list on to the applicator.

Applying the unsatisfied substates would be reasonably easy. Like checking for deviations, provided the way to resolve the issue was defined somewhere – how to change the relevant config file, how to install packages, how to add a user – SSS mk.2 would just have to apply that method, doing something if it failed (printing an error, logging, messaging the user, or similar).


Some expected challenges include:

  • Local vs global changes. On computers that only I use, I’d want the changes to be global. However, on shared computers – such as the Mac I occasionally have to use – the changes would have to be local only, and because I don’t have super user rights, I’d have to install software locally, as well as changing the configuration locally.
  • Depending on how much my chosen language handles low level details, writing a powerful declarative language for the configuration could be challenging. So much so, that I might need to consider an alternative method. I hope not… in Python it would be reasonably easy!
  • Working on computers with different directory structures. My computers all run Arch Linux, so they are easy. However, sometimes the directory structures can be radically different. I’d have to be careful to make sure all locations were not hard coded and instead referenced back to an OS-specific config file. Especially if I ever do manage to make a specialised distro for the Toshiba.
Advertisements

Happy Holidays

Holidays are nearly here!

Well, almost. Several weeks away, in fact. But I’m looking forward to them 🙂

I have been doing some planning, however. The plan so far involves:

  • Creating the iMac case
  • Working on my syncing system, specifically the mk.2 version
  • Bringing cling to a state of basic completion
  • Work on the specialised distro for my Toshiba

Of course, there are other goals as well, but they are outside the scope of this blog 🙂

I’ll see how far I get…


I’ve also managed to make some progress on cling. To date, I’ve added the very basic functionality, and have cleaned it up. The next steps are to finish cleaning the current code up, write some tests, write some CLI binaries using libcling to test that I have the basics covered, and then release 1.0!

Good progress so far 🙂

SSS, mk.2 – Magic syncing x2

The replacement date for SSS looms on the horizon – whenever I get around to it. So, it looms somewhere over the horizon… maybe I am just experiencing a Fata Morgana 🙂

The aim of SSS mk. 2 is to fix the deficiencies of mk. 1: namely, the use of pacman and PKGBUILDs for a job they really aren’t designed for.

The detailed aims are mostly the same as those for SSS mk. 1, with a few changes:

  • Fast – written in Go, C, C++, or a similar compiled language. Not python, because it needs to run in the background on limited hardware like my Toshiba.
  • Automated – automatic syncing of the state files, like with immix.
  • Manages packages and configuration – what SSS does currently 🙂
  • Flexible – can handle my varied computers, and maybe different operating systems? (Mac OSX? Linux distros?).
  • Configured by plain text files stored somewhere obvious (/var/lib/<syncing_system> or similar).
  • Handles roll outs easily – something SSS does well at the moment :). Probably more complex than SSS, but I could make a script that automated most of it. Hopefully.
  • Manages patches to files – configuration changes. Because the configuration files are different, a simple patch system might not work. I’ll have to consider something else.
  • Handles more complex changes – like adding new users.

Possible extra goals:

  • Automatically handles ‘global’ activities – like daily updates.
  • Messages me if something goes wrong, like an update fails, or conflicting things need merging by hand…
  • Looks after disk usage, makes sure all of the services are running as expected.

The possible goals may be better handled by another daemon, or a helper script.

We’ll see what happens…

SSS – automagic syncing

I thought I’d finally post on my automated syncing system (from now on know as Snazzy Syncing System, or SSS).

The initial idea of the scheme came about when I was given my PC. Instead of managing both the PC and my Pi, manually updating the configuration on each one, and adding new packages if I decided to swap, say, from ‘aspell’ to ‘hunspell’, I decided that I’d create a system that would do that automatically!

Why not?

The initial scope for the project was quite extensive, because it needed to handle at least 2 computers with very different hardware and configuration. I looked at using solutions such as Puppet, Bcfg2, and Chef – however, they seemed overly complex for my limited needs, and more suited to larger groups of computers than I’m ever likely to have. I also needed a lightweight solution if I ever wanted to run it on my Pi – an interpreted solution wasn’t ever going to cut the mustard.

I eventually settled on a simple (ish) solution – using PKGBUILDs that depended on packages to ensure that they were pulled in as required, and a sub-system to manage my changes to configuration files.

Using this system, all I need to do to update is run

pacman -Syu

Provided my custom repo is up to date, any changes I have made will be pulled in. If I don’t want my Pi to have a desktop environment, I make a separate PKGBUILD, and only install it on the computers that need it. Combined with immix, which uses Unison to sync my files across all of my computers, I have a system which achieves the basic goals, fairly simply – right?


Well, not really. Pacman (rightly) refuses to overwrite configuration files, so I had to write patchman, and get each configuration file patched at install time. This added extra complexity to the install file, which also has to handle other, more complex tasks.

I am overusing pacman and PKGBUILDs.

For the moment, that is OK. I have a working system, that provides easy rollouts and reinstalls if required, and updates my systems to be consistent (most of the time).

However, it’s not perfect – so I will be writing a replacement at some point. I was thinking that it would be a good introductory project to learn Go with…