Using LXC virtualization for testing server installation

In last article Ubuntu 18.04 and unprivileged LXC I showed how to setup unprivileged containers. The reason why I needed that was that testing of server installation.

At work, we had 64 steps long list of installation instructions (packages, Apache, Tomcat, PostgreSQL config) of production server (CentOS7 on AWS EC2 instances). That list was fine because we do not install new servers often. But suddenly one of our production servers crashed. And it took us 2 hours full of copy&paste to follow that list and prepare new server for restoring all data from backup.

So we decided to prepare automatized installation using Bash scripts. The reason was we don’t have any experiences with tools like Pappet/Ansible/Chef and we are small company with a few servers. So Bash is probably fine, I suppose.

I was thinking how to prepare convenient development environmet. To have possibility quickly deploy my scripts, reset machine to initial config, etc. I wanted to develop on my machine, so I would have to submit all scripts to EC2 instance and execute there. That seemed slow to me. So I decided to develop scripts using LXC virtualization.

Preparing template

First I have created machine named ‘template’ with desired OS:

lxc-create -t download -n template -- -d centos -r 7 -a amd64

Start machine and connect to it:

lxc-start template
lxc-attach template

Within machine, prepare directory for scripts:

mkdir /opt/scripts

Log out (ctrl+d), shutdown machine:

lxc-stop template

Then in file ~/.local/share/lxc/template/config add line that mounts local directory where I develop scripts to machine:

lxc.mount.entry = /home/jarek/devel/installation/scripts /home/jarek/.local/share/lxc/prodtest/rootfs/opt/scripts/ none bind 0 0

Please note that above is in second path prodtest instead of template. It must not be set to template, but name of the future instance. See later.

So you can use your favorite editor on host machine (in folder /home/jarek/devel/installation/scripts), but changes are visible also within LXC container (in folder /opt/scripts). So no synchronization needed.

Also please make sure that user 100000 (process 0 in virtual machine) has acces to your folder/file on host machine. Otherwise file will not be available from virtual machine.

Usage

Create working copy take 2 seconds. Using:

lxc-copy -n template -N prodtest
lxc-start prodtest
lxc-attach prodtest

# Or one liner lxc-copy -n template -N prodtest && lxc-start prodtest && lxc-attach prodtest

Test scripts on virtual machine and edit them locally in parallel:

cd /opt/scripts

./test.sh

# Fix bugs

./test.sh

# Fix bugs

./test.sh

# ...

If machine is broken (some steps cannot be unrolled) just stop it and destroy it:

lxc-stop prodtest
lxc-destroy prodtest

# Or one liner lxc-stop prodtest && lxc-destroy prodtest

And repeat…copy from template, start, attach, test, stop, destroy. Overhead is in order of seconds.

Conclusion

This is way to develop potentially dangerous scripts in local environment, but execute them on sanbox. So no harm can be done to your machine. It is also lightweight - consumes less memory, reacts immediatelly.

You can also use different OS than runs on host.

The coolest think for me is thay you don’t have to sychronize scripts after editing.

Enjoy!