Finally - Fast and Real functional on Node checks in Jenkins

Berlinés verde{width=”500” height=”500”}

Continuous integration and testing is a hot topic. It is so important, it is so good, because it saves us from doing lame stuff ourselfs.

Lago is oVirt’s answer to allow fully automated testing with multiple hosts. And it is good at this. Node support has landed a while ago, we just need to get it into production. But for basic Node centric sanity this full blown setup is not needed.

With a simple setup, we are now running sanity tests on a running VM in jenkins.

Our sanity tests now cover the basic building blocks of Node:

  • Correct bootloader configuration
  • Correct package set
  • Correct LVM layout
  • No SELinux denials
  • Correct enabled services
  • Correct mount options
  • Working imgbased

Thesse items are crucial for the core functionality of node (working upgrades, reliable rollback). And up to now there is nothing oVirt specific, and this is fine, because integration tests in Lago will ensure that the integration is working.

And these tests are isolated (snapshotted), and fast. The complete testing is done in under 2 minutes. The value is high because it will catch at least:

  • Any build related error
  • Any partitioning related error
  • Any severe bug in imgbased

The biggest gap is to ensure that updates are working. More tests can be added, all of this can be driven from python unit-tests.

Let me note that the testing process is robust. The tests are run in real VMs, and are thus not impacted by the host configuration.

The testing is triggered quite easily, in the ovirt-node-ng repository, run:

# Is implicitly called, but this does build the image
# 85% of the time
make squashfs

# This will install the image into a qcow2
# 10% of the time
make installed-squashfs

# This will check the qcow2
# 5% of the time
make check

And this is finally how such a test run looks:

00:23:51.931 The squashfs 'ovirt-node-ng-image.squashfs.img' got installed into the file 'ovirt-node-ng-image.installed.qcow2'
00:23:51.986 ++ sudo -E make check
00:23:51.998 make -C tests check
00:23:52.000 make[1]: Entering directory '/home/jenkins/workspace/ovirt-node-ng_master_build-artifacts-fc22-x86_64/ovirt-node-ng/tests'
00:23:52.004 pyflakes testIntegration.py virt.py agent.py testSanity.py testVirt.py
00:23:52.190 pep8 testIntegration.py virt.py agent.py testSanity.py testVirt.py
00:23:52.481 nosetests --with-xunit -v --no-byte-compile --stop testVirt.py testSanity.py
00:24:43.359 test_agent_works (testVirt.TestNodeTestcase) ... ok
00:24:55.888 test_reboot_works (testVirt.TestNodeTestcase) ... ok
00:25:46.463 test_bootloader (testSanity.TestNode) ... ok
00:25:51.445 test_imgbase (testSanity.TestNode) ... ok
00:25:54.761 test_lvm (testSanity.TestNode) ... ok
00:25:57.731 test_mounts (testSanity.TestNode) ... ok
00:26:00.835 test_packages (testSanity.TestNode) ... ok
00:26:03.741 test_selinux (testSanity.TestNode) ... ok
00:26:06.666 test_services (testSanity.TestNode) ... ok
00:26:06.969 
00:26:06.969 ----------------------------------------------------------------------
00:26:06.970 XML: /home/jenkins/workspace/ovirt-node-ng_master_build-artifacts-fc22-x86_64/ovirt-node-ng/tests/nosetests.xml
00:26:06.970 ----------------------------------------------------------------------
00:26:06.970 Ran 9 tests in 134.311s
00:26:06.970 
00:26:06.970 OK
00:26:06.983 xsltproc nosetests.xslt nosetests.xml
00:26:06.987 
00:26:06.987 Xunit Test Results - For: 
00:26:06.987 Results for nosetests
00:26:06.987 
00:26:06.987 * Tests run: 9
00:26:06.987 * Failures: 0
00:26:06.987 * Skipped: 0
00:26:06.987 * Errors: 0
00:26:06.990 PASSED

The hardest part was to get running correctly in jenkins. But now it does.

::: {#footer} [ June 9th, 2016 8:05pm ]{#timestamp} [ovirt]{.tag} [imgbased]{.tag} [unittest]{.tag} [python]{.tag} [libvirtd]{.tag} [guestfish]{.tag} [virt-install]{.tag} [node]{.tag} :::