System Administration

I spent my weekend working on getting cobbler and koan working on a new server. This server will serve several virtual machines under KVM and I want a way to automatically reprovision VMs while I am working on tuning and testing.

cobbler "is a Linux installation server that allows for rapid setup of network installation environments." It allows for easy management of kickstart scripts, DHCP, and other services needed for provisioning new machines. koan, which stands for "kickstart-over-a-network," is a client for cobbler and is used "for reinstallation and virtualization support."

Directions on how to install cobbler can be found on the site for cobbler. In a nutshell, Fedora users can install it directly from the repository servers and RHEL and CentOS users can install it from the EPEL and EPEL testing repositories. (For this, I am using CentOS 5.4 and the version of cobbler and koan from the EPEL testing repository. cobbler was originally installed from the EPEL repository so this may create some oddities in behavior.) To support some functionality, you will want to install the qspice-libs and yum-utils packages.

For cobbler to work correctly under SELinux, cobbler check suggests the following changes:

/usr/sbin/setsebool -P httpd_can_network_connect true
/usr/sbin/semanage fcontext -a -t public_content_t "/var/lib/tftpboot/.*"
/usr/sbin/semanage fcontext -a -t public_content_t "/var/www/cobbler/images/.*"

This allows the Apache httpd server to connect to the network and assigns the public_content_t file context to files in /var/lib/tftpboot/ and /var/www/cobbler/images/. If the standard firewall is installed, traffic will need to be allowed to TCP ports 80 and 26161. cobbler check will also recommend enabling TFTP and allowing traffic to UDP port 69 but these are not required for provisioning virtual machines with koan. You should also review the settings in /etc/cobbler/settings. At a minimum, the server and next_server settings need to be changed. (If you're using KVM, you probably want to change the default_virt_bridge and default_virt_type settings as well.)

To start working with cobbler, a distribution needs to be created. The man page supports importing a repository. However, this may originally take a while since the base repository tends to be large. (The x86_64 repository for CentOS 5.4 is about 4.4 GB.) An example is:

cobbler import --path=rsync://ftp.linux.ncsu.edu/CentOS/5/os/x86_64/ --name=centos5 --arch=x86_64

This will create a distribution named centos5-x86_64. (You will want to use a mirror local to you.)

cobbler allows identifying other repositories that will be mirrored locally and used for the build process via the repo kickstart option. If you want to install with the newest packages available, you will want to add the updates repository, like so:

cobbler repo add --arch=x86_64 --name=centos5-updates-x86_64 \
    --mirror=rsync://ftp.linux.ncsu.edu/CentOS/5/updates/x86_64/

All this will do is tell cobbler to mirror the repository locally. In order to actually set up the mirror, you will need to run cobbler reposync.

Next, identify an installation profile. This is usually tied to a specific server role. Here, we'll define a profile for a VM for a DNS server:

cobbler profile add --name=centos5-vm_dns --distro=centos5-x86_64 --virt-ram=512 \
    --virt-type=qemu --virt-cpus=1 --repos="centos5-updates-x86_64" \
    --kickstart=/var/lib/cobbler/kickstarts/vm-dns.ks --kopts="serial console=ttyS0,115200" \
    --kopts-post="console=ttyS0,115200"

This specifies that the profile should use the CentOS 5 distribution and the CentOS 5 updates repository created above, configure it to run as a VM under KVM, use 512 MB of RAM, and one virtual CPU. The kickstart file template, which has already been created, resides under /var/lib/cobbler/kickstarts/vm-dns.ks. (Information on setting up the kickstart template can be found on the cobbler website.) The kopts setting specifies that kickstart should be started with the serial and console settings. The kopts-post setting specifies that the installed machine should have the console kernel option defined. (These are needed so that the VM makes a console available that can be used via virsh console.)

Usually each profile will have multiple systems under it. A system corresponds to a machine, physical or virtual. (There is still some benefit in using cobbler when there is only one system per profile like I'm doing on my testing server. However, it will really shine when you have to install many mostly identical machines.) A system is defined in the following manner:

cobbler system add --name example-dns --profile=centos5-vm_dns \
    --ip=192.168.122.3 --gateway=192.168.122.1 --subnet=255.255.255.0 \
    --hostname=dns.example.com --static=1

This creates a system named dns.example.com with the static IP 192.168.122.3.

To install a virtual machine, you use koan like so:

koan --server=192.168.122.1 --virt --system=example-dns --virt-path=/dev/mapper/examplevg-dns

This will install a VM using the example-dns system defined above to /dev/mapper/examplevg-dns, a logical volume created via LVM. This should work properly if using a file or a normal partition.

If you've used virt-install, you're probably used to seeing the console opened immediately once the VM is started. koan does not do this. To see the VM, you will need to open the console manually via virsh console. (For example, to see the console for the VM being created by the koan statement above, use virsh console example-dns.)

A few issues I encountered were:

• Error message: libvir: QEMU error : internal error cannot parse QEMU version number in ''
  As mentioned here, the error is corrected by installing the qspice-libs package.
• Error message:
  

  libvir: QEMU error : internal error unable to start guest: qemu: could not open disk image /dev/mapper/examplevg-dns

  Also, errors like these appear in the audit log:

  type=AVC msg=audit(1266799848.453:623): avc:  denied  { getattr } for  pid=32208 
  comm="qemu-kvm" path="/dev/mapper/examplevg-dns" dev=tmpfs ino=78122 
  scontext=system_u:system_r:qemu_t:s0-s0:c0.c1023 
  tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file
   
  type=AVC msg=audit(1266799848.453:624): avc:  denied  { read } for  pid=32208 comm="qemu-kvm" 
  name="examplevg-dns" dev=tmpfs ino=78122 scontext=system_u:system_r:qemu_t:s0-s0:c0.c1023 
  tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file

  The solution to this is to add the virt_image_t context to the block file with:

  chcon -t virt_image_t /dev/mapper/examplevg-dns

  I tried to set this with semanage but was unsuccessful. This probably means that I need to look at the documentation better.

  This is apparently not an issue in Fedora 11 and later and in the upcoming RHEL 6 due to sVirt. In a mailing list post, Daniel Walsh states that they hope to get this ported into RHEL 5.6.

• Despite the server and next_server settings in /etc/cobbler/settings, koan still tries to install using URLs referencing 127.0.0.1.

  I don't really know the cause of this. It seemed odd but I couldn't trace it. For now, I used a workaround by setting the server for the profile manually with --server=192.168.122.1.

  It could either be that there is something strange with my setup or that this is intended.

• Error when running koan:
  

  exceptions.TypeError
  cannot concatenate 'str' and 'dict' objects
    File "/usr/lib/python2.4/site-packages/koan/app.py", line 215, in main
      k.run()
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 329, in run
      self.virt()
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 652, in virt
      return self.net_install(after_download)
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 571, in net_install
      after_download(self, profile_data)
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 650, in after_download
      self.virt_net_install(profile_data)
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 1103, in virt_net_install
      kextra                        = self.calc_kernel_args(pd)
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 1050, in calc_kernel_args
      kextra = kextra + " " + options

  I'm not certain why I saw this particular error. My solution was to comment out line 1050 and add this line in its place:

  kextra = kextra + " " + utils.hash_to_string(options)

  If I remember, I'll post to the cobbler mailing list.

  Update: I opened ticket #576 for this issue. My email to the cobbler-devel mailing list includes a patch.

If this looks interesting, I definitely suggest looking at the cobbler website. If you install many machines, you may also be interested in looking at the web interface which may make this task even easier.

When installing packages from source on *nix servers, I find I need to compile the software identically across several machines. (I do realize that this would be easier if I built a single package and then installed that using the OS/distribution package management tools but that's for another day.) Or sometimes I need to upgrade a package and keep the configuration information the same.

On my personal webserver (which runs this site), I have a directory that contains scripts that include the ./configure command (or the equivalent for the package) and the options for that command. Whenever I need to rebuild a package that has such a script, I just need to run the script.

However, there are some packages that lack such scripts. In these cases, I have to go to the old source directory, check config.log to find how I built it last and then use the same options in the new directory. (Or the same directory if I am reconfiguring the same version of the package.) I find that due to a feature of config.log and either screen or my SSH client that I have to retype parts of the ./configure command-line because it has wrapped around and the wrapping isn't being honored.

The latter is clearly non-optimal. The former is better but is limited to a single machine. If I need to build a package in a like manner on a different machine, I either have to copy the script or recreate it on the new one. Either way, this process is somewhat cumbersome.

An improvement I am trying is documenting the build process used for all software on a wiki. This provides a central (and version controlled) repository for the information that I can refer to independently of the SSH session. It does require a browser but this is not an issue in today's world of large screens and the movement towards using multiple monitors. (If screen real-estate is at a premium or you're not at a workstation, most wikis render well in lynx. If using ikiwiki, you could even check out the repository and view it with less.) Also, if multiple machines have the same software installed but require slightly different configurations, you can make the note in the wiki entry so you can easily see the differences. Further, since wikis implement revisions, you can see how the options have changed over time.

The challenge is making sure that all packages get documented correctly. However, once in the habit of documenting the process, it should become easier to make sure that future packages are documented as well.