Ithiriel Thoughts From a Sysadmin And Occasional Coder

You may disagree with this statement I made earlier today:

Linux on the desktop will never be as polished as OS X, or even Windows, until some company pours hundreds of thousands of dollars (if not more) into building a usable UI on top of it. However, any company that dumps that much into it isn't likely to make those contributions open source.

It is very possible to make Linux desktop-ready. OS X is fundamentally a BSD-clone with a lot of money and time poured into the user interface. So given some vast amount of time and money, it should be possible to do the same thing with Linux.

I could be wrong but: Most organizations that focus on the open source aspects of Linux (well, properly, GNU/Linux and other stuff) tend to not have a lot of money. So providing the required amount of money will probably fall to a company. And a company that is making a such a significant investment is going to want to capitalize on that. Pouring vast amounts of time and money into something and then giving it away is a losing strategy.

It is possible for the software foundations to fund the development but it will take significantly longer. The added time will ensure that desktop Linux UI would always trail the corporate offerings (Windows, OS X, etc.) because it will have never caught up (assuming the corporate offerings do not stagnate).

Cobbler has the ability to run triggers at specific times. Two of those times include before installing a new machine ("pre-install triggers") and after installing a new machine ("post-install triggers").

"Old-style" triggers involve running executable binaries or scripts (e.g. shell scripts) in specific locations. Pre-install triggers are placed in the directory /var/lib/cobbler/triggers/install/pre/ and post-install triggers are placed in the directory /var/lib/cobbler/triggers/install/post/. The trigger will be passed three pieces of information: the object type, e.g. "system," the name of the object, and the object's IP. (A comment in the run_install_triggers method in remote.py says this passes the name, MAC, and IP but this does not appear to match the name of the variables.) If the trigger requires more information, it will need to pull it from elsewhere or parse cobbler's output. For all but simple tasks, this is probably not a convenient way to go.

Note: There is a bug in cobbler 2.0.3.1 which prevents running "old-style" triggers. See ticket #530 for more information and a possible fix.

"New-style" triggers are written in Python as modules. They reside in cobbler's module directory. (On my system, this is /usr/lib/python2.4/site-packages/cobbler/modules/.) Each module is required to define at least the functions register and run.

The register function takes no arguments and returns a string corresponding to the directory the module would reside in if it were an "old-style" trigger. For a pre-install trigger, it would be:

For a post-install trigger, it would be:

The run function is where the actual code for the trigger should reside. It takes three arguments: A cobbler API reference, an array containing arguments, and a logger reference. The argument array contains the same three values as for "old-style" triggers, i.e. the object type, the name, and the IP address. The logger reference may be set to None and the code should handle that. (In cobbler 2.0.3.1, this will be set to None. This may be fixed when the issue for "old-style" install triggers is.)

For an example of a run function, let's look at one I wrote (based on the trigger in install_post_report.py that is included with cobbler) to automatically sign Puppet certificates:

This starts the method. Note the signature.

This retrieves the settings from /etc/cobbler/settings. To control the trigger, I added another option there named sign_puppet_certs_automatically. If this value either does not exist or is not set to one of the required values showing its enabled, the trigger returns a success code (since it's not supposed to run, it shouldn't return a failure code) and exits.

I also added another option to the cobbler settings called puppetca_path which contains the path to the puppetca command.

This retrieves the object type and name from the argument array. If the object type is not a system, it returns a success code and exits.

This finds the system in the cobbler API and then flattens it to a dictionary. I'm pretty sure this could be improved upon.

This retrieves the path for puppetca and sets up the command to be run to sign the certificate.

This runs the command and logs a warning if either the command fails to be executed or does not succeed. Finally, at the end, it returns a success code.

According to the cobbler documentation, the return code of post-install triggers is ignored so there's no reason not to return anything other than value. Pre-install triggers apparently can halt the process if they return a non-zero value.

Note: The above code will not run correctly if logger is set to None. This is because utils.subprocess_call tries to call logger without verifying that it is not None and throws an exception. To use this with cobbler 2.0.3.1, you must either edit change the call to utils.run_triggers in remote.py's run_install_triggers method or you must change utils.subprocess_call to properly check for logger being set to None.

Also note: Since the original code is under the GPL, the code above is also under the GPL.

Apparently my last post broke the Atom feed, causing it not to validate. I'm looking into how to fix it but it may be related to a known issue with Drupal's Atom module.

I don't expect to have this fixed today. I suggest switching to the RSS feed for now.

I spent my weekend working on getting cobbler and koan working on a new server. This server will serve several virtual machines under KVM and I want a way to automatically reprovision VMs while I am working on tuning and testing.

cobbler "is a Linux installation server that allows for rapid setup of network installation environments." It allows for easy management of kickstart scripts, DHCP, and other services needed for provisioning new machines. koan, which stands for "kickstart-over-a-network," is a client for cobbler and is used "for reinstallation and virtualization support."

Directions on how to install cobbler can be found on the site for cobbler. In a nutshell, Fedora users can install it directly from the repository servers and RHEL and CentOS users can install it from the EPEL and EPEL testing repositories. (For this, I am using CentOS 5.4 and the version of cobbler and koan from the EPEL testing repository. cobbler was originally installed from the EPEL repository so this may create some oddities in behavior.) To support some functionality, you will want to install the qspice-libs and yum-utils packages.

For cobbler to work correctly under SELinux, cobbler check suggests the following changes:

/usr/sbin/setsebool -P httpd_can_network_connect true
/usr/sbin/semanage fcontext -a -t public_content_t "/var/lib/tftpboot/.*"
/usr/sbin/semanage fcontext -a -t public_content_t "/var/www/cobbler/images/.*"

This allows the Apache httpd server to connect to the network and assigns the public_content_t file context to files in /var/lib/tftpboot/ and /var/www/cobbler/images/. If the standard firewall is installed, traffic will need to be allowed to TCP ports 80 and 26161. cobbler check will also recommend enabling TFTP and allowing traffic to UDP port 69 but these are not required for provisioning virtual machines with koan. You should also review the settings in /etc/cobbler/settings. At a minimum, the server and next_server settings need to be changed. (If you're using KVM, you probably want to change the default_virt_bridge and default_virt_type settings as well.)

To start working with cobbler, a distribution needs to be created. The man page supports importing a repository. However, this may originally take a while since the base repository tends to be large. (The x86_64 repository for CentOS 5.4 is about 4.4 GB.) An example is:

cobbler import --path=rsync://ftp.linux.ncsu.edu/CentOS/5/os/x86_64/ --name=centos5 --arch=x86_64

This will create a distribution named centos5-x86_64. (You will want to use a mirror local to you.)

cobbler allows identifying other repositories that will be mirrored locally and used for the build process via the repo kickstart option. If you want to install with the newest packages available, you will want to add the updates repository, like so:

cobbler repo add --arch=x86_64 --name=centos5-updates-x86_64 \
    --mirror=rsync://ftp.linux.ncsu.edu/CentOS/5/updates/x86_64/

All this will do is tell cobbler to mirror the repository locally. In order to actually set up the mirror, you will need to run cobbler reposync.

Next, identify an installation profile. This is usually tied to a specific server role. Here, we'll define a profile for a VM for a DNS server:

cobbler profile add --name=centos5-vm_dns --distro=centos5-x86_64 --virt-ram=512 \
    --virt-type=qemu --virt-cpus=1 --repos="centos5-updates-x86_64" \
    --kickstart=/var/lib/cobbler/kickstarts/vm-dns.ks --kopts="serial console=ttyS0,115200" \
    --kopts-post="console=ttyS0,115200"

This specifies that the profile should use the CentOS 5 distribution and the CentOS 5 updates repository created above, configure it to run as a VM under KVM, use 512 MB of RAM, and one virtual CPU. The kickstart file template, which has already been created, resides under /var/lib/cobbler/kickstarts/vm-dns.ks. (Information on setting up the kickstart template can be found on the cobbler website.) The kopts setting specifies that kickstart should be started with the serial and console settings. The kopts-post setting specifies that the installed machine should have the console kernel option defined. (These are needed so that the VM makes a console available that can be used via virsh console.)

Usually each profile will have multiple systems under it. A system corresponds to a machine, physical or virtual. (There is still some benefit in using cobbler when there is only one system per profile like I'm doing on my testing server. However, it will really shine when you have to install many mostly identical machines.) A system is defined in the following manner:

cobbler system add --name example-dns --profile=centos5-vm_dns \
    --ip=192.168.122.3 --gateway=192.168.122.1 --subnet=255.255.255.0 \
    --hostname=dns.example.com --static=1

This creates a system named dns.example.com with the static IP 192.168.122.3.

To install a virtual machine, you use koan like so:

koan --server=192.168.122.1 --virt --system=example-dns --virt-path=/dev/mapper/examplevg-dns

This will install a VM using the example-dns system defined above to /dev/mapper/examplevg-dns, a logical volume created via LVM. This should work properly if using a file or a normal partition.

If you've used virt-install, you're probably used to seeing the console opened immediately once the VM is started. koan does not do this. To see the VM, you will need to open the console manually via virsh console. (For example, to see the console for the VM being created by the koan statement above, use virsh console example-dns.)

A few issues I encountered were:

• Error message: libvir: QEMU error : internal error cannot parse QEMU version number in ''
  As mentioned here, the error is corrected by installing the qspice-libs package.
• Error message:
  

  libvir: QEMU error : internal error unable to start guest: qemu: could not open disk image /dev/mapper/examplevg-dns

  Also, errors like these appear in the audit log:

  type=AVC msg=audit(1266799848.453:623): avc:  denied  { getattr } for  pid=32208 
  comm="qemu-kvm" path="/dev/mapper/examplevg-dns" dev=tmpfs ino=78122 
  scontext=system_u:system_r:qemu_t:s0-s0:c0.c1023 
  tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file
   
  type=AVC msg=audit(1266799848.453:624): avc:  denied  { read } for  pid=32208 comm="qemu-kvm" 
  name="examplevg-dns" dev=tmpfs ino=78122 scontext=system_u:system_r:qemu_t:s0-s0:c0.c1023 
  tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file

  The solution to this is to add the virt_image_t context to the block file with:

  chcon -t virt_image_t /dev/mapper/examplevg-dns

  I tried to set this with semanage but was unsuccessful. This probably means that I need to look at the documentation better.

  This is apparently not an issue in Fedora 11 and later and in the upcoming RHEL 6 due to sVirt. In a mailing list post, Daniel Walsh states that they hope to get this ported into RHEL 5.6.

• Despite the server and next_server settings in /etc/cobbler/settings, koan still tries to install using URLs referencing 127.0.0.1.

  I don't really know the cause of this. It seemed odd but I couldn't trace it. For now, I used a workaround by setting the server for the profile manually with --server=192.168.122.1.

  It could either be that there is something strange with my setup or that this is intended.

• Error when running koan:
  

  exceptions.TypeError
  cannot concatenate 'str' and 'dict' objects
    File "/usr/lib/python2.4/site-packages/koan/app.py", line 215, in main
      k.run()
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 329, in run
      self.virt()
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 652, in virt
      return self.net_install(after_download)
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 571, in net_install
      after_download(self, profile_data)
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 650, in after_download
      self.virt_net_install(profile_data)
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 1103, in virt_net_install
      kextra                        = self.calc_kernel_args(pd)
     File "/usr/lib/python2.4/site-packages/koan/app.py", line 1050, in calc_kernel_args
      kextra = kextra + " " + options

  I'm not certain why I saw this particular error. My solution was to comment out line 1050 and add this line in its place:

  kextra = kextra + " " + utils.hash_to_string(options)

  If I remember, I'll post to the cobbler mailing list.

  Update: I opened ticket #576 for this issue. My email to the cobbler-devel mailing list includes a patch.

If this looks interesting, I definitely suggest looking at the cobbler website. If you install many machines, you may also be interested in looking at the web interface which may make this task even easier.