System Administration

I haven't been very happy with my way to build packages. I've been looking for a better system for managing it.

Through conversation in #lopsa, I found my way to Mock, a tool that builds packages in chroot environments.

I've been testing it on a VM. So far, it looks promising.

To use Mock, you first need to add your user to the mock group:

sudo /usr/sbin/usermod -a -G mock $user

After that, Mock is simple to use. If you have a source RPM, building a package is as easy as:

mock -r $configuration rebuild $srpm

So, for example, to build my patched kernel RPM:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm

Build environment configurations are defined in /etc/mock. If you're building RPMs for use with RHEL or CentOS, one of the preexisting epel configurations should suffice. There are configurations for Fedora as well. You can configure your own configurations as well.

On 64-bit systems, you can use the 32-bit configurations to build 32-bit packages. If you want to use specify a different architecture, you can use the --arch argument. For example, to build all binary RPMs for my patched kernel:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean

The first command builds the 64-bit packages. The second command builds the 32-bit packages. The third command builds the additional packages that don't have an architecture, e.g. kernel-doc-2.6.32.46-1.el6.oberon.noarch.rpm. The --no-clean argument tells mock not to clean the build environment first. Without this, the third command will remove the packages generated by the first command.

When the commands are done running, the 64-bit and noarch RPMs can be found in the directory /var/lib/mock/epel-6-x86_64/result/ and the 32-bit RPMs can be found in the directory /var/lib/mock/epel-6-i386/result/.

I haven't tried using Mock in my Makefile but that's next on the list. I also need to simplify my builds so they don't rebuild all RPMs. Since the kernel RPMs take about six hours to build (for both 32-bit and 64-bit) on my VM, this makes builds almost prohibitively long.

I have also thought about building a system that uses Mock, some message queueing system, and some cloud interface to spin up EC2 instances (or the like) for builds. However, that seems pretty close to Koji so I should probably look into that further first.

This is mostly for my reference since I keep forgetting this. As per cobbler's documentation:

The kickstart URL for a system is:
http://$server/cblr/svc/op/ks/system/$name_of_system

I've been working on a body of characteristics that a server should ideally have. So far, I have:

Documented

A server should be documented. The documentation should provide enough information to anyone who has to service the server, including allowing them to rebuild the server if necessary. The documentation should also satisfy any auditing requirements of the organization.

Verifiable

A server should have a given known state, described in the documentation, and it should be possible to programmatically determine whether or not the system is in that state.

Secure

A server should be protected against any attacks made by unauthorized entities that would disrupt the server or provide the attacker with information they are not authorized to have.

Monitored

A server should be monitored to ensure that it is in its documented state. Any deviation from that state, either through an uncoordinated change made by operations or through actions of an unauthorized party, should be detected automatically and the operations staff should be notified accordingly.

Backed up

A server should be backed up. Data that may have been lost due to error or tampering should be recoverable within the documented parameters.

Replaceable

A server should be replaceable. If the server fails or significantly goes out of its documented state, any technician should be able to provision and install a replacement while the faulty system can be diagnosed, inspected, and, if possible, corrected outside of the production environment.

Manageable

A server should be manageable and serviceable by any technician authorized to work on it.

Measured

A server should have its statistics measured and recorded on a regular basis. This data should be usable for planning activities and to monitor trends on the server.

Functional

A server should fulfill its documented role.

Some of these are admittedly characteristics of the server's environment and its technicians rather than of the server itself.

If you think I've overlooked anything, please let me know.

Edit: I added the last two originally as comments. I've added them to the actual post to make it easier for anyone who visits in the future.

Recently, I've been trying to build a kernel for CentOS 6 that includes the grsecurity security patch. This was complicated by my desire to build the new kernel as an RPM using the CentOS 6 kernel RPM's spec file.

Why grsecurity?

grsecurity is a kernel patch that adds additional security options and protections. The non-RBAC components of grsecurity will work with SELinux which I will use for now.1

Why use the spec file and not make rpm?

Well, first: Because make rpm didn't work. That was disappointing.

In CentOS kernel install split into separate packages, e.g. kernel, kernel-headers, and kernel-devel. In order to preserve these, I would have to reverse engineer the spec file and it just seems simpler to use what's there. The spec file also makes sure that the new kernel is added to the bootloader's configuration file.

Why use an RPM at all?

Building packages from source doesn't scale.2

One of my current projects is to build a fully functional environment where no packages are installed from source.3 In fact, none of the systems are allowed to have compilers. This is why I've worked on creating my own yum repository.

The process

After downloading the source file for Linux 2.6.32.46 and the grsecurity patch and verifying both4, I changed the spec file to use these. Due to an issue with %setup in the spec file I never fully figured out, I decided to invoke applying the patch manually through the ApplyPatch function defined in the %prep block. I then set about building the kernel initially using rpmbuild -ba SPEC/kernel.spec and ran into problems.

The complications

I ran into six problems during the process of building my RPM. These were:

1. Fixing an error during the %prep phase
2. Getting my configuration changes to persist
3. Signing the modules
4. Turning off the kABI checker
5. Setting the kernel version correctly
6. The system halted on boot

Fixing an error during the %prep phase

During the %prep phase, the spec file tries to verify that all of the configuration options known by the kernel are covered in the configuration files. This relies on the nonint_oldconfig target defined in scripts/kconfig/Makefile in the RPM. However, this option does not exist in 2.6.32.46. Extracting the patch for this (and for the related code in scripts/kconfig/conf.c) from the tarball provided with the RPM, adding it to the spec file, and applying it allowed me to proceed... but not very far. If nonint_oldconfig finds any kernel options that are not covered, like the options added by the grsecurity patch, it returns an error and the build halts.

Getting my configuration changes to persist

The SRPM includes generic and architecture-specific configuration files. These are merged together during the %prep phase. If the configuration changes are not present in these files, they are wiped during the phase.

I decided to keep the configuration files intact and instead placed my overriding kernel options in a separate file. I then modified Makefile.config to check for my files and merge those as well if present.

Signing the modules

CentOS 6 inherits the upstream vendor's patches. One such patch is used to sign the modules with a key so that the file integrity of the modules can be verified later. Any module that fails verification is not loaded. There is a kernel configuration option that requires all modules to be signed. As the spec file is supposed to be used with the distribution's patched kernel source, it verifies that the modules are signed. I could have removed this feature from the spec file but I decided this feature was a good one.

This is not part of the stock 2.6.32.46 kernel and, in fact, does not appear to be included in 3.0.4 either. To use this feature, the code changes have to be isolated from the patched distribution kernel and a new patch needed to be made. After creating a patch from changes made to 54 source files, I added it to the spec file and built signed modules.

Turning off the kABI checker

The CentOS kernel, like the upstream vendor's kernel, is expected to fulfill a given interface contract when built. Anything that would break this contract severely is not allowed to be made as a change to the kernel.5 As a result, the kernel verifies this contract in the spec file.

The standard way to do this is to pass this option to rpmbuild when building the kernel: --without kabichk

As I didn't feel like remembering to specify that every time I build the kernel, I changed the spec file to make sure that the with_kabichk variable set to  . I could have left it in place by copying Module.symvers to the appropriate kABI file in the SOURCES directory and changing the spec file appropriately.

Setting the kernel version correctly

This took me a while to figure out unfortunately. When I built the kernel, the version was being set incorrectly and then this resulted in the system not being able to find modules on boot.

What I figured out in the end was that, for kernel 2.6.32.46, base_sublevel should be set to 32 and stable_update should be set to 46.

grsecurity also adds a file called localversion-grsec that contains the text -grsec. I wrote a patch to remove the file, added it to the spec file, built it, installed it, and...

The system halted on boot

When a CentOS 6 system boots normally, it uses something called dracut. At some point, it tries to mount a partition inside a chroot'd partition. As the kernel is configured to prevent this (through the grsecurity patch), the system would then halt on boot.

As I didn't want to disable this feature permanently and I wanted everything else to work from boot, I wrote a small patch to disable the feature on boot. Adding kernel.grsecurity.chroot_deny_mount = 1 to /etc/sysctl.conf then enabled this feature later in the boot process.

And now it works

After building the RPMs, I installed the kernel and kernel-firmware packages on a test VM, rebooted it, and it came up using the new kernel.

I did find that a lot of messages were being written to the console. Setting kernel.printk to "6 4 1 7" via sysctl corrected this.

What next?

I want to work on backporting some changes from the CentOS 6 kernel to the package I have built. I know that the CentOS 6 kernel contains some features from 2.6.33, e.g. recvmmsg, and some enhancements to KVM. Unfortunately, since the source tarball in the RPM has already been patched, I have to tease out what those patches are. Once I have those patches, I should then check to see there are any grsecurity changes that need to be backported. Since there is no obviously apparent repository for the grsecurity patch, this requires looking at the test patch (currently for the 3.0.4 kernel) and comparing manually.

So where can I get this?

Once I have some things worked out, I will post the RPMs and the SRPM somewhere that it can be downloaded. For now, you'll just have to be patient.

• 1. I have grsecurity installed on another server and I've never gotten around to configuring RBAC. Since a lot of work has gone into the CentOS 6 SELinux policies, I'll just rely on those.
• 2. I've said this before. Some day I'll write a longer post on it.
• 3. I'll try to write about this in the near future.
• 4. You do this whenever you download a source file, right?
• 5. This is why CentOS 5 does not support IPv6 connection tracking without building a custom kernel.

Cobbler has the ability to run triggers at specific times. Two of those times include before installing a new machine ("pre-install triggers") and after installing a new machine ("post-install triggers").

"Old-style" triggers involve running executable binaries or scripts (e.g. shell scripts) in specific locations. Pre-install triggers are placed in the directory /var/lib/cobbler/triggers/install/pre/ and post-install triggers are placed in the directory /var/lib/cobbler/triggers/install/post/. The trigger will be passed three pieces of information: the object type, e.g. "system," the name of the object, and the object's IP. (A comment in the run_install_triggers method in remote.py says this passes the name, MAC, and IP but this does not appear to match the name of the variables.) If the trigger requires more information, it will need to pull it from elsewhere or parse cobbler's output. For all but simple tasks, this is probably not a convenient way to go.

Note: There is a bug in cobbler 2.0.3.1 which prevents running "old-style" triggers. See ticket #530 for more information and a possible fix.

"New-style" triggers are written in Python as modules. They reside in cobbler's module directory. (On my system, this is /usr/lib/python2.4/site-packages/cobbler/modules/.) Each module is required to define at least the functions register and run.

The register function takes no arguments and returns a string corresponding to the directory the module would reside in if it were an "old-style" trigger. For a pre-install trigger, it would be:

For a post-install trigger, it would be:

The run function is where the actual code for the trigger should reside. It takes three arguments: A cobbler API reference, an array containing arguments, and a logger reference. The argument array contains the same three values as for "old-style" triggers, i.e. the object type, the name, and the IP address. The logger reference may be set to None and the code should handle that. (In cobbler 2.0.3.1, this will be set to None. This may be fixed when the issue for "old-style" install triggers is.)

For an example of a run function, let's look at one I wrote (based on the trigger in install_post_report.py that is included with cobbler) to automatically sign Puppet certificates:

This starts the method. Note the signature.

This retrieves the settings from /etc/cobbler/settings. To control the trigger, I added another option there named sign_puppet_certs_automatically. If this value either does not exist or is not set to one of the required values showing its enabled, the trigger returns a success code (since it's not supposed to run, it shouldn't return a failure code) and exits.

I also added another option to the cobbler settings called puppetca_path which contains the path to the puppetca command.

This retrieves the object type and name from the argument array. If the object type is not a system, it returns a success code and exits.

This finds the system in the cobbler API and then flattens it to a dictionary. I'm pretty sure this could be improved upon.

This retrieves the path for puppetca and sets up the command to be run to sign the certificate.

This runs the command and logs a warning if either the command fails to be executed or does not succeed. Finally, at the end, it returns a success code.

According to the cobbler documentation, the return code of post-install triggers is ignored so there's no reason not to return anything other than value. Pre-install triggers apparently can halt the process if they return a non-zero value.

Note: The above code will not run correctly if logger is set to None. This is because utils.subprocess_call tries to call logger without verifying that it is not None and throws an exception. To use this with cobbler 2.0.3.1, you must either edit change the call to utils.run_triggers in remote.py's run_install_triggers method or you must change utils.subprocess_call to properly check for logger being set to None.

Also note: Since the original code is under the GPL, the code above is also under the GPL.