Security

Recently, I've been trying to build a kernel for CentOS 6 that includes the grsecurity security patch. This was complicated by my desire to build the new kernel as an RPM using the CentOS 6 kernel RPM's spec file.

Why grsecurity?

grsecurity is a kernel patch that adds additional security options and protections. The non-RBAC components of grsecurity will work with SELinux which I will use for now.1

Why use the spec file and not make rpm?

Well, first: Because make rpm didn't work. That was disappointing.

In CentOS kernel install split into separate packages, e.g. kernel, kernel-headers, and kernel-devel. In order to preserve these, I would have to reverse engineer the spec file and it just seems simpler to use what's there. The spec file also makes sure that the new kernel is added to the bootloader's configuration file.

Why use an RPM at all?

Building packages from source doesn't scale.2

One of my current projects is to build a fully functional environment where no packages are installed from source.3 In fact, none of the systems are allowed to have compilers. This is why I've worked on creating my own yum repository.

The process

After downloading the source file for Linux 2.6.32.46 and the grsecurity patch and verifying both4, I changed the spec file to use these. Due to an issue with %setup in the spec file I never fully figured out, I decided to invoke applying the patch manually through the ApplyPatch function defined in the %prep block. I then set about building the kernel initially using rpmbuild -ba SPEC/kernel.spec and ran into problems.

The complications

I ran into six problems during the process of building my RPM. These were:

1. Fixing an error during the %prep phase
2. Getting my configuration changes to persist
3. Signing the modules
4. Turning off the kABI checker
5. Setting the kernel version correctly
6. The system halted on boot

Fixing an error during the %prep phase

During the %prep phase, the spec file tries to verify that all of the configuration options known by the kernel are covered in the configuration files. This relies on the nonint_oldconfig target defined in scripts/kconfig/Makefile in the RPM. However, this option does not exist in 2.6.32.46. Extracting the patch for this (and for the related code in scripts/kconfig/conf.c) from the tarball provided with the RPM, adding it to the spec file, and applying it allowed me to proceed... but not very far. If nonint_oldconfig finds any kernel options that are not covered, like the options added by the grsecurity patch, it returns an error and the build halts.

Getting my configuration changes to persist

The SRPM includes generic and architecture-specific configuration files. These are merged together during the %prep phase. If the configuration changes are not present in these files, they are wiped during the phase.

I decided to keep the configuration files intact and instead placed my overriding kernel options in a separate file. I then modified Makefile.config to check for my files and merge those as well if present.

Signing the modules

CentOS 6 inherits the upstream vendor's patches. One such patch is used to sign the modules with a key so that the file integrity of the modules can be verified later. Any module that fails verification is not loaded. There is a kernel configuration option that requires all modules to be signed. As the spec file is supposed to be used with the distribution's patched kernel source, it verifies that the modules are signed. I could have removed this feature from the spec file but I decided this feature was a good one.

This is not part of the stock 2.6.32.46 kernel and, in fact, does not appear to be included in 3.0.4 either. To use this feature, the code changes have to be isolated from the patched distribution kernel and a new patch needed to be made. After creating a patch from changes made to 54 source files, I added it to the spec file and built signed modules.

Turning off the kABI checker

The CentOS kernel, like the upstream vendor's kernel, is expected to fulfill a given interface contract when built. Anything that would break this contract severely is not allowed to be made as a change to the kernel.5 As a result, the kernel verifies this contract in the spec file.

The standard way to do this is to pass this option to rpmbuild when building the kernel: --without kabichk

As I didn't feel like remembering to specify that every time I build the kernel, I changed the spec file to make sure that the with_kabichk variable set to  . I could have left it in place by copying Module.symvers to the appropriate kABI file in the SOURCES directory and changing the spec file appropriately.

Setting the kernel version correctly

This took me a while to figure out unfortunately. When I built the kernel, the version was being set incorrectly and then this resulted in the system not being able to find modules on boot.

What I figured out in the end was that, for kernel 2.6.32.46, base_sublevel should be set to 32 and stable_update should be set to 46.

grsecurity also adds a file called localversion-grsec that contains the text -grsec. I wrote a patch to remove the file, added it to the spec file, built it, installed it, and...

The system halted on boot

When a CentOS 6 system boots normally, it uses something called dracut. At some point, it tries to mount a partition inside a chroot'd partition. As the kernel is configured to prevent this (through the grsecurity patch), the system would then halt on boot.

As I didn't want to disable this feature permanently and I wanted everything else to work from boot, I wrote a small patch to disable the feature on boot. Adding kernel.grsecurity.chroot_deny_mount = 1 to /etc/sysctl.conf then enabled this feature later in the boot process.

And now it works

After building the RPMs, I installed the kernel and kernel-firmware packages on a test VM, rebooted it, and it came up using the new kernel.

I did find that a lot of messages were being written to the console. Setting kernel.printk to "6 4 1 7" via sysctl corrected this.

What next?

I want to work on backporting some changes from the CentOS 6 kernel to the package I have built. I know that the CentOS 6 kernel contains some features from 2.6.33, e.g. recvmmsg, and some enhancements to KVM. Unfortunately, since the source tarball in the RPM has already been patched, I have to tease out what those patches are. Once I have those patches, I should then check to see there are any grsecurity changes that need to be backported. Since there is no obviously apparent repository for the grsecurity patch, this requires looking at the test patch (currently for the 3.0.4 kernel) and comparing manually.

So where can I get this?

Once I have some things worked out, I will post the RPMs and the SRPM somewhere that it can be downloaded. For now, you'll just have to be patient.

• 1. I have grsecurity installed on another server and I've never gotten around to configuring RBAC. Since a lot of work has gone into the CentOS 6 SELinux policies, I'll just rely on those.
• 2. I've said this before. Some day I'll write a longer post on it.
• 3. I'll try to write about this in the near future.
• 4. You do this whenever you download a source file, right?
• 5. This is why CentOS 5 does not support IPv6 connection tracking without building a custom kernel.

Via Dr. Anton Chukavin's post, I found the 2009 Verizon Business Data Breach Investigations Report. The document is a fascinating read to get an idea of the state of things last year. Chances are that this year will build on last year.

The targeting of financial institutions by organized crime entities is surprising to me and yet not surprising. Based on the behavior of would-be fraud perpetrators I have observed, there are a lot of credit card numbers out in the wild with little rhyme or reason to them. Compromising a financial institution or a merchant account provider, e.g. Heartland, would be an easy way to get credit card numbers into the open. However, the interest in account numbers and PINs is more disturbing and provides less recourse for the victim. This trend can be seen even in the US where ATM skimmers are becoming more prevalent.

The major thing I come away from the report with is: The big fish have a lot more to be worried about in the past. But that doesn't mean the small fish are safe in the water.