On state
Servers are similar to programming objects. A lot of techniques intended for object-oriented program (or just programming in general), e.g. BDD, can apply to building and maintaining servers. After all, just as a given object has a given interface that other objects may access it by and a set of contracts that are fulfilled by that interface, a given server runs at least one remotely accessible service that has a given interface that other systems may access it by and a set of contracts that are fulfilled by its services' interfaces.
An important facet of servers is state. Systems, unlike objects, have a malleable state that affects how they operate. There exist few computer programs whose underlying runtime logic can be completely modified while the program itself is running.
BDD alone is therefore insufficient for modeling a server. While behavior is very important, it is only one aspect of the system. BDD only allows for testing specific, known functionality.1 With systems, we not only have to worry about changes that affect existing functionality but we also have to worry about changes that introduce new functionality.
Tests to exercise and verify the behavior of the system are still important. While behavior is a function of state, we may not be able to easily verify behavior from state alone.
The state of a system should be verifiable. Some stakeholders, particularly operations and security, need to be able to prove that the system is in a given known state and that this state is known to be secure (and compliant with any required standards, e.g. PCI DSS).
Most systems should be documented. Documentation of a system is usually nothing more than a written statement of the system's state (or the state the document author believes the system to be in). Anything documented should be verifiable automatically through a test of either state or behavior. If there are no state-level tests, there is no way to verify that the system's state actually matches the documentation.
As Chris Siebermann states, the configuration specified for a given system in Puppet or Chef or another similar tool is not a complete specification of the system. As a result, it cannot be used to verify the system's state completely.2 And, as stated before, behavior tests are also insufficient for verifying a system's state.
On using RSpec
BDD in Ruby traditionally uses two tools, Cucumber and RSpec. Cucumber is used for the higher-level behavior of the software while RSpec is used for finer-grained behavioral tests.
I believe that using Cucumber for state tests is a good idea. We want to have behavioral tests and Cucumber is a good fit for this. However, we want to keep them separate from our state tests. Otherwise we run the risk of writing state tests in place of behavior tests or vice versa.
RSpec is not particularly intended for state tests, having been written with BDD in mind. However, we can co-opt it for our purposes. Other benefitsof RSpec include its documentation output formatter.
The major problem I have thought of with using RSpec is running tests conditionally. I believe I have this solved. Here's an example of why we want conditional testing:
Example.com's baseline Linux configuration requires that the
usb_storagekernel module be blacklisted. However, off-site backups are done using USB drives. As a result, the backup server, to which these drives are attached, does not have this module blacklisted.
Using RSpec, a first attempt for this aspect of the baseline might look like:
-
shared_examples "baseline Linux configuration" do |options|
-
it "blacklists the usb_storage kernel module" do
-
kernel_module('usb_storage').should be_blacklisted
-
end -
end
Including it for the backup server would look like:
-
host "backup.example.com" do
-
include_examples "baseline Linux configuration" -
end
(Assume that host works just like describe but only functions if the system's hostname matches the parameter.)
This fails because the test expects the usb_storage kernel module to be blacklisted but it should not be on backup.example.com. We have to override the test so that it does not run on the backup server.
A way to implement this may look like:
-
shared_examples "baseline Linux configuration" -
it "blacklists the usb_storage kernel module", :override_usb_storage => false do
-
kernel_module('usb_storage').should be_blacklisted
-
end -
end -
-
host "backup.example.com" do
-
set_tag :override_usb_storage -
-
include_examples "baseline Linux configuration" -
-
it "OVERRIDES baseline and does not blacklist the usb_storage kernel module" do
-
kernel_module('usb_storage').should_not be_blacklisted
-
end -
end
This approach uses RSpec's tagging functionality to skip the test. For this to work, I'm assuming that a function, set_tag, can be written to set tags within RSpec and adjust the filtering appropriately. As long as it is possible to write both this function and the host function above, we can conditionally run tests elegantly and without significant issue. We can also limit tests to only those that should be running on the specific host.
There may be some confusion that arises from overriding or excluding tests in the baseline examples or elsewhere. However, anyone writing documentation should use the tests, not RSpec's output.
On building systems
System requirements come in two flavors:
- Behavioral: The system must do something.
- State: The system must be something.
When dealing with behavioral requirements, the procedure would be similar to traditional BDD: First, write your feature in Cucumber with scenarios for the behavior tests. Make sure they fail. Then drop down to RSpec and write the state tests. Once you get the state tests running, make sure the tests now pass.
For writing state tests, you should follow a procedure like this:
- Write the state test in RSpec.
- If possible, ensure it fails.
- Add the appropriate configuration to your CM tool to get the test to pass. (If you're not using a CM tool, you can do this by hand.If you're not using a CM tool, you should really start using one. It doesn't matter if you use Puppet or Chef or CFEngine or any other offering. Just use something.)
- Apply the changes using your CM tool.
- Ensure the test passes.
- If possible, refactor to make the CM tool configuration simpler.
If you're familiar with BDD or TDD, this process should seem similar (red/green/refactor).
By following this, you end up with a set of tests you can use to prove that the system's behavior and state both match what is expected. If you use something like cucumber-nagios, you can integrate your behavior tests with your monitoring tool.
On dealing with existing systems
If you are dealing with an existing system, you can use the system's existing documentation to write behavior and state tests. You can then run these to verify that the system's behavior and state matches the documentation and make appropriate changes if they don't match.
Since you can't verify that the tests fail first and then pass, the tests you write may not be as accurate as if you were building a new system. This is an unfortunate risk which you will just have to live with. However, having these tests should provide you with more confidence for making changes to the system.
On benefits and tradeoffs
The benefits of this testing approach have been mentioned above: you can prove that the system is behaving in the desired fashion and that it is in the desired state. You can integrate behavior tests with monitoring applications. You can integrate state tests with your existing system audits to hopefully detect unauthorized changes, either by outside attackers or co-workers attempting to avoid the change control process. Together, they can alert you to problems much faster.
State tests provide benefit even if you are using a CM tool. Your configuration management tool will enforce a given state but it may not enforce all aspects of the system that the tests monitor. If a system's state changes out from under the CM tool, the fact that the state has changed is vastly more interesting than having the CM tool restore the state. After all, if your system state suddenly changes, you have someone making unauthorized changes to the system. If you just let the CM tool reset the state, you haven't addressed the incident and this will cause even larger problems later.
The major tradeoff is: writing tests is not free. Each test you write has an up-front time cost. Only you can determine whether or not that cost is worth it. However, to accurately make that judgment, you should also keep in mind the amount of time you may spend troubleshooting future issues that could have been avoided if tests had been in place.
On limitations of this approach
State testing, like other forms of automated testing, suffers from a very significant limitation: The tests can only verify what they have been written to verify. If a given portion of a system isn't tested, the tests will not show if any changes have been made to that part of the system. You should therefore include other methods for checking the system state and configuration, such as file integrity monitoring and package auditing.
Achieving 100% coverage with state testing and similar tools may not be feasible since it would require a significant amount of effort. However, state testing should provide you with significantly more coverage of your systems.
In closing
Computer systems have two interesting aspects, behavior and state. While there is a general approach for testing system behavior, there is not one for testing system state. RSpec may be usable for testing system state but more research needs to be done.
The major benefit of testing system state is that you can prove that a system is set up in a specific way. If your tests match your system documentation, you can use the tests to verify that the system is in the documented configuration. This provides greater certainty that unauthorized changes have not been made to the system.
Finally, I want to leave you with three questions: If you don't test your system's behavior, how do you know it's behaving the way you want it to? If you don't test your system's state, how do you know it matches your documentation? If you don't test your system at all, how do you know someone hasn't compromised your system?