Fuzzing Files for Microsoft SDL

Microsoft SDL states that an application has to consume 100,000 corrupted files without crashing or becoming otherwise unstable. If an application manages to consume 999,999 corrupted files and falls over on the 100,000th, the application has to be repaired and all 100,000 files need to be run again. At first blush this looks to be an immensely time consuming process to both set it up and execute. To set it up there need to be 100,000 files, all unique, and the application has to open and close every one of them.

Holodeck provides a simple solution to the problem which when used in conjunction with automated testing tools like Rational Robot™, allows the test to be accomplished as fast as the automation allows. What Holodeck does is intercepts calls to a specific file that the application under test wants to use, virtualizes it and injects “fuzz” into the file stream, simulating a corruption. The actual file is never touched by the application, just the virtualized version. Holodeck provides the option of generating a new corruption pattern every time the virtualized file is touched, which allows testers to generate test automation scripts for their tooling that simply opens and closes the same file 100,000 times and monitors the result—simple.

Try This (or something similar)
For this example we’ll pick on Notepad because it’s simple and obviously consumes files.

1. Use Notepad to create a file on your desktop called “testpattern.txt” with some text in it; preferably something recognizable like ‘AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA’; and then close Notepad.

2. Start a new Holodeck Project using \Windows\Notepad.exe as the executable.

3. Select Application\Create a New File Corruption Fault from the main menu

4. Browse to and select the “testpattern.txt” file you’ve just created. This will be the source file that Holodeck will virtualize and corrupt. Note that there are several options for fuzzing the stream including random, specific and regular expression corruptions.

5. Check the “Regenerate …” checkbox at the bottom of the dialog. This will cause Holodeck to generate a new corruption pattern each time the file is opened.

6. Click Next. Leave the defaults in place but note that you can generate random fuzz using several techniques that may generate different results depending on the type of application and what it expects from a file.

7. Click Next and then Click Finish.

8. Use Notepad to open the “testpattern.txt”, check the file for differences from the expected results. Select File/New followed by File/Open on “testpattern.txt” and observe the results. If Notepad crashes, select Application/Restart, but recall that it only takes one crash for the application to fail the test.

Automating the Process
Once you’ve got the file corruption set up it’s a very simple matter to generate a script for your automation framework that executes the process as many times as needed. Remember that the rule is that if an application crashes, it has to be repaired and then re-run, so the automation can just stop at the first crash. Holodeck generates log files, records the pattern for each corruption it applies, logs exceptions and will automatically generate a mini-dump as appropriate—so everything a developer needs to recreate a problem is immediately available.