Support fuzzing multiple `@FuzzTest`s at the same time

[cameronbraid]

When I run JAZZER_FUZZ=1 mvn test and JAZZER runs it only fuzzes ONE method annotated with @FuzzTest then exists.

How do I run them all ?

[bertschneider]

Currently, multiple fuzzing runs can not be started cleanly in one JVM instance. As they tend to be quite long running, they should probably be executed individually anyways.

We are thinking about if and how to support this, but in the meantime you would need to script around this restriction.

[cameronbraid]

How do you specify which one to run?

[bertschneider]

If you want to use Maven you can set the testsuite name like:
JAZZER_FUZZ=1 mvn test -Dtest=MyTestClassName.
Or use the JUnit 5 console launcher for more configuration options.

Furthermore, you could also use Jazzer directly (without the JUnit integration), but that's probably not the way for you to go.

[Marcono1234]

For Maven you can run multiple fuzzer tests by configuring multiple executions for the Surefire plugin in the POM, each executing a different test method. For example:

POM plugin configuration (click)

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>3.1.2</version>
  <configuration>
    <environmentVariables>
      <!-- Enable Jazzer fuzzing mode -->
      <JAZZER_FUZZ>1</JAZZER_FUZZ>
    </environmentVariables>
    <!-- Fail if any of the test methods specified below don't exist -->
    <failIfNoTests>true</failIfNoTests>
  </configuration>
  <executions>
    <!-- Skip default execution -->
    <execution>
      <id>default-test</id>
      <configuration>
        <skipTests>true</skipTests>
      </configuration>
    </execution>
    <!-- Explicitly list separate fuzzing test methods -->
    <execution>
      <id>fuzz-MyFuzzingTest#method1</id>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <test>MyFuzzingTest#method1</test>
      </configuration>
    </execution>
    <execution>
      <id>fuzz-MyFuzzingTest#method2</id>
      <goals>
        <goal>test</goal>
      </goals>
      <configuration>
        <test>MyFuzzingTest#method2</test>
      </configuration>
    </execution>
    ...
  </executions>
</plugin>

Though this is a hacky workaround because it is quite error-prone maintaining this, and also seems to have other drawbacks, such as surefire-reports only containing the results for the last executed test method.

---

What is a bit unfortunate though is that this limitation of @FuzzTest is currently not obvious:

• Maven does not seem to show the reason why tests are skipped in the console output by default, so you don't see the "Only one fuzz test can be run at a time..." message
• The README of this repo here which mentions @FuzzTest does not mention this limitation
• The @FuzzTest Javadoc only mentions this near the end of the documentation, without any highlighting
• The fact that examples/junit/src/test/java contains multiple test classes and that some of them also have multiple methods annotated with @FuzzTest (HermeticInstrumentationFuzzTest, InvalidFuzzTests, ValidFuzzTests) gives the impression that they are all executed
  (or at least you probably have that expectation coming from the regular @Test annotation)

[fmeum]

@Marcono1234 I submitted #870 to clarify the docs on this. Another convenient way to run fuzz tests in fuzzing mode is to use cifuzz.

[Marcono1234]

@Marcono1234 I submitted #870 to clarify the docs on this. Another convenient way to run fuzz tests in fuzzing mode is to use cifuzz.

Thanks!

But does cifuzz support running multiple @FuzzTest methods of a Maven project at once? For me that failed with:

❌ Invalid usage: Multiple fuzz tests found in MyTest

Hi @cameronbraid ! Not sure that we answered your question, although @bertschneider is right that most of the fuzzing runs one at a time.
Regardless, a lot has changed in Jazzer since January 2023.
Do you still want to explore this?
Ping me to discuss.
david[dot]merian [at] code-intelligence[dot]com

@Marcono1234 - thank you for this input! Are you still using Jazzer? I'm also curious to understand how you use/have used Jazzer.
Willing to share?
david[dot]merian [at] code-intelligence[dot]com

[Marcono1234]

@David-Merian-CI, I was using Jazzer for some improvised fuzzing of third-party open source projects: I created a dummy Maven project with those projects as dependencies and then created @FuzzTests for fuzzing them. After I had inspected the code of those projects manually, I used Jazzer to validate my findings and it also uncovered additional issues I had overlooked.
As part of this I disclosed potential vulnerabilities to the maintainers of those projects, but no security advisory has been published yet.

It is quite unfortunate that you won't be continuing development of Jazzer as open source project. But if that approach was not profitable for you as company (or if you had other reasons), then that choice is understandable. Would it have helped if Jazzer got more publicity and recognition? For example it was (and still is) used by oss-fuzz, and probably has had a big impact there. Maybe it would have been possible to explicitly mention Jazzer whenever a fuzzing setup using it found a vulnerability? The oss-fuzz maintainers are evaluating options now that Jazzer open source development won't continue (google/oss-fuzz#11652), but I assume they are also in contact with you.

Or would a licensing model similar to GitHub's CodeQL license work for you? That is, you restrict usage of (future) Jazzer versions to open source projects and research, and for everything else a commercial license is necessary. (Though I don't know how easy that is to enforce, and if changing the license will be possible.) And maybe for open source and research projects kindly ask to be explicitly mentioned as used tool to get recognition.

(Thanks also for un-archiving this repository, otherwise I wouldn't have been able to respond here.)