diff --git a/.prettierignore b/.prettierignore index c4582fe53..7dd8e70cc 100644 --- a/.prettierignore +++ b/.prettierignore @@ -3,4 +3,5 @@ build coverage node_modules .idea +.vscode compile_commands.json diff --git a/README.md b/README.md index d396ae2de..a7fed41fa 100644 --- a/README.md +++ b/README.md @@ -58,12 +58,6 @@ To use Jazzer.js in your own project follow these few simple steps: const fuzzerData = data.toString(); myAwesomeCode(fuzzerData); }; - - // Alternatively, using ES6 syntax is also possible - export function fuzz(data /*: Buffer */) { - const fuzzerData = data.toString(); - myAwesomeCode(fuzzerData); - } ``` 3. Start the fuzzer using the fuzz target @@ -77,10 +71,15 @@ To use Jazzer.js in your own project follow these few simple steps: ## Usage Jazzer.js can be used in two ways: Creating dedicated fuzz targets, as shown in -the `Quickstart` section, or integrated into the Jest test framework. +the [`Quickstart`](#quickstart) section, or integrated into the +[Jest test framework](https://jestjs.io/). ### Using test framework integration +**Note**: Using the test framework integration is the easiest and most +convenient way to fuzz your code, hence, it is recommended to use this approach +whenever possible. + To use fuzzing in your normal development workflow, a tight integration with the [Jest test framework](https://jestjs.io/) is provided. This coupling allows the execution of fuzz tests alongside your normal unit tests and seamlessly detect @@ -94,16 +93,24 @@ Jest tests. **Note**: Detailed explanation on how to use the Jest integration can be found at [docs/jest-integration.md](docs/jest-integration.md). -A fuzz test in Jest looks similar to the following example: +A Jest fuzz test, in this case written in TypeScript, looks similar to the +following example: -```js -describe("My function", () => { - it.fuzz("can be fuzzed", (data) => { +```typescript +// file: "Target.fuzz.ts" +import * as target from "./target"; + +describe("Target", () => { + it.fuzz("executes a method", (data: Buffer) => { target.fuzzMe(data); }); }); ``` +**Note**: Please take a look at +[Enabling TypeScript in Jest tests](docs/jest-integration.md#enabling-typescript-jest-tests) +for further information on how to set up Jest fuzz tests written in TypeScript. + ### Using fuzz targets Creating fuzz targets and executing those via CLI commands is straightforward diff --git a/docs/fuzz-targets.md b/docs/fuzz-targets.md index 8d47bf58f..87710ecde 100644 --- a/docs/fuzz-targets.md +++ b/docs/fuzz-targets.md @@ -102,6 +102,11 @@ as long as a module exporting a `fuzz` function is generated. An example on how to use TypeScript to fuzz a library can be found at [examples/js-yaml/package.json](../examples/js-yaml/package.json). +**Note**: Directly executing fuzz targets written in TypeScript is **NOT** +supported! However, it is possible to use the +[Jest integration](jest-integration.md) to execute Jest fuzz tests written in +TypeScript. + ### ⚠️ Using Jazzer.js on pure ESM projects ⚠️ ESM brings a couple of challenges to the table, which are currently not fully diff --git a/docs/jest-integration.md b/docs/jest-integration.md index fc3c3d488..5c6ae9dd1 100644 --- a/docs/jest-integration.md +++ b/docs/jest-integration.md @@ -13,9 +13,9 @@ normal Jest tests. The Jest integration provides two modes of execution, which will be explained in detail further down on this page. -- **Fuzzing Mode**: Fuzzing a function through a Jest test. -- **Regression Mode**: Using initially provided seeds and inputs of found - problems to execute the Jest test with. +- **[Fuzzing Mode](#fuzzing-mode)**: Fuzzing a function through a Jest test. +- **[Regression Mode](#regression-mode)**: Using initially provided seeds and + inputs of found problems to execute the Jest test with. ## Setting up the Jazzer.js Jest integration @@ -87,6 +87,65 @@ which can be specified through the CLI client. } ``` +## Enabling TypeScript Jest tests + +Jest supports execution of tests written in other languages than JavaScript via +dedicated extensions. Probably most prominent is its TypeScript support, which +can be enabled via [`ts-jest`](https://kulshekhar.github.io/ts-jest/). + +We assume you already set up your TypeScript project according to the `ts-jest` +documentation. The following section shows a minimal configuration to enable +TypeScript support for Jest fuzz tests. Furthermore, an example project is +available at +[jest_typescript_integration](../examples/jest_typescript_integration). + +In addition to the configuration shown in the last section, `ts-jest` has to be +added as dev-dependency to the project. + +```shell +npm install --save-dev ts-jest +``` + +The Jazzer.js runner configuration also needs to reference `ts-jest`, most +commonly by setting the `preset` property to `ts-jest`. Also make sure to +actually include test files with the `.fuzz.ts` extension. + +```typescript +{ + displayName: { + name: "Jazzer.js", + color: "cyan", + }, + preset: "ts-jest", + runner: "@jazzer.js/jest-runner", + testEnvironment: "node", + testMatch: ["/*.fuzz.[jt]s"], +}, +``` + +To introduce the `fuzz` function types globally, add the following import to +`globals.d.ts`. This could also be done in the individual test files. + +```typescript +import "@jazzer.js/jest-runner/jest-extension"; +``` + +To provide accurate coverage reports for TypeScript fuzz tests, make sure to +enable source map generation in the TypeScript compiler options: + +```json +{ + "compilerOptions": { + "sourceMap": true + } +} +``` + +These settings should be enough to start writing Jest fuzz tests in TypeScript. + +**Note**: Using custom hooks written in TypeScript is currently not supported, +as those are not pre-processed by Jest. + ## Writing a Jest fuzz test To create a fuzz test, the `fuzz` function on Jest's `test` and `it` can be @@ -160,6 +219,26 @@ describe("My describe", () => { )}; ``` +### TypeScript Jest fuzz tests + +After the setup mentioned previously, Jest fuzz tests can be written in +TypeScript, just as one would expect. + +**Note**: To satisfy TypeScript's type checker, add an import of +`@jazzer.js/jest-runner/jest-extension` in `globals.d.ts` or in the individual +test file. + +```typescript +import "@jazzer.js/jest-runner/jest-extension"; +import * as target from "./target"; + +describe("Target", () => { + it.fuzz("executes a method", (data: Buffer) => { + target.fuzzMe(data); + }); +}); +``` + ### Setup and teardown The Jazzer.js fuzz test runner supports Jest's setup and teardown functions, as @@ -299,7 +378,7 @@ To generate a coverage report, run jest with the `--coverage` flag: npx jest --coverage ``` -Note that unlike for the Jazzer.js CLI Jest only accepts the long flag of +**Note**: Unlike the Jazzer.js CLI, Jest only accepts the long flag of `--coverage`! Additional options for coverage report generation are described in the @@ -337,4 +416,3 @@ reimplemented. - Mock functions - Isolated workers -- Typescript or any other non-Javascript test files diff --git a/examples/jest_typescript_integration/.jazzerjsrc b/examples/jest_typescript_integration/.jazzerjsrc new file mode 100644 index 000000000..b789cbcb5 --- /dev/null +++ b/examples/jest_typescript_integration/.jazzerjsrc @@ -0,0 +1,4 @@ +{ + "includes": ["target"], + "excludes": ["node_modules"] +} diff --git a/examples/jest_typescript_integration/README.md b/examples/jest_typescript_integration/README.md new file mode 100644 index 000000000..f7ae0fb69 --- /dev/null +++ b/examples/jest_typescript_integration/README.md @@ -0,0 +1,63 @@ +# Jest Typscript Integration Example + +Detailed documentation on the Jest integration is available in the main +[Jazzer.js](https://github.com/CodeIntelligenceTesting/jazzer.js/blob/main/docs/jest-integration.md) +documentation. + +## Quickstart + +To use the [Jest](https://jestjs.io/) integration install the +`@jazzer.js/jest-runner` and `ts-jest` packages then configure `jest-runner` as +a dedicated test runner in `package.json` or `jest.config.{ts|js}`. + +The example below shows how to configure the Jazzer.js Jest integration in +combination with the normal Jest runner. + +```json + "jest": { + "projects": [ + { + "displayName": "Jest", + "preset": "ts-jest", + }, + { + "displayName": { + "name": "Jazzer.js", + "color": "cyan", + }, + "preset": "ts-jest", + "runner": "@jazzer.js/jest-runner", + "testEnvironment": "node", + "testMatch": ["/*.fuzz.[jt]s"], + }, + ], + "coveragePathIgnorePatterns": ["/node_modules/", "/dist/"], + "modulePathIgnorePatterns": ["/node_modules", "/dist/"], + } +``` + +Further configuration can be specified in `.jazzerjsrc`, like in any other +project, in the following format: + +```json +{ + "includes": ["*"], + "excludes": ["node_modules"], + [...] +} +``` + +Write a Jest fuzz test like: + +```typescript +// file: jazzerjs.fuzz.ts +import "@jazzer.js/jest-runner/jest-extension"; +describe("My describe", () => { + it.fuzz("My fuzz test", (data: Buffer) => { + target.fuzzMe(data); + }); +}); +``` + +**Note:** the `import` statement extends `jest`'s `It` interface to include the +`fuzz` property and is necessary for TypeScript to compile the test file. diff --git a/examples/jest_typescript_integration/globals.d.ts b/examples/jest_typescript_integration/globals.d.ts new file mode 100644 index 000000000..7483401e2 --- /dev/null +++ b/examples/jest_typescript_integration/globals.d.ts @@ -0,0 +1 @@ +import "@jazzer.js/jest-runner/jest-extension"; diff --git a/examples/jest_typescript_integration/integration.fuzz.ts b/examples/jest_typescript_integration/integration.fuzz.ts new file mode 100644 index 000000000..31e9c6db8 --- /dev/null +++ b/examples/jest_typescript_integration/integration.fuzz.ts @@ -0,0 +1,38 @@ +/* + * Copyright 2023 Code Intelligence GmbH + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// Import the fuzz testing extension definition to compile TS code, +// or import it globally in globals.d.ts, like in this example. +// import "@jazzer.js/jest-runner/jest-extension"; + +import * as target from "./target"; + +describe("Target", () => { + it.fuzz("executes sync methods", (data: Buffer) => { + target.fuzzMe(data); + }); + + it.fuzz("executes async methods", async (data: Buffer) => { + await target.asyncFuzzMe(data); + }); + + it.fuzz( + "executes methods with a done callback", + (data: Buffer, done: (e?: Error) => void) => { + target.callbackFuzzMe(data, done); + } + ); +}); diff --git a/examples/jest_typescript_integration/integration.fuzz/Target/executes_async_methods/seed b/examples/jest_typescript_integration/integration.fuzz/Target/executes_async_methods/seed new file mode 100644 index 000000000..e69de29bb diff --git a/examples/jest_typescript_integration/integration.fuzz/Target/executes_methods_with_a_done_callback/seed b/examples/jest_typescript_integration/integration.fuzz/Target/executes_methods_with_a_done_callback/seed new file mode 100644 index 000000000..e69de29bb diff --git a/examples/jest_typescript_integration/integration.fuzz/Target/executes_sync_methods/seed b/examples/jest_typescript_integration/integration.fuzz/Target/executes_sync_methods/seed new file mode 100644 index 000000000..e69de29bb diff --git a/examples/jest_typescript_integration/integration.test.ts b/examples/jest_typescript_integration/integration.test.ts new file mode 100644 index 000000000..3ad4519ca --- /dev/null +++ b/examples/jest_typescript_integration/integration.test.ts @@ -0,0 +1,39 @@ +/* + * Copyright 2023 Code Intelligence GmbH + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* eslint no-undef: 0 */ + +import * as target from "./target"; + +describe("My describe", () => { + it("My normal Jest test", () => { + expect(1).toEqual(1); + }); + + it("My done callback Jest test", (done) => { + expect(1).toEqual(1); + done(); + }); + + it("My async Jest test", async () => { + expect(1).toEqual(1); + }); + + it("Test target function", () => { + const data = Buffer.from("a"); + target.fuzzMe(data); + }); +}); diff --git a/examples/jest_typescript_integration/jest.config.ts b/examples/jest_typescript_integration/jest.config.ts new file mode 100644 index 000000000..55b2597f6 --- /dev/null +++ b/examples/jest_typescript_integration/jest.config.ts @@ -0,0 +1,25 @@ +import type { Config } from "jest"; + +const config: Config = { + verbose: true, + projects: [ + { + displayName: "Jest", + preset: "ts-jest", + }, + { + displayName: { + name: "Jazzer.js", + color: "cyan", + }, + preset: "ts-jest", + runner: "@jazzer.js/jest-runner", + testEnvironment: "node", + testMatch: ["/*.fuzz.[jt]s"], + }, + ], + coveragePathIgnorePatterns: ["/node_modules/", "/dist/"], + modulePathIgnorePatterns: ["/node_modules", "/dist/"], +}; + +export default config; diff --git a/examples/jest_typescript_integration/package.json b/examples/jest_typescript_integration/package.json new file mode 100644 index 000000000..2a5b2c035 --- /dev/null +++ b/examples/jest_typescript_integration/package.json @@ -0,0 +1,19 @@ +{ + "name": "jest_typescript_integration", + "version": "1.0.0", + "description": "An example showing how Jazzer.js integrates with Jest and TypeScript", + "scripts": { + "build": "tsc", + "dryRun": "jest", + "fuzz": "JAZZER_FUZZ=1 jest --coverage", + "coverage": "jest --coverage" + }, + "devDependencies": { + "@jazzer.js/jest-runner": "file:../../packages/jest-runner", + "@types/jest": "^29.4.0", + "jest": "^29.4.1", + "ts-jest": "^29.0.5", + "ts-node": "^10.9.1", + "typescript": "^4.9.5" + } +} diff --git a/examples/jest_typescript_integration/target.ts b/examples/jest_typescript_integration/target.ts new file mode 100644 index 000000000..662e2c269 --- /dev/null +++ b/examples/jest_typescript_integration/target.ts @@ -0,0 +1,58 @@ +/* + * Copyright 2023 Code Intelligence GmbH + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +export function fuzzMe(data: Buffer) { + const s = data.toString(); + if (s.length !== 16) { + return; + } + if ( + s.slice(0, 8) === "Awesome " && + s.slice(8, 15) === "Fuzzing" && + s[15] === "!" + ) { + throw Error("Welcome to Awesome Fuzzing!"); + } +} + +export function callbackFuzzMe(data: Buffer, done: (e?: Error) => void) { + // Use setImmediate here to unblock the event loop but still have better + // performance compared to setTimeout. + setImmediate(() => { + try { + fuzzMe(data); + done(); + } catch (e: unknown) { + if (e instanceof Error) { + done(e); + } else { + done(new Error(`Error: ${e}`)); + } + } + }); +} + +export async function asyncFuzzMe(data: Buffer) { + return new Promise((resolve, reject) => { + callbackFuzzMe(data, (e?: Error) => { + if (e) { + reject(e); + } else { + resolve(null); + } + }); + }); +} diff --git a/examples/jest_typescript_integration/tsconfig.json b/examples/jest_typescript_integration/tsconfig.json new file mode 100644 index 000000000..7363a7c88 --- /dev/null +++ b/examples/jest_typescript_integration/tsconfig.json @@ -0,0 +1,17 @@ +{ + "compilerOptions": { + "target": "ES2022", + "module": "NodeNext", + "moduleResolution": "node", + "allowJs": true, + "rootDir": ".", + "outDir": "./dist", + "esModuleInterop": true, + "forceConsistentCasingInFileNames": true, + "strict": true, + "skipLibCheck": true, + "declaration": true, + "composite": true, + "sourceMap": true + } +} diff --git a/packages/instrumentor/SourceMapRegistry.ts b/packages/instrumentor/SourceMapRegistry.ts new file mode 100644 index 000000000..4f9dce0e1 --- /dev/null +++ b/packages/instrumentor/SourceMapRegistry.ts @@ -0,0 +1,104 @@ +/* + * Copyright 2023 Code Intelligence GmbH + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import { RawSourceMap } from "source-map"; +import sms from "source-map-support"; + +export interface SourceMaps { + [file: string]: SourceMap | undefined; +} + +export type SourceMap = { + version: number; + sources: string[]; + names: string[]; + sourceRoot?: string | undefined; + sourcesContent?: string[] | undefined; + mappings: string; + file: string; +}; + +// Regex to extract inline source maps from code strings. The regex is based on +// the one used by the convert-source-map library. It captures the base64 +// encoded source map in capture group 5. +const regex = RegExp( + "^\\s*?\\/[/*][@#]\\s+?sourceMappingURL=data:(((?:application|text)\\/json)(?:;charset=([^;,]+?)?)?)?(?:;(base64))?,(.*?)$", + "mg" +); + +/** + * Extracts the inline source map from a code string. + * + * Inline source maps can be added to the end of a code file during offline + * and online transpilation. Babel transformers or the TypeScript compiler + * are examples of this. + */ +export function extractInlineSourceMap(code: string): SourceMap | undefined { + const match = regex.exec(code); + if (match) { + const buf = Buffer.from(match[5], "base64"); + return JSON.parse(buf.toString()); + } +} + +export function toRawSourceMap( + sourceMap?: SourceMap +): RawSourceMap | undefined { + if (sourceMap) { + return { + version: sourceMap.version.toString(), + sources: sourceMap.sources ?? [], + names: sourceMap.names, + sourcesContent: sourceMap.sourcesContent, + mappings: sourceMap.mappings, + }; + } +} + +export class SourceMapRegistry { + private sourceMaps: SourceMaps = {}; + + registerSourceMap(filename: string, sourceMap: SourceMap) { + this.sourceMaps[filename] = sourceMap; + } + + getSourceMap(filename: string): SourceMap | undefined { + return this.sourceMaps[filename]; + } + + /* Installs source-map-support handlers and returns a reset function */ + installSourceMapSupport(): () => void { + // Use the source-map-support library to enable in-memory source maps of + // transformed code and error stack rewrites. + // As there is no way to populate the source map cache of source-map-support, + // an additional buffer is used to pass on the source maps from babel to the + // library. This could be memory intensive and should be replaced by + // tmp source map files, if it really becomes a problem. + sms.install({ + hookRequire: true, + retrieveSourceMap: (source) => { + const sourceMap = toRawSourceMap(this.getSourceMap(source)); + return sourceMap + ? { + map: sourceMap, + url: source, + } + : null; + }, + }); + return sms.resetRetrieveHandlers; + } +} diff --git a/packages/instrumentor/instrument.test.ts b/packages/instrumentor/instrument.test.ts index afb692bfc..a523d3680 100644 --- a/packages/instrumentor/instrument.test.ts +++ b/packages/instrumentor/instrument.test.ts @@ -16,10 +16,8 @@ /* eslint @typescript-eslint/ban-ts-comment:0 */ -import { codeCoverage } from "./plugins/codeCoverage"; -import { MemorySyncIdStrategy } from "./edgeIdStrategy"; +import * as ts from "typescript"; import { Instrumentor } from "./instrument"; -import { sourceCodeCoverage } from "./plugins/sourceCodeCoverage"; describe("shouldInstrument check", () => { it("should consider includes and excludes", () => { @@ -77,7 +75,7 @@ describe("transform", () => { const sourceFileName = "sourcemap-test.js"; const errorLocation = sourceFileName + ":5:13"; const content = ` - module.exports.functionThrowingAnError = function foo () { + module.exports.functionThrowingAnError = function foo () { // eslint-disable-next-line no-constant-condition if (1 < 2) { throw Error("Expected test error"); // error thrown at ${errorLocation} @@ -89,13 +87,7 @@ describe("transform", () => { // import/require without eval. //@ sourceURL=${sourceFileName}`; try { - // Use the codeCoverage plugin to add additional lines, so that the - // resulting error stack does not match the original code anymore. - const result = instrumentor.transform(sourceFileName, content, [ - codeCoverage(new MemorySyncIdStrategy()), - ]); - const fn = eval(result?.code || ""); - fn(); + evalWithInstrumentor(instrumentor, content, sourceFileName); fail("Error expected but not thrown."); } catch (e: unknown) { if (!(e instanceof Error && e.stack)) { @@ -108,46 +100,59 @@ describe("transform", () => { }); }); - describe("transform", () => { - it("should use source maps to correct error stack traces, also with enabled coverage", () => { - withSourceMap((instrumentor: Instrumentor) => { - const sourceFileName = "sourcemap-test002.js"; - const errorLocation = sourceFileName + ":5:13"; - const content = ` - module.exports.functionThrowingAnError = function foo () { - // eslint-disable-next-line no-constant-condition - if (1 < 2) { - throw Error("Expected test error"); // error thrown at ${errorLocation} - } - }; - // sourceURL is required for the snippet to reference a filename during - // eval and so be able to lookup the appropriate source map later on. - // This is only necessary for this test and not when using normal - // import/require without eval. - //@ sourceURL=${sourceFileName}`; - try { - // Use the codeCoverage plugin to add additional lines, so that the - // resulting error stack does not match the original code anymore. - const result = instrumentor.transform(sourceFileName, content, [ - sourceCodeCoverage(sourceFileName), - codeCoverage(new MemorySyncIdStrategy()), - ]); - const fn = eval(result?.code || ""); - fn(); - fail("Error expected but not thrown."); - } catch (e: unknown) { - if (!(e instanceof Error && e.stack)) { - throw e; - } - // Verify that the received error was corrected via a source map - // by checking the original error location. - expect(e.stack).toContain(errorLocation); + it("should use inline source maps to correct error stack traces", () => { + const sourceFileName = "inline-sourcemap-test.ts"; + const errorLocation = sourceFileName + ":7:12"; + const content = ` + interface Foo { + bar: string; + } + export default function foo (): Foo { + if (1 < 2) { + throw new Error("Expected test error"); // error thrown at ${errorLocation} + } else { + return { bar: "baz" }; } - }); + } + // sourceURL is required for the snippet to reference a filename during + // eval and so be able to lookup the appropriate source map later on. + // This is only necessary for this test and not when using normal + // import/require without eval. + //@ sourceURL=${sourceFileName}`; + + const transpiledCode = ts.transpile( + content, + { + fileName: sourceFileName, + inlineSourceMap: true, + }, + sourceFileName + ); + + withSourceMap((instrumentor: Instrumentor) => { + try { + evalWithInstrumentor(instrumentor, transpiledCode, sourceFileName); + fail("Error expected but not thrown."); + } catch (e: unknown) { + if (!(e instanceof Error && e.stack)) { + throw e; + } + expect(e.stack).toContain(errorLocation); + } }); }); }); +function evalWithInstrumentor( + instrumentor: Instrumentor, + content: string, + fileName: string +) { + const result = instrumentor.instrument(content, fileName); + const fn = eval(result); + fn(); +} + function withSourceMap(fn: (instrumentor: Instrumentor) => void) { // @ts-ignore const oldFuzzer = globalThis.Fuzzer; @@ -155,13 +160,12 @@ function withSourceMap(fn: (instrumentor: Instrumentor) => void) { globalThis.Fuzzer = { // @ts-ignore coverageTracker: { - // eslint-disable-next-line @typescript-eslint/no-unused-vars - incrementCounter: (edgeId: number) => { - // ignore + incrementCounter: () => { + // ignored }, }, }; - const instrumentor = new Instrumentor(); + const instrumentor = new Instrumentor([], [], [], true, false); const resetSourceMapHandlers = instrumentor.init(); try { fn(instrumentor); diff --git a/packages/instrumentor/instrument.ts b/packages/instrumentor/instrument.ts index f1c13092b..9ed57defd 100644 --- a/packages/instrumentor/instrument.ts +++ b/packages/instrumentor/instrument.ts @@ -1,5 +1,5 @@ /* - * Copyright 2022 Code Intelligence GmbH + * Copyright 2023 Code Intelligence GmbH * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -14,8 +14,6 @@ * limitations under the License. */ -import sms from "source-map-support"; -import { RawSourceMap } from "source-map"; import { BabelFileResult, PluginItem, @@ -23,18 +21,17 @@ import { transformSync, } from "@babel/core"; import { hookRequire, TransformerOptions } from "istanbul-lib-hook"; +import { hookManager } from "@jazzer.js/hooking"; import { codeCoverage } from "./plugins/codeCoverage"; import { sourceCodeCoverage } from "./plugins/sourceCodeCoverage"; import { compareHooks } from "./plugins/compareHooks"; import { functionHooks } from "./plugins/functionHooks"; -import { hookManager } from "@jazzer.js/hooking"; import { EdgeIdStrategy, MemorySyncIdStrategy } from "./edgeIdStrategy"; - -interface SourceMaps { - [file: string]: RawSourceMap; -} - -const sourceMaps: SourceMaps = {}; +import { + extractInlineSourceMap, + SourceMapRegistry, + toRawSourceMap, +} from "./SourceMapRegistry"; export { EdgeIdStrategy, @@ -49,9 +46,10 @@ export class Instrumentor { private readonly customHooks: string[] = [], private readonly shouldCollectSourceCodeCoverage = false, private readonly isDryRun = false, - private readonly idStrategy: EdgeIdStrategy = new MemorySyncIdStrategy() + private readonly idStrategy: EdgeIdStrategy = new MemorySyncIdStrategy(), + private readonly sourceMapRegistry: SourceMapRegistry = new SourceMapRegistry() ) { - // This is our default case where we want to include everthing and exclude the "node_modules" folder. + // This is our default case where we want to include everything and exclude the "node_modules" folder. if (includes.length === 0 && excludes.length === 0) { includes.push("*"); excludes.push("node_modules"); @@ -64,14 +62,16 @@ export class Instrumentor { if (this.includes.includes("jazzer.js")) { this.unloadInternalModules(); } - return Instrumentor.installSourceMapSupport(); + return this.sourceMapRegistry.installSourceMapSupport(); } instrument(code: string, filename: string): string { + // Extract inline source map from code string and use it as input source map + // in further transformations. + const inputSourceMap = extractInlineSourceMap(code); const transformations: PluginItem[] = []; const shouldInstrumentFile = this.shouldInstrumentForFuzzing(filename); - if (shouldInstrumentFile) { transformations.push(codeCoverage(this.idStrategy), compareHooks); } @@ -81,7 +81,12 @@ export class Instrumentor { } if (this.shouldCollectCodeCoverage(filename)) { - transformations.push(sourceCodeCoverage(filename)); + transformations.push( + sourceCodeCoverage( + filename, + this.asInputSourceOption(toRawSourceMap(inputSourceMap)) + ) + ); } if (shouldInstrumentFile) { @@ -89,7 +94,12 @@ export class Instrumentor { } const transformedCode = - this.transform(filename, code, transformations)?.code || code; + this.transform( + filename, + code, + transformations, + this.asInputSourceOption(inputSourceMap) + )?.code || code; if (shouldInstrumentFile) { this.idStrategy.commitIdCount(filename); @@ -98,6 +108,17 @@ export class Instrumentor { return transformedCode; } + // eslint-disable-next-line @typescript-eslint/no-explicit-any + private asInputSourceOption(inputSourceMap: any): any { + // Empty input source maps mess up the coverage report. + if (inputSourceMap) { + return { + inputSourceMap, + }; + } + return {}; + } + transform( filename: string, code: string, @@ -115,46 +136,17 @@ export class Instrumentor { ...options, }); if (result?.map) { - const sourceMap = result.map; - sourceMaps[filename] = { - version: sourceMap.version.toString(), - sources: sourceMap.sources ?? [], - names: sourceMap.names, - sourcesContent: sourceMap.sourcesContent, - mappings: sourceMap.mappings, - }; + this.sourceMapRegistry.registerSourceMap(filename, result.map); } return result; } - /* Installs source-map-support handlers and returns a reset function */ - static installSourceMapSupport(): () => void { - // Use the source-map-support library to enable in-memory source maps of - // transformed code and error stack rewrites. - // As there is no way to populate the source map cache of source-map-support, - // an additional buffer is used to pass on the source maps from babel to the - // library. This could be memory intensive and should be replaced by - // tmp source map files, if it really becomes a problem. - sms.install({ - hookRequire: true, - retrieveSourceMap: (source) => { - if (sourceMaps[source]) { - return { - map: sourceMaps[source], - url: source, - }; - } - return null; - }, - }); - return sms.resetRetrieveHandlers; - } - private unloadInternalModules() { console.log( "DEBUG: Unloading internal Jazzer.js modules for instrumentation..." ); [ + "@jazzer.js/bug-detectors", "@jazzer.js/core", "@jazzer.js/fuzzer", "@jazzer.js/hooking", @@ -206,6 +198,10 @@ export function registerInstrumentor(instrumentor: Instrumentor) { () => true, (code: string, opts: TransformerOptions): string => { return instrumentor.instrument(code, opts.filename); - } + }, + // required to allow jest to run typescript files + // jest's typescript integration will transform the typescript into javascript before giving it to the + // instrumentor but the filename will still have a .ts extension + { extensions: [".js", ".mjs", ".cjs", ".ts", ".mts", ".cts"] } ); } diff --git a/packages/instrumentor/package.json b/packages/instrumentor/package.json index 58121fba4..053184347 100644 --- a/packages/instrumentor/package.json +++ b/packages/instrumentor/package.json @@ -31,7 +31,8 @@ "@types/istanbul-lib-instrument": "^1.7.4", "@types/node": "^20.3.1", "@types/proper-lockfile": "^4.1.2", - "@types/source-map-support": "^0.5.6" + "@types/source-map-support": "^0.5.6", + "typescript": "^5.0.4" }, "engines": { "node": ">= 14.0.0", diff --git a/packages/jest-runner/jest-extension.ts b/packages/jest-runner/jest-extension.ts new file mode 100644 index 000000000..473c981af --- /dev/null +++ b/packages/jest-runner/jest-extension.ts @@ -0,0 +1,12 @@ +import { FuzzTest } from "./fuzz"; + +// Global definition of the Jest fuzz test extension function. +// This is required to allow the Typescript compiler to recognize it. +declare global { + // eslint-disable-next-line @typescript-eslint/no-namespace + namespace jest { + interface It { + fuzz: FuzzTest; + } + } +} diff --git a/packages/jest-runner/readme.md b/packages/jest-runner/readme.md index d44f05a5d..e3dcad5f9 100644 --- a/packages/jest-runner/readme.md +++ b/packages/jest-runner/readme.md @@ -1,50 +1,21 @@ # Jest Fuzz Runner -Custom runner that executes fuzz tests in regression or fuzzing mode. This -package also augments the available [Jest](https://jestjs.io/) test functions -with a `fuzz` extension. +Custom Jest runner to executes fuzz tests via Jazzer.js, detailed documentation +can be found at the +[Jazzer.js GitHub page](https://github.com/CodeIntelligenceTesting/jazzer.js). -## Idea +A fuzz test in Jest, in this case written in TypeScript, would look similar to +the following example: -- Define a custom runner that execute fuzz tests, `it.fuzz`, in regression or - fuzzing mode. -- Registers our instrumentation during startup to instrument code (both cases) -- Registers own implementations of global test functions, like `test` and `it`, - which register functions to execute when the fuzz test module is loaded -- Executes the registered functions in one of the modes and reports the - `TestResult`s back to Jest -- In regression mode: - - Uses an input dir located besides the tests to load seed files -- In fuzzing mode: - - Provides a custom fuzz target function which is called by the fuzzer and - passes input to the actual fuzz test +```typescript +// file: "Target.fuzz.ts +// Import the fuzz testing extension to compile TS code. +import "@jazzer.js/jest-runner/jest-extension"; +import * as target from "./target"; -## How a test is executed - -1. "jest" exported in "packages/jest-cli", calls "run" from same package -2. Calls "runCLI" from "packages/jest-core/cli" -3. Calls "runJest' from "packages/jest-core" -4. Loads all tests form FS -5. Calls "scheduleTests" from "packages/jest-core" "TestScheduler" -6. Transforms module (?what does it do?) -7. Executes test file with runner -8. Runner provides feedback via events ("test-file-start", "test-file-success", - "test-case-result", ...) or callbacks -9. "TestRunner" from "packages/jest-runner" "index" implements "runTests" -10. Executes "runTestInternal" of "packages/jest-runner" "runTests" -11. Uses "jestAdapter" of - "packages/jest-circus/src/legacy-code-todo-rewrite/jestAdapter" to execute - test (or jasmine for old configs) - -## Useful links - -- [Light Runner](https://github.com/nicolo-ribaudo/jest-light-runner) - Light weight `jest-circus` runner. Our runner could use a similar approach. -- [Create Jest Runner](https://github.com/jest-community/create-jest-runner) - Wrapper to create test runners, probably not powerful enough -- [Test Result Type](https://github.com/facebook/jest/blob/main/packages/jest-types/src/TestResult.ts) -- [Jest each](https://github.com/facebook/jest/tree/main/packages/jest-each) - The fuzz extension works similar to `each`. -- [VS Code Jest extension](https://github.com/jest-community/vscode-jest) - This extension can be used to investigate how the IDE integration of Jest - tests work. +describe("Target", () => { + it.fuzz("executes a method", (data: Buffer) => { + target.fuzzMe(data); + }); +}); +``` diff --git a/packages/jest-runner/worker.ts b/packages/jest-runner/worker.ts index ecb6916e7..9b2fb3a73 100644 --- a/packages/jest-runner/worker.ts +++ b/packages/jest-runner/worker.ts @@ -27,6 +27,8 @@ import { inspect } from "util"; import { fuzz, FuzzerStartError, skip } from "./fuzz"; import { cleanupJestRunnerStack, removeTopFramesFromError } from "./errorUtils"; import { Finding } from "@jazzer.js/bug-detectors"; +import { createScriptTransformer } from "@jest/transform"; +import "./jest-extension"; function isGeneratorFunction(obj?: unknown): boolean { return ( @@ -147,7 +149,14 @@ export class JazzerWorker { private async loadTests(test: Test): Promise { circus.resetState(); - await this.importFile(test.path); + + // Don't cache transformed files, as that will interfere with the internal + // transformations. Config is read only, so a copy is needed. + const config = { ...test.context.config }; + config.cache = false; + const transformer = await createScriptTransformer(config); + await transformer.requireAndTranspileModule(test.path); + return circus.getState(); } diff --git a/tests/code_coverage/coverage.test.js b/tests/code_coverage/coverage.test.js index 83cb80e5e..4e54c573c 100644 --- a/tests/code_coverage/coverage.test.js +++ b/tests/code_coverage/coverage.test.js @@ -14,232 +14,157 @@ * limitations under the License. */ -/* eslint no-undef: 0 */ -// eslint-disable-next-line @typescript-eslint/no-var-requires +/* eslint no-undef: 0, @typescript-eslint/no-var-requires: 0 */ const fs = require("fs"); -// eslint-disable-next-line @typescript-eslint/no-var-requires -const { spawnSync } = require("child_process"); -// eslint-disable-next-line @typescript-eslint/no-var-requires const path = require("path"); +const { spawnSync } = require("child_process"); // current working directory const testDirectory = path.join(process.cwd(), "sample_fuzz_test"); const defaultCoverageDirectory = path.join(testDirectory, "coverage"); const expectedCoverageDirectory = path.join(testDirectory, "expected_coverage"); -const libFile = path.join(testDirectory, "lib.js"); -const targetFile = path.join(testDirectory, "fuzz.js"); -const jestRunnerFile = path.join(testDirectory, "codeCoverage.fuzz.js"); -const hookFile = path.join(testDirectory, "custom-hooks.js"); -describe("Source code coverage reports for regular fuzz targets", () => { - it("Expect no coverage reports", () => { - executeFuzzTest(false, false, false, false, false); - expect(fs.existsSync(defaultCoverageDirectory)).toBe(false); - }); - it("Want coverage, but no includes active. Expect no coverage reports", () => { - executeFuzzTest(false, false, false, false, true); - expect(fs.existsSync(defaultCoverageDirectory)).toBe(false); - }); - it("Want coverage in dry run mode, no custom hooks", () => { - executeFuzzTest(true, true, true, false, true); - expect(fs.existsSync(defaultCoverageDirectory)).toBe(true); - const coverageJson = readCoverageJson(defaultCoverageDirectory); - const expectedCoverage = readExpectedCoverage("fuzz+lib.json"); - expect(coverageJson).toBeTruthy(); - // lib.js - expect(coverageJson[libFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[libFile], expectedCoverage["lib.js"]); - // fuzz.js - expect(coverageJson[targetFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[targetFile], expectedCoverage["fuzz.js"]); - // custom-hooks.js - expect(coverageJson[hookFile]).toBeFalsy(); - }); +const libFile = "lib.js"; +const targetFile = "fuzz.js"; +const testFile = "codeCoverage.fuzz.js"; +const otherTestFile = "otherCodeCoverage.fuzz.ts"; +const hookFile = "custom-hooks.js"; - it("Want coverage in dry run mode, with custom hooks", () => { - executeFuzzTest(true, true, true, true, true); - expect(fs.existsSync(defaultCoverageDirectory)).toBe(true); - const coverageJson = readCoverageJson(defaultCoverageDirectory); - const expectedCoverage = readExpectedCoverage("fuzz+lib+customHooks.json"); - expect(coverageJson).toBeTruthy(); - // lib.js - expect(coverageJson[libFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[libFile], expectedCoverage["lib.js"]); - // fuzz.js - expect(coverageJson[targetFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[targetFile], expectedCoverage["fuzz.js"]); - // custom-hooks.js - // work in dry run mode - expect(coverageJson[hookFile]).toBeTruthy(); - expectEqualCoverage( - coverageJson[hookFile], - expectedCoverage["custom-hooks.js"] - ); - }); +describe("Source code coverage reports", () => { + describe("for regular fuzz targets", () => { + it("expect no coverage reports", () => { + executeFuzzTest(false, false, false, false, false); + expect(defaultCoverageDirectory).not.toBeCreated(); + }); - it("Want coverage, instrumentation enabled, with custom hooks", () => { - executeFuzzTest(false, true, true, true, true); - expect(fs.existsSync(defaultCoverageDirectory)).toBe(true); - const coverageJson = readCoverageJson(defaultCoverageDirectory); - const expectedCoverage = readExpectedCoverage("fuzz+lib+customHooks.json"); - expect(coverageJson).toBeTruthy(); - // lib.js - expect(coverageJson[libFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[libFile], expectedCoverage["lib.js"]); - // fuzz.js - expect(coverageJson[targetFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[targetFile], expectedCoverage["fuzz.js"]); - // custom-hooks.js - // work in dry run mode - expect(coverageJson[hookFile]).toBeTruthy(); - expectEqualCoverage( - coverageJson[hookFile], - expectedCoverage["custom-hooks.js"] - ); - }); + it("want coverage, but no includes active. Expect no coverage reports", () => { + executeFuzzTest(false, false, false, false, true); + expect(defaultCoverageDirectory).not.toBeCreated(); + }); - it("Want coverage in a non-default directory, instrumentation enabled, with custom hooks", () => { - const coverageDirectory = "coverage002"; - const coverageAbsoluteDirectory = path.join( - testDirectory, - coverageDirectory - ); - executeFuzzTest(false, true, true, true, true, coverageDirectory); - expect(fs.existsSync(coverageAbsoluteDirectory)).toBe(true); - const coverageJson = readCoverageJson(coverageAbsoluteDirectory); - const expectedCoverage = readExpectedCoverage("fuzz+lib+customHooks.json"); - expect(coverageJson).toBeTruthy(); - // lib.js - expect(coverageJson[libFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[libFile], expectedCoverage["lib.js"]); - // fuzz.js - expect(coverageJson[targetFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[targetFile], expectedCoverage["fuzz.js"]); - // custom-hooks.js - // work in dry run mode - expect(coverageJson[hookFile]).toBeTruthy(); - expectEqualCoverage( - coverageJson[hookFile], - expectedCoverage["custom-hooks.js"] - ); - }); -}); + it("want coverage in dry run mode, no custom hooks", () => { + executeFuzzTest(true, true, true, false, true); + expect(defaultCoverageDirectory).toBeCreated(); + const coverageJson = readCoverageJson(defaultCoverageDirectory); + const expectedCoverage = readExpectedCoverage("fuzz+lib.json"); + expect(libFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(targetFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(hookFile).toHaveMissingCoverageIn(coverageJson); + }); -describe("Source code coverage reports for our custom Jest runner", () => { - it("Jest runner: Expect no coverage reports", () => { - const coverageDirectory = defaultCoverageDirectory; - executeJestRunner(false, false, false, true); - expect(fs.existsSync(coverageDirectory)).toBe(true); - const coverageJson = readCoverageJson(coverageDirectory); - // Jest generates an empty coverage report (unlike our non-jest fuzzer) - expect(coverageJson).toBeTruthy(); - expect(coverageJson).toStrictEqual({}); - expect(coverageJson[targetFile]).toBeFalsy(); - expect(coverageJson[targetFile]).toBeFalsy(); - expect(coverageJson[hookFile]).toBeFalsy(); - }); + it("want coverage in dry run mode, with custom hooks", () => { + executeFuzzTest(true, true, true, true, true); + expect(defaultCoverageDirectory).toBeCreated(); + const coverageJson = readCoverageJson(defaultCoverageDirectory); + const expectedCoverage = readExpectedCoverage( + "fuzz+lib+customHooks.json" + ); + expect(libFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(targetFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(hookFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + }); - it("Jest runner: want coverage, no custom hooks", () => { - const coverageDirectory = defaultCoverageDirectory; - executeJestRunner(true, true, false, true); - expect(fs.existsSync(coverageDirectory)).toBe(true); - const coverageJson = readCoverageJson(coverageDirectory); - const expectedCoverage = readExpectedCoverage( - "fuzz+lib+codeCoverage-fuzz.json" - ); - expect(coverageJson).toBeTruthy(); - // lib.js - expect(coverageJson[libFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[libFile], expectedCoverage["lib.js"]); - // fuzz.js - expect(coverageJson[targetFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[targetFile], expectedCoverage["fuzz.js"]); - // codeCoverage.fuzz.js (the main fuzz test) - expect(coverageJson[targetFile]).toBeTruthy(); - expectEqualCoverage( - coverageJson[jestRunnerFile], - expectedCoverage["codeCoverage.fuzz.js"] - ); - // custom-hooks.js - expect(coverageJson[hookFile]).toBeFalsy(); + it("want coverage, instrumentation enabled, with custom hooks", () => { + executeFuzzTest(false, true, true, true, true); + expect(defaultCoverageDirectory).toBeCreated(); + const coverageJson = readCoverageJson(defaultCoverageDirectory); + const expectedCoverage = readExpectedCoverage( + "fuzz+lib+customHooks.json" + ); + expect(libFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(targetFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(hookFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + }); + + it("want coverage in a non-default directory, instrumentation enabled, with custom hooks", () => { + const coverageDirectory = "coverage002"; + const coverageAbsoluteDirectory = path.join( + testDirectory, + coverageDirectory + ); + executeFuzzTest(false, true, true, true, true, coverageDirectory); + expect(fs.existsSync(coverageAbsoluteDirectory)).toBe(true); + const coverageJson = readCoverageJson(coverageAbsoluteDirectory); + const expectedCoverage = readExpectedCoverage( + "fuzz+lib+customHooks.json" + ); + expect(libFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(targetFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(hookFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + }); }); - it("Jest runner: want coverage, with custom hooks", () => { - const coverageDirectory = defaultCoverageDirectory; - executeJestRunner(true, true, false, true); - expect(fs.existsSync(coverageDirectory)).toBe(true); - const coverageJson = readCoverageJson(coverageDirectory); - const expectedCoverage = readExpectedCoverage( - "fuzz+lib+codeCoverage-fuzz.json" - ); - expect(coverageJson).toBeTruthy(); - // lib.js - expect(coverageJson[libFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[libFile], expectedCoverage["lib.js"]); - // fuzz.js - expect(coverageJson[targetFile]).toBeTruthy(); - expectEqualCoverage(coverageJson[targetFile], expectedCoverage["fuzz.js"]); - // codeCoverage.fuzz.js (the main fuzz test) - expect(coverageJson[targetFile]).toBeTruthy(); - expectEqualCoverage( - coverageJson[jestRunnerFile], - expectedCoverage["codeCoverage.fuzz.js"] - ); - // custom-hooks.js - expect(coverageJson[hookFile]).toBeFalsy(); + describe("for our custom Jest runner", () => { + it("Expect no coverage reports", () => { + executeJestRunner("**.fuzz.js", false, false, true); + expect(defaultCoverageDirectory).toBeCreated(); + const coverageJson = readCoverageJson(defaultCoverageDirectory); + // Jest generates an empty coverage report (unlike our non-jest fuzzer) + expect(coverageJson).toStrictEqual({}); + }); + + it("want coverage, no custom hooks", () => { + executeJestRunner("**.fuzz.js", true, true, true); + expect(defaultCoverageDirectory).toBeCreated(); + const coverageJson = readCoverageJson(defaultCoverageDirectory); + const expectedCoverage = readExpectedCoverage( + "fuzz+lib+codeCoverage-fuzz.json" + ); + expect(libFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(targetFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(testFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(hookFile).toHaveMissingCoverageIn(coverageJson); + }); + + it("want coverage, with custom hooks", () => { + executeJestRunner("**.fuzz.js", true, true, true); + expect(defaultCoverageDirectory).toBeCreated(); + const coverageJson = readCoverageJson(defaultCoverageDirectory); + const expectedCoverage = readExpectedCoverage( + "fuzz+lib+codeCoverage-fuzz.json" + ); + expect(libFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(targetFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(testFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(hookFile).toHaveMissingCoverageIn(coverageJson); + }); + + it("want coverage for TypeScript fuzz test", () => { + executeJestRunner("**.fuzz.ts", true, true, true); + expect(defaultCoverageDirectory).toBeCreated(); + const coverageJson = readCoverageJson(defaultCoverageDirectory); + const expectedCoverage = readExpectedCoverage( + "fuzz+lib+otherCodeCoverage-fuzz.json" + ); + expect(libFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(targetFile).toHaveEqualCoverageIn(coverageJson, expectedCoverage); + expect(otherTestFile).toHaveEqualCoverageIn( + coverageJson, + expectedCoverage + ); + expect(hookFile).toHaveMissingCoverageIn(coverageJson); + }); }); }); -/** - * @param {string} coverageDirectory - */ function readCoverageJson(coverageDirectory) { - return JSON.parse( + const coverageJson = JSON.parse( fs .readFileSync(path.join(coverageDirectory, "coverage-final.json")) .toString() ); + expect(coverageJson).toBeTruthy(); + return coverageJson; } -/** - * @param {string} name - */ function readExpectedCoverage(name) { return JSON.parse( fs.readFileSync(path.join(expectedCoverageDirectory, name)).toString() ); } -/** - * @param {{ statementMap: any; s: any; fnMap: any; f: any; branchMap: any; b: any; }} coverage - * @param {{ statementMap: any; s: any; fnMap: any; f: any; branchMap: any; b: any; }} expectedCoverage - */ -function expectEqualCoverage(coverage, expectedCoverage) { - expect(coverage.statementMap).toStrictEqual(expectedCoverage.statementMap); - expect(coverage.s).toStrictEqual(expectedCoverage.s); - expect(coverage.fnMap).toStrictEqual(expectedCoverage.fnMap); - expect(coverage.f).toStrictEqual(expectedCoverage.f); - expect(coverage.branchMap).toStrictEqual(expectedCoverage.branchMap); - expect(coverage.b).toStrictEqual(expectedCoverage.b); -} - -/** - * @param {boolean} includeLib - * @param {boolean} includeTarget - * @param {boolean} useCustomHooks - * @param {boolean} _coverage - */ -function executeJestRunner( - includeLib, - includeTarget, - useCustomHooks, - _coverage, - coverageOutputDir = "coverage", - excludePattern = ["nothing"], - verbose = false -) { +function removeCoverageDir(coverageOutputDir) { try { - // remove the coverage folder if it exists fs.rmSync(path.join(testDirectory, coverageOutputDir), { recursive: true, force: true, @@ -247,17 +172,29 @@ function executeJestRunner( } catch (err) { // ignore } +} + +function executeJestRunner( + testMatch, + includeLib = true, + includeTarget = true, + coverage = true, + useCustomHooks = [], + coverageOutputDir = "coverage", + excludePattern = [], + verbose = false +) { + removeCoverageDir(coverageOutputDir); const includes = []; - if (includeLib) includes.push("lib.js"); - if (includeTarget) includes.push("fuzz.js"); + if (includeLib) includes.push(libFile); + if (includeTarget) includes.push(targetFile, "fuzz.ts"); if (!includeLib && !includeTarget) includes.push("nothing"); const config = { includes: includes, excludes: excludePattern, - fuzzerOptions: [], - customHooks: useCustomHooks ? ["custom-hooks.js"] : [], + customHooks: useCustomHooks, }; // write the config file, overwriting any existing one fs.writeFileSync( @@ -265,7 +202,8 @@ function executeJestRunner( JSON.stringify(config) ); - let command = ["jest", "--coverage"]; + const cov = coverage ? "--coverage" : ""; + const command = ["jest", cov, `--testMatch "${testMatch}"`]; const process = spawnSync("npx", command, { stdio: "pipe", cwd: testDirectory, @@ -274,13 +212,6 @@ function executeJestRunner( if (verbose) console.log(process.output.toString()); } -/** - * @param {boolean} dryRun - * @param {boolean} includeLib - * @param {boolean} includeTarget - * @param {boolean} useCustomHooks - * @param {boolean} coverage - */ function executeFuzzTest( dryRun, includeLib, @@ -291,25 +222,17 @@ function executeFuzzTest( excludePattern = "nothing", verbose = false ) { - try { - // remove the coverage folder if it exists - fs.rmSync(path.join(testDirectory, coverageOutputDir), { - recursive: true, - force: true, - }); - } catch (err) { - // ignore - } + removeCoverageDir(coverageOutputDir); let options = ["jazzer", "fuzz", "-e", excludePattern, "--corpus", "corpus"]; // add dry run option if (dryRun) options.push("-d"); if (includeLib) { options.push("-i"); - options.push("lib.js"); + options.push(libFile); } if (includeTarget) { options.push("-i"); - options.push("fuzz.js"); + options.push(targetFile); } if (!includeLib && !includeTarget) { options.push("-i"); @@ -329,7 +252,6 @@ function executeFuzzTest( } options.push("--"); options.push("-runs=0"); - // eslint-disable-next-line @typescript-eslint/no-unused-vars const process = spawnSync("npx", options, { stdio: "pipe", cwd: testDirectory, @@ -337,3 +259,25 @@ function executeFuzzTest( }); if (verbose) console.log(process.output.toString()); } + +expect.extend({ + toHaveEqualCoverageIn(file, actualCoverage, expectedCoverage) { + const actual = actualCoverage[path.join(testDirectory, file)]; + const expected = expectedCoverage[file]; + expect(actual).toBeDefined(); + expect(actual.statementMap).toStrictEqual(expected.statementMap); + expect(actual.s).toStrictEqual(expected.s); + expect(actual.fnMap).toStrictEqual(expected.fnMap); + expect(actual.f).toStrictEqual(expected.f); + expect(actual.branchMap).toStrictEqual(expected.branchMap); + expect(actual.b).toStrictEqual(expected.b); + return { pass: true }; + }, + toHaveMissingCoverageIn(file, actualCoverage) { + expect(actualCoverage[path.join(testDirectory, file)]).toBeUndefined(); + return { pass: true }; + }, + toBeCreated(dir) { + return { pass: fs.existsSync(dir) }; + }, +}); diff --git a/tests/code_coverage/sample_fuzz_test/expected_coverage/fuzz+lib+otherCodeCoverage-fuzz.json b/tests/code_coverage/sample_fuzz_test/expected_coverage/fuzz+lib+otherCodeCoverage-fuzz.json new file mode 100644 index 000000000..49d8c61ac --- /dev/null +++ b/tests/code_coverage/sample_fuzz_test/expected_coverage/fuzz+lib+otherCodeCoverage-fuzz.json @@ -0,0 +1,187 @@ +{ + "otherCodeCoverage.fuzz.ts": { + "statementMap": { + "0": { + "start": { "line": 17, "column": 0 }, + "end": { "line": 17, "column": 47 } + }, + "1": { + "start": { "line": 18, "column": 0 }, + "end": { "line": 18, "column": 33 } + }, + "2": { + "start": { "line": 20, "column": 0 }, + "end": { "line": 24, "column": 3 } + }, + "3": { + "start": { "line": 21, "column": 1 }, + "end": { "line": 23, "column": 4 } + }, + "4": { + "start": { "line": 22, "column": 2 }, + "end": { "line": 22, "column": 13 } + } + }, + "fnMap": { + "0": { + "name": "(anonymous_0)", + "decl": { + "start": { "line": 20, "column": 24 }, + "end": { "line": 20, "column": 27 } + }, + "loc": { + "start": { "line": 20, "column": 29 }, + "end": { "line": 24, "column": 1 } + } + }, + "1": { + "name": "(anonymous_1)", + "decl": { + "start": { "line": 21, "column": 27 }, + "end": { "line": 21, "column": 28 } + }, + "loc": { + "start": { "line": 21, "column": 44 }, + "end": { "line": 23, "column": 2 } + } + } + }, + "branchMap": {}, + "s": { + "0": 1, + "1": 1, + "2": 1, + "3": 1, + "4": 2 + }, + "f": { + "0": 1, + "1": 2 + }, + "b": {} + }, + "fuzz.js": { + "statementMap": { + "0": { + "start": { "line": 18, "column": 12 }, + "end": { "line": 18, "column": 28 } + }, + "1": { + "start": { "line": 23, "column": 0 }, + "end": { "line": 29, "column": 2 } + }, + "2": { + "start": { "line": 24, "column": 1 }, + "end": { "line": 24, "column": 41 } + }, + "3": { + "start": { "line": 25, "column": 1 }, + "end": { "line": 27, "column": 2 } + }, + "4": { + "start": { "line": 26, "column": 2 }, + "end": { "line": 26, "column": 9 } + }, + "5": { + "start": { "line": 28, "column": 1 }, + "end": { "line": 28, "column": 18 } + } + }, + "fnMap": { + "0": { + "name": "(anonymous_0)", + "decl": { + "start": { "line": 23, "column": 22 }, + "end": { "line": 23, "column": 23 } + }, + "loc": { + "start": { "line": 23, "column": 38 }, + "end": { "line": 29, "column": 1 } + }, + "line": 23 + } + }, + "branchMap": { + "0": { + "loc": { + "start": { "line": 25, "column": 1 }, + "end": { "line": 27, "column": 2 } + }, + "type": "if", + "locations": [ + { + "start": { "line": 25, "column": 1 }, + "end": { "line": 27, "column": 2 } + }, + { "start": {}, "end": {} } + ], + "line": 25 + } + }, + "s": { "0": 1, "1": 1, "2": 2, "3": 2, "4": 1, "5": 1 }, + "f": { "0": 2 }, + "b": { "0": [1, 1] }, + "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9", + "hash": "d5b411e8de7efcd2798a7cd78efd7bd8347f647a" + }, + "lib.js": { + "statementMap": { + "0": { + "start": { "line": 2, "column": 1 }, + "end": { "line": 2, "column": 29 } + }, + "1": { + "start": { "line": 3, "column": 1 }, + "end": { "line": 5, "column": 2 } + }, + "2": { + "start": { "line": 4, "column": 2 }, + "end": { "line": 4, "column": 11 } + }, + "3": { + "start": { "line": 6, "column": 1 }, + "end": { "line": 6, "column": 11 } + }, + "4": { + "start": { "line": 9, "column": 0 }, + "end": { "line": 11, "column": 2 } + } + }, + "fnMap": { + "0": { + "name": "foo", + "decl": { + "start": { "line": 1, "column": 9 }, + "end": { "line": 1, "column": 12 } + }, + "loc": { + "start": { "line": 1, "column": 16 }, + "end": { "line": 7, "column": 1 } + }, + "line": 1 + } + }, + "branchMap": { + "0": { + "loc": { + "start": { "line": 3, "column": 1 }, + "end": { "line": 5, "column": 2 } + }, + "type": "if", + "locations": [ + { + "start": { "line": 3, "column": 1 }, + "end": { "line": 5, "column": 2 } + }, + { "start": {}, "end": {} } + ], + "line": 3 + } + }, + "s": { "0": 1, "1": 1, "2": 1, "3": 0, "4": 1 }, + "f": { "0": 1 }, + "b": { "0": [1, 0] }, + "_coverageSchema": "1a1c01bbd47fc00a2c39e90264f33305004495a9", + "hash": "9606a42b6e4a5e9a5c23554dda63403d86a4a9a2" + } +} diff --git a/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz.ts b/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz.ts new file mode 100644 index 000000000..6e5a26d7d --- /dev/null +++ b/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz.ts @@ -0,0 +1,24 @@ +/* + * Copyright 2023 Code Intelligence GmbH + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import "@jazzer.js/jest-runner/jest-extension"; +import { fuzz } from "./fuzz.js"; + +describe("My describe", () => { + test.fuzz("My fuzz test", (data: Buffer) => { + fuzz(data); + }); +}); diff --git a/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz/My_describe/My_fuzz_test/a b/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz/My_describe/My_fuzz_test/a new file mode 100644 index 000000000..2e65efe2a --- /dev/null +++ b/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz/My_describe/My_fuzz_test/a @@ -0,0 +1 @@ +a \ No newline at end of file diff --git a/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz/My_describe/My_fuzz_test/b b/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz/My_describe/My_fuzz_test/b new file mode 100644 index 000000000..bb5d25b4d --- /dev/null +++ b/tests/code_coverage/sample_fuzz_test/otherCodeCoverage.fuzz/My_describe/My_fuzz_test/b @@ -0,0 +1 @@ +bbbbbbbbbbbbbbbbbbbbbbbbbbb \ No newline at end of file diff --git a/tests/code_coverage/sample_fuzz_test/package.json b/tests/code_coverage/sample_fuzz_test/package.json index 5d2ffa3aa..774878fd0 100644 --- a/tests/code_coverage/sample_fuzz_test/package.json +++ b/tests/code_coverage/sample_fuzz_test/package.json @@ -1,28 +1,27 @@ { - "name": "Jazzer.js code coverage tests", + "name": "jazzer.js-code-coverage-tests", "version": "1.0.0", - "description": "", "scripts": { - "test": "jest" + "test:coverage:js": "jest --coverage --testMatch \"**/*.fuzz.js\"", + "test:coverage:ts": "jest --coverage --testMatch \"**/*.fuzz.ts\"" }, "devDependencies": { "@jazzer.js/core": "file:../../../packages/core", - "@jazzer.js/jest-runner": "file:../../../packages/jest-runner" + "@jazzer.js/jest-runner": "file:../../../packages/jest-runner", + "jest": "^29.4.1", + "ts-jest": "^29.0.5", + "typescript": "^4.9.5" }, "jest": { "projects": [ { - "displayName": "test" - }, - { - "runner": "@jazzer.js/jest-runner", "displayName": { "name": "Jazzer.js", "color": "cyan" }, - "testMatch": [ - "/**/*.fuzz.js" - ] + "runner": "@jazzer.js/jest-runner", + "preset": "ts-jest", + "testEnvironment": "node" } ] }