Skip to content

Latest commit

 

History

History
276 lines (201 loc) · 12.1 KB

commands.md

File metadata and controls

276 lines (201 loc) · 12.1 KB

solana-test-suite

Validate effects from clients using Protobuf messages.

Usage:

$ solana-test-suite [OPTIONS] COMMAND [ARGS]...

Options:

  • --install-completion: Install completion for the current shell.
  • --show-completion: Show completion for the current shell, to copy it or customize the installation.
  • --help: Show this message and exit.

Commands:

  • create-env: Set up environment for debugging a...
  • create-fixtures: Create test fixtures from a directory of...
  • debug-mismatches: Run tests on a set of targets with a list...
  • decode-protobufs: Convert Context and/or Fixture messages to...
  • exec-fixtures: Execute fixtures and check for correct...
  • execute: Execute Context or Fixture message(s) and...
  • fix-to-ctx: Extract Context messages from Fixtures.
  • list-harness-types: List harness types available for use.
  • mass-regenerate-fixtures: Regenerate features for fixtures in...
  • regenerate-fixtures: Regenerate features in fixture messages.
  • run-tests: Run tests on a set of targets with a...

solana-test-suite create-env

Set up environment for debugging a mismatch from FuzzCorp

Usage:

$ solana-test-suite create-env [OPTIONS]

Options:

  • -s, --solana-target PATH: Solana (or ground truth) shared object (.so) target file path [default: impl/lib/libsolfuzz_agave_v2.0.so]
  • -h, --default-harness-type TEXT: Harness type to use for Context protobufs [default: InstrHarness]
  • -t, --target PATH: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have required function entrypoints defined [default: impl/lib/libsolfuzz_firedancer.so]
  • -o, --output-dir PATH: Output directory for messages [default: debug_mismatch]
  • -u, --repro-urls TEXT: Comma-delimited list of FuzzCorp mismatch links
  • -n, --section-names TEXT: Comma-delimited list of FuzzCorp section names
  • -f, --fuzzcorp-url TEXT: Comma-delimited list of FuzzCorp section names [default: https://api.dev.fuzzcorp.asymmetric.re/uglyweb/firedancer-io/solfuzz/bugs/]
  • -l, --log-level INTEGER: FD logging level [default: 5]
  • -r, --randomize-output-buffer: Randomizes bytes in output buffer before shared library execution
  • -p, --num-processes INTEGER: Number of processes to use [default: 4]
  • -l, --section-limit INTEGER: Limit number of fixture per section [default: 0]
  • -fd, --firedancer-repo PATH: Path to firedancer repository
  • -tv, --test-vectors-repo PATH: Path to test-vectors repository
  • --help: Show this message and exit.

solana-test-suite create-fixtures

Create test fixtures from a directory of Context and/or Fixture messages. Effects are generated by the target passed in with --solana-target or -s. You can also pass in additional targets with --target or -t and use --keep-passing or -k to only generate effects for test cases that match.

Usage:

$ solana-test-suite create-fixtures [OPTIONS]

Options:

  • -i, --input PATH: Input protobuf file or directory of protobuf files [default: corpus8]
  • -h, --default-harness-type TEXT: Harness type to use for Context protobufs [default: InstrHarness]
  • -s, --solana-target PATH: Solana (or ground truth) shared object (.so) target file path [default: impl/lib/libsolfuzz_agave_v2.0.so]
  • -t, --target PATH: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have required function entrypoints defined
  • -o, --output-dir PATH: Output directory for fixtures [default: test_fixtures]
  • -p, --num-processes INTEGER: Number of processes to use [default: 4]
  • -r, --readable: Output fixtures in human-readable format
  • -k, --keep-passing: Only keep passing test cases
  • -g, --group-by-program: Group fixture output by program type
  • -l, --log-level INTEGER: FD logging level [default: 5]
  • --help: Show this message and exit.

solana-test-suite debug-mismatches

Run tests on a set of targets with a list of FuzzCorp mismatch links.

Note: each .so target filename must be unique.

Usage:

$ solana-test-suite debug-mismatches [OPTIONS]

Options:

  • -s, --solana-target PATH: Solana (or ground truth) shared object (.so) target file path [default: impl/lib/libsolfuzz_agave_v2.0.so]
  • -h, --default-harness-type TEXT: Harness type to use for Context protobufs [default: InstrHarness]
  • -t, --target PATH: Shared object (.so) target file paths (pairs with --keep-passing). Targets must have required function entrypoints defined [default: impl/lib/libsolfuzz_firedancer.so]
  • -o, --output-dir PATH: Output directory for messages [default: debug_mismatch]
  • -u, --repro-urls TEXT: Comma-delimited list of FuzzCorp mismatch links
  • -n, --section-names TEXT: Comma-delimited list of FuzzCorp section names
  • -f, --fuzzcorp-url TEXT: Comma-delimited list of FuzzCorp section names [default: https://api.dev.fuzzcorp.asymmetric.re/uglyweb/firedancer-io/solfuzz/bugs/]
  • -l, --log-level INTEGER: FD logging level [default: 5]
  • -r, --randomize-output-buffer: Randomizes bytes in output buffer before shared library execution
  • -p, --num-processes INTEGER: Number of processes to use [default: 4]
  • -l, --section-limit INTEGER: Limit number of fixture per section [default: 0]
  • --help: Show this message and exit.

solana-test-suite decode-protobufs

Convert Context and/or Fixture messages to human-readable format.

Usage:

$ solana-test-suite decode-protobufs [OPTIONS]

Options:

  • -i, --input PATH: Input protobuf file or directory of protobuf files [default: raw_context]
  • -o, --output-dir PATH: Output directory for base58-encoded, Context and/or Fixture human-readable messages [default: readable_context]
  • -p, --num-processes INTEGER: Number of processes to use [default: 4]
  • -h, --default-harness-type TEXT: Harness type to use for Context protobufs [default: InstrHarness]
  • --help: Show this message and exit.

solana-test-suite exec-fixtures

Execute fixtures and check for correct effects

Usage:

$ solana-test-suite exec-fixtures [OPTIONS]

Options:

  • -i, --input PATH: Input protobuf file or directory of protobuf files
  • -t, --target PATH: Shared object (.so) target file path to execute [default: impl/firedancer/build/native/clang/lib/libfd_exec_sol_compat.so]
  • -o, --output-dir PATH: Output directory for test results [default: test_results]
  • -r, --randomize-output-buffer: Randomizes bytes in output buffer before shared library execution
  • -l, --log-level INTEGER: FD logging level [default: 2]
  • -p, --num-processes INTEGER: Number of processes to use [default: 4]
  • --help: Show this message and exit.

solana-test-suite execute

Execute Context or Fixture message(s) and print the Effects.

Usage:

$ solana-test-suite execute [OPTIONS]

Options:

  • -i, --input PATH: Input protobuf file or directory of protobuf files
  • -h, --default-harness-type TEXT: Harness type to use for Context protobufs [default: InstrHarness]
  • -t, --target PATH: Shared object (.so) target file path to execute [default: impl/firedancer/build/native/clang/lib/libfd_exec_sol_compat.so]
  • -r, --randomize-output-buffer: Randomizes bytes in output buffer before shared library execution
  • -l, --log-level INTEGER: FD logging level [default: 2]
  • --help: Show this message and exit.

solana-test-suite fix-to-ctx

Extract Context messages from Fixtures.

Usage:

$ solana-test-suite fix-to-ctx [OPTIONS]

Options:

  • -i, --input PATH: Input Fixture file or directory of Fixture files [default: fixtures]
  • -o, --output-dir PATH: Output directory for messages [default: instr]
  • -p, --num-processes INTEGER: Number of processes to use [default: 4]
  • --help: Show this message and exit.

solana-test-suite list-harness-types

List harness types available for use.

Usage:

$ solana-test-suite list-harness-types [OPTIONS]

Options:

  • --help: Show this message and exit.

solana-test-suite mass-regenerate-fixtures

Regenerate features for fixtures in provided test-vectors folder.

Usage:

$ solana-test-suite mass-regenerate-fixtures [OPTIONS]

Options:

  • -i, --input PATH: Input test-vectors directory [default: corpus8]
  • -o, --output-dir PATH: Output directory for regenerated fixtures [default: /tmp/regenerated_fixtures]
  • -t, --target PATH: Shared object (.so) target file path to execute [default: impl/lib/libsolfuzz_agave_v2.0.so]
  • -s, --stubbed-target PATH: Stubbed shared object (.so) target file path to execute [default: impl/lib/libsolfuzz_firedancer.so]
  • -f, --add-feature TEXT: List of feature pubkeys to force add to the fixtures.
  • -r, --remove-feature TEXT: List of feature pubkeys to force remove from the fixtures.
  • -k, --rekey-feature TEXT: List of feature pubkeys to rekey in the fixtures, formatted 'old/new' (e.g. --rekey-feature old/new).
  • -m, --merge-with-latest: Merge with the latest cleaned-up and supported featureset pulled from the target.
  • -a, --regenerate-all: Regenerate all fixtures, regardless of feature set changes
  • --help: Show this message and exit.

solana-test-suite regenerate-fixtures

Regenerate features in fixture messages.

Usage:

$ solana-test-suite regenerate-fixtures [OPTIONS]

Options:

  • -i, --input PATH: Either a file or directory containing messages [default: corpus8]
  • -t, --target PATH: Shared object (.so) target file path to execute [default: impl/lib/libsolfuzz_agave_v2.0.so]
  • -o, --output-dir PATH: Output directory for regenerated fixtures [default: regenerated_fixtures]
  • -d, --dry-run: Only print the fixtures that would be regenerated
  • -f, --add-feature TEXT: List of feature pubkeys to force add to the fixtures.
  • -r, --remove-feature TEXT: List of feature pubkeys to force remove from the fixtures.
  • -k, --rekey-feature TEXT: List of feature pubkeys to rekey in the fixtures, formatted 'old/new' (e.g. --rekey-feature old/new).
  • -m, --merge-with-latest: Merge with the latest cleaned-up and supported featureset pulled from the target.
  • -a, --regenerate-all: Regenerate all fixtures, regardless of feature set changes
  • -l, --log-level INTEGER: FD logging level [default: 5]
  • --help: Show this message and exit.

solana-test-suite run-tests

Run tests on a set of targets with a directory of Context and/or Fixture messages.

Note: each .so target filename must be unique.

Usage:

$ solana-test-suite run-tests [OPTIONS]

Options:

  • -i, --input PATH: Input protobuf file or directory of protobuf files [default: corpus8]
  • -h, --default-harness-type TEXT: Harness type to use for Context protobufs [default: InstrHarness]
  • -s, --solana-target PATH: Solana (or ground truth) shared object (.so) target file path [default: impl/lib/libsolfuzz_agave_v2.0.so]
  • -t, --target PATH: Shared object (.so) target file paths [default: impl/lib/libsolfuzz_firedancer.so]
  • -o, --output-dir PATH: Output directory for test results [default: test_results]
  • -p, --num-processes INTEGER: Number of processes to use [default: 4]
  • -r, --randomize-output-buffer: Randomizes bytes in output buffer before shared library execution
  • -c, --chunk-size INTEGER: Number of test results per file [default: 10000]
  • -v, --verbose: Verbose output: log failed test cases
  • -c, --consensus-mode: Only fail on consensus failures. One such effect is to normalize error codes when comparing results. Note: Cannot be used with --core-bpf-mode.
  • -cb, --core-bpf-mode: Deliberately skip known mismatches between BPF programs and builtins, only failing on genuine mimatches. For example, builtin programs may throw errors on readonly account state violations sooner than BPF programs, compute unit usage will be different, etc. This feature is primarily used to test a BPF program against a builtin. Note: Cannot be used with --consensus-mode.
  • -f, --failures-only: Only log failed test cases
  • -sf, --save-failures: Saves failed test cases to results directory
  • -ss, --save-successes: Saves successful test cases to results directory
  • -l, --log-level INTEGER: FD logging level [default: 5]
  • --help: Show this message and exit.