Skip to content

Commit

Permalink
[test/recipes] Split test_fuzz into separate recipes
Browse files Browse the repository at this point in the history
When using `HARNESS_JOBS` to run the tests in parallel, no matter the
level of parallelism that can be used, the monolithic `test_fuzz` takes
a long time to run, conditioning the duration of the whole build.

This commit splits the single `test_fuzz` recipe into separate recipes
for each fuzzer.
The previous mechanism to select individual fuzz tests using the
`FUZZ_TESTS` environment variable is also dropped (and documentation
updated).

Reviewed-by: Richard Levitte <[email protected]>
(Merged from openssl#13307)
  • Loading branch information
romen committed Nov 12, 2020
1 parent 9ce8e0d commit a7da4d4
Show file tree
Hide file tree
Showing 16 changed files with 287 additions and 57 deletions.
6 changes: 3 additions & 3 deletions fuzz/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,15 +114,15 @@ To do all the tests of a specific fuzzer such as asn1 you can run

fuzz/asn1-test fuzz/corpora/asn1
or
make test TESTS=fuzz_test FUZZ_TESTS=asn1
make test TESTS=fuzz_test_asn1

To run several fuzz tests you can use for instance:

make test TESTS=test_fuzz FUZZ_TESTS="cmp cms"
make test TESTS='test_fuzz_cmp test_fuzz_cms'

To run all fuzz tests you can use:

make test TESTS=test_fuzz
make test TESTS='test_fuzz_*'

Random numbers
--------------
Expand Down
6 changes: 3 additions & 3 deletions test/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,11 +98,11 @@ it's VMS style wildcards)

Run all tests except for the fuzz tests:

$ make TESTS=-test_fuzz test
$ make TESTS='-test_fuzz*' test

or, if you want to be explicit:

$ make TESTS='alltests -test_fuzz' test
$ make TESTS='alltests -test_fuzz*' test

Run all tests that have a name starting with "test_ssl" but not those
starting with "test_ssl_":
Expand All @@ -123,7 +123,7 @@ Run all tests in test groups 80 to 99 except for tests in group 90:

To run specific fuzz tests you can use for instance:

$ make test TESTS=test_fuzz FUZZ_TESTS="cmp cms"
$ make test TESTS='test_fuzz_cmp test_fuzz_cms'

To stochastically verify that the algorithm that produces uniformly distributed
random numbers is operating correctly (with a false positive rate of 0.01%):
Expand Down
38 changes: 0 additions & 38 deletions test/recipes/99-test_fuzz.t

This file was deleted.

22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_asn1.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "asn1";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_asn1parse.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "asn1parse";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_bignum.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "bignum";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_bndiv.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "bndiv";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_client.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "client";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
25 changes: 25 additions & 0 deletions test/recipes/99-test_fuzz_cmp.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "cmp";
setup("test_fuzz_${fuzzer}");

plan skip_all => "This test requires $fuzzer support"
if disabled($fuzzer);

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
25 changes: 25 additions & 0 deletions test/recipes/99-test_fuzz_cms.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "cms";
setup("test_fuzz_${fuzzer}");

plan skip_all => "This test requires $fuzzer support"
if disabled($fuzzer);

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_conf.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "conf";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_crl.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "crl";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
25 changes: 25 additions & 0 deletions test/recipes/99-test_fuzz_ct.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "ct";
setup("test_fuzz_${fuzzer}");

plan skip_all => "This test requires $fuzzer support"
if disabled($fuzzer);

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_server.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "server";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
22 changes: 22 additions & 0 deletions test/recipes/99-test_fuzz_x509.t
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
#!/usr/bin/env perl
# Copyright 2016-2020 The OpenSSL Project Authors. All Rights Reserved.
#
# Licensed under the Apache License 2.0 (the "License"). You may not use
# this file except in compliance with the License. You can obtain a copy
# in the file LICENSE in the source distribution or at
# https://www.openssl.org/source/license.html

use strict;
use warnings;

use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test::Utils;

my $fuzzer = "x509";
setup("test_fuzz_${fuzzer}");

plan tests => 2; # one more due to below require_ok(...)

require_ok(srctop_file('test','recipes','fuzz.pl'));

fuzz_ok($fuzzer);
21 changes: 8 additions & 13 deletions test/recipes/fuzz.pl
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,17 @@
use warnings;

use OpenSSL::Glob;
use OpenSSL::Test qw/:DEFAULT srctop_file/;
use OpenSSL::Test qw/:DEFAULT srctop_dir/;

sub fuzz_tests {
my @fuzzers = @_;
sub fuzz_ok {
die "Only one argument accepted" if scalar @_ != 1;

foreach my $f (@fuzzers) {
subtest "Fuzzing $f" => sub {
my @dir = glob(srctop_file('fuzz', 'corpora', "$f"));
my $f = $_[0];
my $d = srctop_dir('fuzz', 'corpora', $f);

plan skip_all => "No directory fuzz/corpora/$f" unless @dir;
plan tests => scalar @dir; # likely 1

foreach (@dir) {
ok(run(fuzz(["$f-test", $_])));
}
}
SKIP: {
skip "No directory $d", 1 unless -d $d;
ok(run(fuzz(["$f-test", $d])), "Fuzzing $f");
}
}

Expand Down

0 comments on commit a7da4d4

Please sign in to comment.