OpenSSL #7
Comments
|
I can patch the |
|
Does goblint/cil#60 resolve this? |
|
After goblint/cil#60 the issue now is typedef _Atomic _Bool atomic_bool;
typedef _Atomic char atomic_char;
typedef _Atomic signed char atomic_schar;
typedef _Atomic unsigned char atomic_uchar;
typedef _Atomic short atomic_short;
typedef _Atomic unsigned short atomic_ushort;
typedef _Atomic int atomic_int; |
|
With goblint/cil#61 there are some warnings during the merging phase, but CFG generation succeeds and solving starts. |
|
|
Thanks for looking into the parsing errors. I'm guessing with all the patches that's with the commands I used in my first attempt, without any Also, what about the |
Yup, just with the first command.
Seems like it, I get no error there. |
Still somewhat on the small side :/ |
|
Something seems off there. Why would it take 30s to parse just 22000 lines? And why would it be almost all dead? |
|
The generated CIL file is |
|
The reason so many things are dead is because it is combining several definitions of |
|
Here there 6 different definitions of |
|
Maybe the default OpenSSL configuration doesn't make all the command line utilities? Because those are just tests, but I thought there should be the Also, does the line count roughly match up if checked with |
|
|
|
Might be an issue with With https://github.com/nickdiego/compiledb I get a compilation db that is a lot bigger, but also new failures |
|
After goblint/analyzer#530, it chokes on this wonderful snippet: # 171 "crypto/lhash/lhash.c" 3 4
__extension__ ({ __auto_type __atomic_store_ptr = (
# 171 "crypto/lhash/lhash.c"
((_Atomic int *)&lh->error)
# 171 "crypto/lhash/lhash.c" 3 4
); __typeof__ (*__atomic_store_ptr) __atomic_store_tmp = (
# 171 "crypto/lhash/lhash.c"
(0)
# 171 "crypto/lhash/lhash.c" 3 4
); __atomic_store (__atomic_store_ptr, &__atomic_store_tmp, (
# 171 "crypto/lhash/lhash.c"
memory_order_relaxed
# 171 "crypto/lhash/lhash.c" 3 4
)); })
# 171 "crypto/lhash/lhash.c"
;which is what GCC turns calls to https://en.cppreference.com/w/c/atomic/atomic_store The issue seems to be with the |
|
With goblint/cil#67 and the support for It in this case does not get to the Merging phase even but fails in Also, it takes several minutes to get to that state. |
|
It is 1300 files it wants to parse out of which all are unique. |
That's interesting. I might have to try with a newer bear version on Ubuntu 21.04.
Does Goblint's verbose output show it continuously parsing all the files as the memory usage is growing or maybe it gets stuck on one, where it does something weird that causes a memory leak? |
Yes, each individual file does not seem to take too long, and the speed at which it parses each one seems constant too. Memory usage just keeps increasing until it is then killed at some point. I'll try on the server that has more RAM than my machine. If it also runs out there, there probably is some bug (as opposed to just inefficient memory usage). |
|
On the server it succeeds with peak memory usage at around 20GB and the CIL phase taking a few minutes. |
|
Here's something that might be manageable: ./Configure no-asm
make build_generated
bear -- make -j12 apps/openssl
goblint -v .It only builds the The evals count kept going up, but none of the other numbers increased (called fluctuated up and down by a few). Not sure, maybe some termination problem? |
|
Is it normal that the build fails for this configuration? If yes, how do we know that all meaningful calls to the compiler happened before the build was terminated (and are hence in the database)? |
|
I don't think it failed for me. Maybe you need to |
|
Ok, I needed to reclone for some reason. With the smaller compilation database but without removing |
|
I might have to try with On another note, flipping the def_exc widen to join fixes that termination problem it seems: It has definitely reached a lot more variables now. So we seem to have a much bigger benchmark now, but it's way too big for anything interactive if we're so hopelessly bottlenecked by parsing it. |
Was that with goblint/analyzer#532? For me that leads to merging being horribly slow. |
|
No, it was without as well. I might try a grouping version of it at some point to see if there's any hope to keep the maximum memory usage lower without impacting performance notably. |
|
By the way, I let it running for the whole day on my laptop today and it got this far before I finally killed it: And it didn't seem completely stuck because vars kept increasing too. It might be smarter to try analyzing single test suites instead, because the |
|
Not sure if this is an issue with the database we obtain from It finds a total of @sim642 did you run your previous experiments with Goblint was a72afa46a5cb826f7a936420428fef844dcf971c (with |
int main(int argc, char *argv[])
{
FUNCTION f, *fp;
LHASH_OF(FUNCTION) *prog = NULL;
char *pname;
const char *fname;
ARGS arg;
int global_help = 0;
int ret = 0;
arg.argv = NULL;
arg.size = 0;
/* Set up some of the environment. */
bio_in = dup_bio_in(FORMAT_TEXT);
///!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
// Everything after this is dead(!)
///!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
bio_out = dup_bio_out(FORMAT_TEXT); // never called
bio_err = dup_bio_err(FORMAT_TEXT); // never called
#if defined(OPENSSL_SYS_VMS) && defined(__DECC)
argv = copy_argv(&argc, argv);
#elif defined(_WIN32)
/* Replace argv[] with UTF-8 encoded strings. */
win32_utf8argv(&argc, &argv);
#endif
#ifndef OPENSSL_NO_TRACE
setup_trace(getenv("OPENSSL_TRACE"));
#endif
if ((fname = "apps_startup", !apps_startup())
|| (fname = "prog_init", (prog = prog_init()) == NULL)) {
BIO_printf(bio_err,
"FATAL: Startup failure (dev note: %s()) for %s\n",
fname, argv[0]);
ERR_print_errors(bio_err); //dead
ret = 1; //dead
goto end;
}
pname = opt_progname(argv[0]); // dead
default_config_file = CONF_get1_default_config_file(); //dead
if (default_config_file == NULL) // dead
app_bail_out("%s: could not get default config file\n", pname); //dead
/* first check the program name */
f.name = pname; //dead
fp = lh_FUNCTION_retrieve(prog, &f); //dead
if (fp == NULL) { //dead
/* We assume we've been called as 'openssl ...' */
global_help = argc > 1
&& (strcmp(argv[1], "-help") == 0 || strcmp(argv[1], "--help") == 0
|| strcmp(argv[1], "-h") == 0 || strcmp(argv[1], "--h") == 0);
argc--; //dead
argv++; //dead
opt_appname(argc == 1 || global_help ? "help" : argv[0]); //dead
} else {
argv[0] = pname; // dead
}
/* If there's a command, run with that, otherwise "help". */
ret = argc == 0 || global_help
? do_cmd(prog, 1, help_argv)
: do_cmd(prog, argc, argv);
end:
OPENSSL_free(default_config_file); // dead
lh_FUNCTION_free(prog); // dead
OPENSSL_free(arg.argv); // dead
if (!app_RAND_write()) // dead
ret = EXIT_FAILURE; // dead
BIO_free(bio_in); // dead
BIO_free_all(bio_out); // dead
apps_shutdown(); // dead
BIO_free(bio_err); // dead
EXIT(ret); // dead
} |
|
Your compilation database has 1005 entries, mine had 1007 I think, so that's probably correct. 67 lines out of 156196 being live is highly suspicious. I guess there's an unknown function through which most of it becomes live. |
Absolutely, I am digging through it to find the issue. |
|
This seems to be the culprit. It should call its argument. |
|
With goblint/analyzer#547, more code is live However, this seems quite low, so I guess there is more issues. |
|
Actually, we are not reaching a fixpoint in this example: |
|
It is also very strange that |
|
Besides that there are ~50 instances of |
|
With the output of the diff after goblint/analyzer#548, it gets even more interesting: and what the vids also printed: |
|
For further experiments. Contains manual fix for a case where CIL's alpha renaming gets confused. |
|
All allocations happen via wrappers, leading to only one heap object with content I'll try if adding some custom wrappers helps: |
|
Did not really have any effect. I think we can probably give up on getting meaningful results here in near future, barring major improvements to several parts. |
|
What's with the fixpoint error and CIL's alpha renaming though? Should probably have issues/PRs for those even if we cannot get anything useful on OpenSSL. |
|
For the fixpoint issue I'm running creduce since this morning (down to The alpha renaming thing is probably not so critical, but I guess I can also open an issue for that. It is just non-trivial to reproduce. |
|
At least the part we get from |
|
Attempt 1
Error:
The problem is
_Float32here:In
engines/e_afalg.c:_GNU_SOURCEenables (through a number of intermediates) the following in the unpreprocessedstdlib.h:I think OpenSSL itself doesn't even use
_Float32or that function, but it just gets pulled in with_GNU_SOURCEamong tons of other GNU-specific things.Attempt 2
In
floatn-common.hthere is the following typedef:So in an extremely crude hack to bypass that and cause the typedef to be used in the standard headers, I tried the following to pretend to be GCC 6.0 for the purposes of preprocessing:
goblint -v . --set cppflags[+] '-D__GNUC__=6' --set cppflags[+] '-D__GNUC_MINOR__=0'Error:
The problem is the
_Noreturnhere:_Noreturnis a C11 feature that CIL doesn't support (goblint/cil#13).Attempt 3
In an even more desperate attempt, I tried to just preprocess that keyword away (it's just extra information that technically shouldn't matter):
goblint -v . --set cppflags[+] '-D__GNUC__=6' --set cppflags[+] '-D__GNUC_MINOR__=0' --set cppflags[+] '-D_Noreturn='Error:
The problematic code is
__float128:CIL has has some kind of support for it (goblint/cil#8), but it defines both
__float128and_Float128as builtin type names, so maybe that's screwing with the parsing of this.I think that aspect of CIL has been updated for GCC >=7.0, so my version hack above is probably causing this one to happen. The appropriate solution would be to handle the other GNU float types directly as well.
TODO
_Noreturnsupport to CIL (Support for C11_Noreturncil#58)._Atomicsupport to CIL (Support for_Atomiccil#61).The text was updated successfully, but these errors were encountered: