ld.so is at a different path than on Linux

[pwmarcz]

(As discussed in #2625).

• Graphene mounts the dynamic linker under /lib/ld-linux-x86-64.so.2, together with other glibc libraries. However, Linux executables (for x86-64 and glibc) have the ELF interpreter path set to /lib64/ld-linux-x86-64.so.2.

• Because of that, Graphene does not use the ELF interpreter path directly (as Linux does) but tries paths LD_LIBRARY_PATH (falling back to /lib:/lib64 if LD_LIBRARY_PATH is not set).

• Unfortunately we cannot just mount the glibc libraries at /lib64, because that's where we mount system libraries (arch_libdir) under systems such as Fedora. (Under Debian/Ubuntu, the system libraries are at /lib/x86_64-linux-gnu/).

Is this a problem?

I think this behavior is not very harmful. Mostly, the interpreter search is needlessly complicated, and it's confusing why Graphene behaves differently than Linux. I'm not sure how (or if) to fix it.

Possible solutions

• Keep this behavior, document and explain it (the reason is we want to keep our libraries separate from system ones and /lib is a convenient path).
• Mount the runtime dir from at /lib, not /lib64... but what about Fedora?
• Mount the individual glibc files at /lib64 so that they override system ones. Seems brittle.
• Hardcode this interpreter path in LibOS (i.e. unconditionally replace /lib64/ld-linux-x86-64.so.2 with /lib/ld-linux-x86-64.so.2). After all, this is the only interpreter we currently support.
• ???

[dimakuv]

Doesn't seem like much of a problem to me. But ideally (and for other reasons) I would prefer to not use a /lib path for Graphene-patched libs but instead something like /graphene-lib. And force users to specify LD_LIBRARY_PATH in the manifest file. And hard-code /graphene-lib instead of /lib in Graphene's find-interpreter code.

One of the reasons for this is making it less magic for novice users: the trick of mounting Graphene libs under /lib via fs.mount is too smart for many.

[boryspoplawski]

1. We need to use the loader path from ELF headers. What if some app uses /app/my_ld.so? Also note that using our patched glibc on SGX PAL is purely runtime performance optimization.
2. Direct mounting of host libraries directories is just for tests. In production environments you need to provide all libraries. Case of GSC, where we do use "host" libraries and mount the paths 1 to 1 is not a problem, since all the paths are known before hand i.e. they are not generic (do not depend on distro etc)