static linking and dlopen

[Paul Schutte]

Hi,

I have a strong preference towards static linking these days because the
running program use so much less memory.

When I link a binary statically and that binary then use dlopen, would that
work 100% ?

What would open if the shared object that was dlopened want's to call
functions in other shared libraries ?
I understand that when using dynamic linking those libraries would just get
loaded, but I am not sure what would happen with static linking.

Regards
Paul

[Szabolcs Nagy]

Post by Paul Schutte
When I link a binary statically and that binary then use dlopen, would that
work 100% ?

not required to work

and musl does not implement it (always returns 0)

Post by Paul Schutte
What would open if the shared object that was dlopened want's to call
functions in other shared libraries ?

see the first paragraph in the description of dlopen:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html

[Rob Landley]

Post by Paul Schutte

Post by Paul Schutte
When I link a binary statically and that binary then use dlopen,

would that

Post by Paul Schutte
work 100% ?

not required to work
and musl does not implement it (always returns 0)

Post by Paul Schutte
What would open if the shared object that was dlopened want's to

call

Post by Paul Schutte
functions in other shared libraries ?

http://pubs.opengroup.org/onlinepubs/9699919799/functions/dlopen.html

Please add this to the FAQ, or whatever "how shared libraries work"
documentation you wind up doing. There were also earlier discussions of
the horrors of dlclose that should be noted somewhere.

(Unless this is a "belling the cat" situation where suggesting it makes
it my responsibility, but I don't actually know this stuff...)

Rob

[Rich Felker]

Post by Paul Schutte
Hi,
I have a strong preference towards static linking these days because the
running program use so much less memory.
When I link a binary statically and that binary then use dlopen, would that
work 100% ?

Presently, it does not work at all. At best, it loses all the
advantages of static linking.

Post by Paul Schutte
What would open if the shared object that was dlopened want's to call
functions in other shared libraries ?

Dependencies of any loaded library also get loaded.

Post by Paul Schutte
I understand that when using dynamic linking those libraries would just get
loaded, but I am not sure what would happen with static linking.

With static linking, they would have to be loaded too. This means a
static-linked program using dlopen would have to contain the entire
dynamic linker logic. What's worse, it would also have to contain at
least the entire libc, and if you were using static versions of any
other library in the main program, and a loaded module referenced a
dynamic version of the same library, you'd probably run into
unpredictable crashing when the versions do not match exactly.

The source of all these problems is basically the same as the benefit
of static linking: the fact that the linker resolves, statically at
link time, which object files are needed to satisfy the needs of the
program. With dlopen, however, there is no static answer; *any* object
is potentially-needed, not directly by the main program, but possibly
by loaded modules. Consider what happens now if you only link part of
libc into the main program statically: additional modules loaded at
runtime won't necessarily have all the stuff they need, so dlopen
would also have to load libc.so. But now you're potentially using two
different versions of libc in the same program; if
implementation-internal data structures like FILE or the pthread
structure are not identical between the 2 versions, you'll hit an ABI
incompatibility, despite the fact that these data structures were
intended to be implementation-internal and never affect ABI. Even
without that issue, you have issues like potentially 2 copies of
malloc trying to manage the heap without being aware of one another,
and thus clobbering it.

For libc, the issues are all fixable by making sure that a static
version of dlopen depends on every single function in libc, so that
another copy never needs to get loaded. However, for other static
libraries pulled into the main program, there is really no fix without
help from the linker (it would have to pull in the entire library, and
somehow leave a note for dlopen to see that library is already loaded
and avoid loading it dynamically too).

Note that, even if we could get this working with a reasonable level
of robustness, almost all the advantages of static linking would be
gone. Static-linked programs using dlopen would be huge and ugly.

If you really want to make single-file binaries with no dependencies
and dlopen support, I think the solution is to first build them
dynamically linked, then merge the main program and all .so files into
a single ELF file. I don't know of any tools capable of doing this,
but in principle it's possible to write one. There are at least 2
different approaches to this. One is to process the ELF files and
merge their list of LOAD segments, symbol and relocation tables, etc.
all into a single ELF file, leaving the relocations in place for the
dynamic linker to perform at startup. This would require some
modification to the dynamic linker still. The other approach is the
equivalent of emacs' unexec dumper: place some kind of hook to run
after the dynamic linker loads everything, but before any other
application code runs, which dumps the entire memory space to an ELF
file which, when run, will reconstruct itself.

Rich

[Charlie Kester]

I wonder if most of what people want to do with dlopen couldn't be done
just as well (or better) with good old fork() and exec(), along with
some suitable interprocess communication.

[Rich Felker]

Post by Charlie Kester
I wonder if most of what people want to do with dlopen couldn't be
done just as well (or better) with good old fork() and exec(), along
with some suitable interprocess communication.

I think it depends a lot on what you're using dlopen for. A lot of
programs these days use it as a ridiculous part of the development
model rather than for any real purpose; in my book, this is among the
highest levels of Considered Harmful. There's really no reason for any
code that's internal to an application (developed in the same tree, by
the same authors, and built at the same time) to be dynamically linked
at all, much less dynamically loaded. All this accomplishes is making
the program a lot slower and more bloated.

On the flip side, the main legitimate uses for dynamic linking and
loading are (1) sharing code that's used by a wide range of
applications and allowing it to be upgraded system-wide all at once,
and (2) facilitating the extension of an application with third-party
code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
loading (dlopen).

As for your suggestion that dlopen could largely be replaced by
running an external program, that's definitely an alternate way to
accomplish extensions/plug-ins (see GIMP, for example). How well it
works probably depends on the performance requirements (it's hard to
get good performance if the plugin is dealing with high volumes of
data unless you develop complex IPC methods using shared memory, so
there's a complexity trade-off too) and whether your extensions need
to survive across fork (which does not duplicate a process's child
process trees; this could very well matter for extension modules used
by language interpreters/runtimes like Python, Perl, etc.).

Rich

[Paul Schutte]

Post by Rich Felker
I think it depends a lot on what you're using dlopen for. A lot of
programs these days use it as a ridiculous part of the development
model rather than for any real purpose; in my book, this is among the
highest levels of Considered Harmful. There's really no reason for any
code that's internal to an application (developed in the same tree, by
the same authors, and built at the same time) to be dynamically linked
at all, much less dynamically loaded. All this accomplishes is making
the program a lot slower and more bloated.
I can not agree with you more on this.
On the flip side, the main legitimate uses for dynamic linking and
loading are (1) sharing code that's used by a wide range of
applications and allowing it to be upgraded system-wide all at once,
and (2) facilitating the extension of an application with third-party
code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
loading (dlopen).

Point 1 is probably the reason why most libraries end up as dynamic
libraries.

I was wondering about distributing all libraries as static libraries and
then have the package manager link the application statically as the final
step of the installation. This way the package manager can keep track
of dependencies and re-link them if a library change.

Distributions like Gentoo who install from source is actually in a very
good position to take advantage of static linking.

But I can see a lot of compiling/linking happening with this approach.

Another idea would be to just install a stub where the binary would be.
First time you run this stub, it will link the binary and store it on the
disk in some sort of cache. Then just do an exec of that binary. Second
time that you run this stub, it will check in this cache, link it again if
it is not there or just exec it if found. This way only the stuff that gets
used will be re-linked. You can force a re-link by clearing the cache. This
what made me wonder about programs that use dlopen.

I also wonder if the gain would be worth the trouble. I have seen a
reduction of up to 50% RSS usage on programs that has a lot of shared
libraries. It should improve responsiveness as there will be less paging.

Thanks for all the answers.

Regards
Paul

[Rich Felker]

Post by Paul Schutte

Post by Rich Felker
On the flip side, the main legitimate uses for dynamic linking and
loading are (1) sharing code that's used by a wide range of
applications and allowing it to be upgraded system-wide all at once,
and (2) facilitating the extension of an application with third-party
code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
loading (dlopen).

Point 1 is probably the reason why most libraries end up as dynamic
libraries.
I was wondering about distributing all libraries as static libraries and
then have the package manager link the application statically as the final
step of the installation. This way the package manager can keep track
of dependencies and re-link them if a library change.

This is a very reasonable design. There is _some_ risk of breakage if
the static libraries depend on the application being built using the
exact same headers as the library, but most such dependencies would
also correspond to ABI breakage for the shared library, so I think the
risk is low. The main difficulty is getting applications' build
processes to stop before the final linking and give you output that
you can relink when needed.

Post by Paul Schutte
Distributions like Gentoo who install from source is actually in a very
good position to take advantage of static linking.
But I can see a lot of compiling/linking happening with this approach.
Another idea would be to just install a stub where the binary would be.
First time you run this stub, it will link the binary and store it on the
disk in some sort of cache. Then just do an exec of that binary. Second
time that you run this stub, it will check in this cache, link it again if
it is not there or just exec it if found. This way only the stuff that gets
used will be re-linked. You can force a re-link by clearing the cache. This

This approach is a bit more difficult, because you need to manage
things like who has privileges to update the binaries. Surely you can
do it with suid and/or a daemon, but it's not entirely trivial.

Post by Paul Schutte
what made me wonder about programs that use dlopen.

Actually, I know one more solution for the dlopen issue, but it
requires some application-level hackery. You just link all the modules
you'll need into the main binary with a table of strings identifying
them, and make a dummy dlopen/dlsym implementation that gives you
access to stuff already linked into the application. The level of
"evil hackery" is pretty comparable to most of the stuff gnulib
does...

Post by Paul Schutte
I also wonder if the gain would be worth the trouble. I have seen a
reduction of up to 50% RSS usage on programs that has a lot of shared
libraries. It should improve responsiveness as there will be less paging.

I think the solution that achieves the best balance between reducing
bloat/slowness/paging and not spending huge amounts of effort is to
abandon the requirement of static linking everything, and instead go
with shared libraries for things that are used by a huge portion of
applications. For shared library "stacks" that have a chain of 10+ .so
files each depending on the rest, you could replace them with a single
.so file containing the whole library stack, as long as none of them
pollute the namespace horribly. This would cut most of the cost of
dynamic linking right there. For libs that aren't used by many apps,
or that are written in C++ (which results in huge dynamic-linking
bloat), I'd just use static versions.

Rich

[Paul Schutte]

Hi,

Post by Rich Felker

Post by Paul Schutte

Post by Rich Felker
On the flip side, the main legitimate uses for dynamic linking and
loading are (1) sharing code that's used by a wide range of
applications and allowing it to be upgraded system-wide all at once,
and (2) facilitating the extension of an application with third-party
code. Usage 1 applies mostly to dynamic linking; 2 mostly to dynamic
loading (dlopen).

Point 1 is probably the reason why most libraries end up as dynamic
libraries.
I was wondering about distributing all libraries as static libraries and
then have the package manager link the application statically as the

final

Post by Paul Schutte
step of the installation. This way the package manager can keep track
of dependencies and re-link them if a library change.

This is a very reasonable design. There is _some_ risk of breakage if
the static libraries depend on the application being built using the
exact same headers as the library, but most such dependencies would
also correspond to ABI breakage for the shared library, so I think the
risk is low. The main difficulty is getting applications' build
processes to stop before the final linking and give you output that
you can relink when needed.

I was thinking that one can butcher libtool. Software that use libtool
should then in theory work.

Post by Rich Felker

Post by Paul Schutte
Distributions like Gentoo who install from source is actually in a very
good position to take advantage of static linking.
But I can see a lot of compiling/linking happening with this approach.
Another idea would be to just install a stub where the binary would be.
First time you run this stub, it will link the binary and store it on the
disk in some sort of cache. Then just do an exec of that binary. Second
time that you run this stub, it will check in this cache, link it again

if

Post by Paul Schutte
it is not there or just exec it if found. This way only the stuff that

gets

Post by Paul Schutte
used will be re-linked. You can force a re-link by clearing the cache.

This
This approach is a bit more difficult, because you need to manage
things like who has privileges to update the binaries. Surely you can
do it with suid and/or a daemon, but it's not entirely trivial.

You are right. That started me thinking along the lines of a "magic"
filesystem based on fuse.
The filesystem code can then do all these things "behind the scenes".

Post by Rich Felker

Post by Paul Schutte
what made me wonder about programs that use dlopen.

Actually, I know one more solution for the dlopen issue, but it
requires some application-level hackery. You just link all the modules
you'll need into the main binary with a table of strings identifying
them, and make a dummy dlopen/dlsym implementation that gives you
access to stuff already linked into the application. The level of
"evil hackery" is pretty comparable to most of the stuff gnulib
does...

Post by Paul Schutte
I also wonder if the gain would be worth the trouble. I have seen a
reduction of up to 50% RSS usage on programs that has a lot of shared
libraries. It should improve responsiveness as there will be less paging.

I think the solution that achieves the best balance between reducing
bloat/slowness/paging and not spending huge amounts of effort is to
abandon the requirement of static linking everything, and instead go
with shared libraries for things that are used by a huge portion of
applications. For shared library "stacks" that have a chain of 10+ .so
files each depending on the rest, you could replace them with a single
.so file containing the whole library stack, as long as none of them
pollute the namespace horribly. This would cut most of the cost of
dynamic linking right there. For libs that aren't used by many apps,
or that are written in C++ (which results in huge dynamic-linking
bloat), I'd just use static versions.

This makes sense.
Wonder if there is a spesific reason why the browser folks does'nt produce
a statically linked browser anymore.
The browsers would be good candidates for what you mentioned about C++ and
dynamic bloat.

Post by Rich Felker
Rich

Thanks. I understand it a lot better now.

Regards
Paul

[Rich Felker]

Post by Paul Schutte
This makes sense.
Wonder if there is a spesific reason why the browser folks does'nt produce
a statically linked browser anymore.
The browsers would be good candidates for what you mentioned about C++ and
dynamic bloat.

It's because dynamic linking is part of their _development_ model. My
understanding is that they lack proper functional makefiles that would
facilitate clean incremental compiling and linking, so they instead
break the project up into a number of separate library components, and
they can then rebuild just a single component to test (since it gets
dynamically loaded anyway) rather than having to rebuild the whole
program from scratch.

While I think this is a stupid development model, as long as they're
just doing it for development, it doesn't really harm end users that
much. The problem is that they don't have a "release" build mode that
just links everything together the right way. It's not clear to me
whether this would be easy to change; it's possible that, due to
always using dynamic linking internally, a number of dependencies on
dynamic-linking-related behavior (symbol interposition, accessing
things via dlsym, etc.) crept into the code, and would be painful to
exorcise (especially under the constraint of not breaking their
dynamic builds).

Rich

[Szabolcs Nagy]

Post by Rich Felker
just links everything together the right way. It's not clear to me
whether this would be easy to change; it's possible that, due to
always using dynamic linking internally, a number of dependencies on
dynamic-linking-related behavior (symbol interposition, accessing
things via dlsym, etc.) crept into the code, and would be painful to
exorcise (especially under the constraint of not breaking their
dynamic builds).

i guess it's hard to fix (legacy from netscape 2.0)

https://en.wikipedia.org/wiki/NPAPI
https://developer.mozilla.org/en-US/docs/Gecko_Plugin_API_Reference/Plug-in_Basics

this plugin system is a core part of dom
and uses the nspr (netscape portable runtime)
to do the dlopen/dlclose/dlsym

https://www.mozilla.org/projects/nspr/reference/html/prlink.html

and most likely there are other dynamic linking
dependencies, this is just an obvious one

[Rich Felker]

Post by Szabolcs Nagy

Post by Rich Felker
just links everything together the right way. It's not clear to me
whether this would be easy to change; it's possible that, due to
always using dynamic linking internally, a number of dependencies on
dynamic-linking-related behavior (symbol interposition, accessing
things via dlsym, etc.) crept into the code, and would be painful to
exorcise (especially under the constraint of not breaking their
dynamic builds).

i guess it's hard to fix (legacy from netscape 2.0)
https://en.wikipedia.org/wiki/NPAPI
https://developer.mozilla.org/en-US/docs/Gecko_Plugin_API_Reference/Plug-in_Basics
this plugin system is a core part of dom
and uses the nspr (netscape portable runtime)
to do the dlopen/dlclose/dlsym

Yes, but being that it's abstracted by nspr, it could be replaced with
a system to use internal 'plugins'. Anyway I was less concerned with
features that are based on dynamic loading (plugins) and more
interested in the problem that the whole core browser is made up of
tons of shared libraries that are always-loaded (whether by dlopen or
at startup-time, I'm not sure..)

Rich

[Szabolcs Nagy]

Post by Paul Schutte
I was wondering about distributing all libraries as static libraries and
then have the package manager link the application statically as the final
step of the installation. This way the package manager can keep track
of dependencies and re-link them if a library change.

note that binaries built by the user outside of the
package manager wont be relinked which may or may
not be what you want

static linking takes a lot of time and memory
(gold is better at this than gnu ld, but a libc
update would take some time either way)

you cannot easily collapse dependency chains:
eg. with x,y binaries, a,b,c objects and
x->a->b->c
y->b->c
dependency chains, the b->c linking is done twice
(so not much work can be shared between the linking
of various binaries and the linking of a huge binary
takes a lot of time even if an insignificant small
part changes)

other than dlopen the LD_PRELOAD and LD_LIBRARY_PATH
hacks would not work, nor system components that depend
on dynamic linking (vdso, proprietary libraries
(graphics drivers etc), nsswitch, pam)

i'm not saying your idea is bad, but it's non-trivial

Post by Paul Schutte
Another idea would be to just install a stub where the binary would be.
First time you run this stub, it will link the binary and store it on the
disk in some sort of cache. Then just do an exec of that binary. Second

i guess the stub would need setuid

[c***@openwall.com]

Hi folks,

Post by Charlie Kester
I wonder if most of what people want to do with dlopen couldn't be done
just as well (or better) with good old fork() and exec(), along with
some suitable interprocess communication.

The thing I recalled right off, is that, e.g., interpreters of Tcl are able
to load C-written functions to extend the functionality of the interpreter
(e.g. with a domain-specific set of functions), and this is done with
dlopen. I think Tcl is not the only interpreter doing so. I even used the
same technique in one of my works (it was an interpreter of (small subset
of) Lisp, extendable in the same way). Well, may be this (in theory) can
be done with fork/exec, with some protocol on the pipes, etc., but as for
me, I would never use such a solution, preferring to keep the interpreter
non-extendable: having library functions' implementations in an external
process conflicts with my sense of aesthetics :-)

IMHO, for some (monstrous) programs such as browsers, or Gimp, or whatever
alike, having all the dynamic linking logic inside is not a catastrophe,
because the catastrophe is their size as such. For interpreters, the
catastrophe is the fact of interpreted execution, and dlopen will add
next to nothing to it.

Anyway, for a lot of existing programs, the absense of dlopen will mean a
necessity for heavy rewritings in order to compile them with musl - perhaps
too heavy to be done at all.



P.S. thank you for the great work you do!

--
Cheers,
Croco

[Isaac Dunham]

On Sun, 9 Dec 2012 10:36:58 +0400

Post by c***@openwall.com
Anyway, for a lot of existing programs, the absense of dlopen will mean a
necessity for heavy rewritings in order to compile them with musl - perhaps
too heavy to be done at all.

I think there's a misunderstanding here:

musl includes a dlopen function.
When a binary is statically linked, it is a stub.
When a binary is dynamically linked, it loads the shared library requested.

Post by c***@openwall.com
P.S. thank you for the great work you do!
--
Cheers,
Croco

[Charlie Kester]

Post by Isaac Dunham
On Sun, 9 Dec 2012 10:36:58 +0400

Post by c***@openwall.com
Anyway, for a lot of existing programs, the absense of dlopen will mean a
necessity for heavy rewritings in order to compile them with musl - perhaps
too heavy to be done at all.

Yes, and I apologize if I contributed to that misunderstanding with my
musings on alternative designs.

Post by Isaac Dunham
musl includes a dlopen function.
When a binary is statically linked, it is a stub.
When a binary is dynamically linked, it loads the shared library requested.

[c***@openwall.com]

Post by Isaac Dunham
musl includes a dlopen function.
When a binary is statically linked, it is a stub.
When a binary is dynamically linked, it loads the shared library requested.

Oh, well, I didn't catch the situation; if this is the case, the things are
definitely better. However, to my mind, static linkage is good for
creating portable binaries (besides all the other advantages), and I can
easily imagine a situation in which I dislike the idea of a
dynamically-linked main binary (e.g. I ship some unusual software to
endusers, and they have different Linux distros but are not going to build
the soft from sources - yes, there are such Linux users who panic when they
hear the word "compiler" - and I've got no hope someone else will package
my soft for different distros, because it is too unusual, so either I spend
my time installing 20+ different versions of various distros and prepare
packages for them all, or I opt for -static). So, I'd like to have all the
libs inside the binary of, e.g., my interpreter (actually, this can be a
program which does its job being controlled by embedded interpreter). But,
at the same time, it is very possible I need these loadable modules, which
extend the functionality of the interpreter. Surely it is not a
catastrophe, as I can link all libs but musl statically, and provide
libmusl.so along with the binary, also having a script which sets
LD_LIBRARY_PATH and then runs the binary; but it is a bit, errr...
/strange/ :)

Actually, when it comes to -static, the linker only picks the modules that
contain unresolved symbols, so it should (am I right?) be easy to break the
things down to modules so that all the dynamic linkage mechanics is linked
into the binary only in case it calls dlopen. And, okay, to mention in the
man 3 dlopen that using it from within a statically-linked binary will
increase the size of the binary by another megabyte, and that the .so to be
loaded must itself use statically-linked version of libraries so that some
functions will be loaded to the code segment twice. Such practice should
be discouraged but I don't think it should be made impossible at all.

I realize, however, that it is possible I simply miss something.


--
Croco

[Szabolcs Nagy]

Post by c***@openwall.com
packages for them all, or I opt for -static). So, I'd like to have all the
libs inside the binary of, e.g., my interpreter (actually, this can be a
program which does its job being controlled by embedded interpreter). But,
at the same time, it is very possible I need these loadable modules, which

dalias just described why static linking with dlopen is not possible

(he also specifically mentioned interpreters using dlopen..)

Post by c***@openwall.com
loaded must itself use statically-linked version of libraries so that some
functions will be loaded to the code segment twice. Such practice should
be discouraged but I don't think it should be made impossible at all.

it is impossible

two version of the same function cannot go together:

they may work correctly only if some global state exists
only once or modified from a single place
(malloc with brk pointer and exit with file stream list
were such examples)

so you can use dlopen only if you can ensure you don't use
the same libraries as the dlopened code or all shared code
is pure (no writeable global state, no sideeffects), but if
you use dlopen you already depend on libc and if the dlopened
code also uses libc you have a problem

(different library version was an additional issue)

so you can have dlopen in your static interpreter if
1) you know all the dlopened code and verified that they don't
depend on libc or anything impure on which the interpreter depends
(but in this case you should link those statically)
2) include all the dependent libraries entirely into the
statically linked binary and have dlopen use the symbols from
these (for which you may need to modify the linker and make sure
the binary is not stripped or implement dlopen so it somehow has
a list of all the extern symbols)
(but this does not seem practical and still does not solve abi
breaking library versioning issues)

[Rich Felker]

Post by Szabolcs Nagy

Post by c***@openwall.com
packages for them all, or I opt for -static). So, I'd like to have all the
libs inside the binary of, e.g., my interpreter (actually, this can be a
program which does its job being controlled by embedded interpreter). But,
at the same time, it is very possible I need these loadable modules, which

dalias just described why static linking with dlopen is not possible

To clarify, it's not possible right now, and difficult or impossible
to do "right". There are various partly-working approaches though.

Post by Szabolcs Nagy
so you can use dlopen only if you can ensure you don't use
the same libraries as the dlopened code or all shared code
is pure (no writeable global state, no sideeffects), but if
you use dlopen you already depend on libc and if the dlopened
code also uses libc you have a problem

The dynamic linker/libc already refuses to load itself, so in musl
there's not really the danger of libc being loaded twice. The issue is
that we would have to ensure that the whole libc gets linked into the
main program so it's available to loaded modules -- and that the
symbols are all kept so that they can be used. Both of these, while
possible, are not entirely trivial to do.

I'm not against consideration of support for this, but I would like to
first explore the alternative design I suggested: initially linking as
a dynamic-linked program, then using a special utility to combine all
of the shared libraries into a single file. While this approach does
have some disadvantages (PIC code, startup time cost, etc. much like
dynamic linking), it has some of the advantages of static linking
(not searching/loading multiple files all over the filesystem,
single-file distribution, etc.) and an additional benefit is that,
with support from the dynamic linker, even the .so files intended for
loading with dlopen could be packed into the main file.

Rich

[Rob Landley]

Post by Isaac Dunham

Post by Isaac Dunham
musl includes a dlopen function.
When a binary is statically linked, it is a stub.
When a binary is dynamically linked, it loads the shared library

requested.
Oh, well, I didn't catch the situation; if this is the case, the things are
definitely better. However, to my mind, static linkage is good for
creating portable binaries (besides all the other advantages),

And musl has good support for static linking. Musl has good support for
dynamic linking. Mixing the two is a bad idea at the design level,
which posix does not guarantee to work.

Post by Isaac Dunham
and I can
easily imagine a situation in which I dislike the idea of a
dynamically-linked main binary (e.g. I ship some unusual software to
endusers, and they have different Linux distros but are not going to build
the soft from sources - yes, there are such Linux users who panic when they
hear the word "compiler" - and I've got no hope someone else will package
my soft for different distros, because it is too unusual, so either I spend
my time installing 20+ different versions of various distros and prepare
packages for them all, or I opt for -static).

Yay static linking.

Post by Isaac Dunham
So, I'd like to have all the
libs inside the binary of, e.g., my interpreter (actually, this can be a
program which does its job being controlled by embedded
interpreter). But,
at the same time, it is very possible I need these loadable modules, which
extend the functionality of the interpreter.

If you need dynamic linking, then you need dynamic linking.

Post by Isaac Dunham
Surely it is not a
catastrophe, as I can link all libs but musl statically, and provide
libmusl.so along with the binary, also having a script which sets
LD_LIBRARY_PATH and then runs the binary; but it is a bit, errr...
/strange/ :)

You want a statically, dynamically linked program. Presumably combining
PIC and non-PIC code.

Good luck.

Post by Isaac Dunham
Actually, when it comes to -static, the linker only picks the modules that
contain unresolved symbols, so it should (am I right?) be easy to break the
things down to modules so that all the dynamic linkage mechanics is linked
into the binary only in case it calls dlopen. And, okay, to mention in the
man 3 dlopen that using it from within a statically-linked binary will
increase the size of the binary by another megabyte, and that the .so to be
loaded must itself use statically-linked version of libraries so that some
functions will be loaded to the code segment twice. Such practice should
be discouraged but I don't think it should be made impossible at all.
I realize, however, that it is possible I simply miss something.

You're confusing "possible" with "a good idea". Many truly horrible
ideas aren't actually impossible, as Windows extensively demonstrates.

Possibly what you want is large chunks of the musl dynamic loader
factored out so you can easily suck them into your program and keep the
pieces when it breaks.

Rob

[Paul Schutte]

Thanks for the comprehensive answer, this is the answer I was looking for.

There is currently not a situation that require me to do this. I was just
thinking about it and decided to ask the experts.

Post by Rich Felker

Post by Paul Schutte
Hi,
I have a strong preference towards static linking these days because the
running program use so much less memory.
When I link a binary statically and that binary then use dlopen, would

that

Post by Paul Schutte
work 100% ?

Presently, it does not work at all. At best, it loses all the
advantages of static linking.

Post by Paul Schutte
What would open if the shared object that was dlopened want's to call
functions in other shared libraries ?

Dependencies of any loaded library also get loaded.

Post by Paul Schutte
I understand that when using dynamic linking those libraries would just

get

Post by Paul Schutte
loaded, but I am not sure what would happen with static linking.

With static linking, they would have to be loaded too. This means a
static-linked program using dlopen would have to contain the entire
dynamic linker logic. What's worse, it would also have to contain at
least the entire libc, and if you were using static versions of any
other library in the main program, and a loaded module referenced a
dynamic version of the same library, you'd probably run into
unpredictable crashing when the versions do not match exactly.
The source of all these problems is basically the same as the benefit
of static linking: the fact that the linker resolves, statically at
link time, which object files are needed to satisfy the needs of the
program. With dlopen, however, there is no static answer; *any* object
is potentially-needed, not directly by the main program, but possibly
by loaded modules. Consider what happens now if you only link part of
libc into the main program statically: additional modules loaded at
runtime won't necessarily have all the stuff they need, so dlopen
would also have to load libc.so. But now you're potentially using two
different versions of libc in the same program; if
implementation-internal data structures like FILE or the pthread
structure are not identical between the 2 versions, you'll hit an ABI
incompatibility, despite the fact that these data structures were
intended to be implementation-internal and never affect ABI. Even
without that issue, you have issues like potentially 2 copies of
malloc trying to manage the heap without being aware of one another,
and thus clobbering it.
For libc, the issues are all fixable by making sure that a static
version of dlopen depends on every single function in libc, so that
another copy never needs to get loaded. However, for other static
libraries pulled into the main program, there is really no fix without
help from the linker (it would have to pull in the entire library, and
somehow leave a note for dlopen to see that library is already loaded
and avoid loading it dynamically too).
Note that, even if we could get this working with a reasonable level
of robustness, almost all the advantages of static linking would be
gone. Static-linked programs using dlopen would be huge and ugly.
If you really want to make single-file binaries with no dependencies
and dlopen support, I think the solution is to first build them
dynamically linked, then merge the main program and all .so files into
a single ELF file. I don't know of any tools capable of doing this,
but in principle it's possible to write one. There are at least 2
different approaches to this. One is to process the ELF files and
merge their list of LOAD segments, symbol and relocation tables, etc.
all into a single ELF file, leaving the relocations in place for the
dynamic linker to perform at startup. This would require some
modification to the dynamic linker still. The other approach is the
equivalent of emacs' unexec dumper: place some kind of hook to run
after the dynamic linker loads everything, but before any other
application code runs, which dumps the entire memory space to an ELF
file which, when run, will reconstruct itself.
Rich

[Szabolcs Nagy]

This post might be inappropriate. Click to display it.

[Rich Felker]

Post by Szabolcs Nagy
i think one could make a dso that has no dependency
(all dependencies are linked into it including libc),
and no pointers are passed (directly or indirectly)
to types which are not entirely part of the abi
between the dso and the main binary
(ie no FILE* or anything that might have different
internals on the two sides)

It's not that simple. There's always at least one piece of state
that's shared: the brk. 2 versions of malloc running will always cause
horrible problems unless they're aware of each other or avoid using
brk at all.

There's also the issue of stdio flushing at exit. The main program's
exit() can't flush the stdio streams belonging to the dso.

Thread-local storage in the dso would also be another problematic
area..

Rich

[Isaac Dunham]

On Sun, 9 Dec 2012 03:55:30 +0100

Post by Szabolcs Nagy
i think one could make a dso that has no dependency
(all dependencies are linked into it including libc),
and no pointers are passed (directly or indirectly)
to types which are not entirely part of the abi
between the dso and the main binary
(ie no FILE* or anything that might have different
internals on the two sides)

Under at least glibc, this is supported.
This is what gcc/ld does with "-static -shared".

Post by Szabolcs Nagy
in that case dlopen() would work and even dlclose()
can be safe (as the programmer has control over the
entire code)
of course care should be taken and it might not be
very useful but one could implement loadable/unloadable
plugins with static linking

[Rob Landley]

Post by Rich Felker
The source of all these problems is basically the same as the benefit
of static linking: the fact that the linker resolves, statically at
link time, which object files are needed to satisfy the needs of the
program. With dlopen, however, there is no static answer; *any* object
is potentially-needed, not directly by the main program, but possibly
by loaded modules.

Is there some way you could put a dlopen() symbol in libc.a that forces
a build break? Perhaps a redirect to
dlopen_is_not_compatible_with_static_linking_go_read_posix?

Rob

[Rich Felker]

Post by Rob Landley

Post by Rich Felker
The source of all these problems is basically the same as the benefit
of static linking: the fact that the linker resolves, statically at
link time, which object files are needed to satisfy the needs of the
program. With dlopen, however, there is no static answer; *any* object
is potentially-needed, not directly by the main program, but possibly
by loaded modules.

Is there some way you could put a dlopen() symbol in libc.a that
forces a build break? Perhaps a redirect to
dlopen_is_not_compatible_with_static_linking_go_read_posix?

The dlopen in libc.a simply always-fails. For programs where loading
modules is optional, this is probably the ideal behavior.

Rich