[musl] Question regarding dynamic loader

[Gernot Reisinger]

Hi,
I recently stumbled upon an issue with preloading a shared object into a Go
application (see related Go ticket https://github.com/golang/go/issues/28909
).

In short - Go comes with an internal linker which will not link crt code to
the application. The entry point will directly execute Go standard library
code. As musl libc calls shared object constructors in crt code, the shared
objects constructors subsequently will never be invoked. Things will work
on glibc systems / processes. it It seems to be a subtle - but in this case
wide reaching - behavioral difference to glibc.

I wonder if calling constructor functions from crt code is an intended musl
libc behavior. My personal - non expert - gut feeling considers glibc
behavior "more correct". Is there a chance that musl will change this
behavior?
br
Gernot

[Rich Felker]

Post by Gernot Reisinger
Hi,
I recently stumbled upon an issue with preloading a shared object into a Go
application (see related Go ticket https://github.com/golang/go/issues/28909
).
In short - Go comes with an internal linker which will not link crt code to
the application. The entry point will directly execute Go standard library
code. As musl libc calls shared object constructors in crt code, the shared

I don't think this assessment of what musl does is correct. It calls
the (initially loaded) shared object constructor via
__libc_start_main. If the program is not entered via
__libc_start_main, libc is not usable. Necessary initialization will
have been bypassed. This has little to do with whether the crt code
was linked, except that *crt1.o is normally responsible for calling
__libc_start_main. If the linking process bypasses crt1, it needs to
ensure that __libc_start_main ends up getting called in some other
way. As far as I know this is also true for glibc, so I'm not sure why
it differs.

Post by Gernot Reisinger
objects constructors subsequently will never be invoked. Things will work
on glibc systems / processes. it It seems to be a subtle - but in this case
wide reaching - behavioral difference to glibc.
I wonder if calling constructor functions from crt code is an intended musl
libc behavior. My personal - non expert - gut feeling considers glibc
behavior "more correct". Is there a chance that musl will change this
behavior?

The musl behavior here is intentional. For FDPIC targets, it's
impossible to run *any* application code, in the main application or
shared libraries, before the main application's crt1 has executed,
because there are (essentially -- the equivalent of) self-relocations
performed at that stage that the dynamic linker can't see. If any
ctors were invoked directly by the dynamic linker before passing
control the the main application's entry point, they would run without
these relocations in the main application having been performed,
possibly resulting in runaway-wrong execution.

I believe Go is doing some bad hacks here with regard to its C FFI,
but it's likely fixable in some reasonable way. We should get more
eyes looking at it.

Rich

[Gernot Reisinger]

Thanks for your swift and extensive reply. Your explanations make a lot of
sense. (sorry for my sloppy description - __libc_start_main invoked by *crt
is the place where constructor calls happen as you did outline).
I did no extensive research how glibc executes these constructor calls. At
least the call stack indicates that they are partially executed in dynamic
linker context - _dl_start_user () in /lib64/ld-linux-x86-64.so
calling _dl_init.

I will add a reference to reply to the Go ticket.

Post by Gernot Reisinger

Post by Gernot Reisinger
Hi,
I recently stumbled upon an issue with preloading a shared object into a

Go

Post by Gernot Reisinger
application (see related Go ticket

https://github.com/golang/go/issues/28909

Post by Gernot Reisinger
).
In short - Go comes with an internal linker which will not link crt code

to

Post by Gernot Reisinger
the application. The entry point will directly execute Go standard

library

Post by Gernot Reisinger
code. As musl libc calls shared object constructors in crt code, the

shared
I don't think this assessment of what musl does is correct. It calls
the (initially loaded) shared object constructor via
__libc_start_main. If the program is not entered via
__libc_start_main, libc is not usable. Necessary initialization will
have been bypassed. This has little to do with whether the crt code
was linked, except that *crt1.o is normally responsible for calling
__libc_start_main. If the linking process bypasses crt1, it needs to
ensure that __libc_start_main ends up getting called in some other
way. As far as I know this is also true for glibc, so I'm not sure why
it differs.

Post by Gernot Reisinger
objects constructors subsequently will never be invoked. Things will work
on glibc systems / processes. it It seems to be a subtle - but in this

case

Post by Gernot Reisinger
wide reaching - behavioral difference to glibc.
I wonder if calling constructor functions from crt code is an intended

musl

Post by Gernot Reisinger
libc behavior. My personal - non expert - gut feeling considers glibc
behavior "more correct". Is there a chance that musl will change this
behavior?

The musl behavior here is intentional. For FDPIC targets, it's
impossible to run *any* application code, in the main application or
shared libraries, before the main application's crt1 has executed,
because there are (essentially -- the equivalent of) self-relocations
performed at that stage that the dynamic linker can't see. If any
ctors were invoked directly by the dynamic linker before passing
control the the main application's entry point, they would run without
these relocations in the main application having been performed,
possibly resulting in runaway-wrong execution.
I believe Go is doing some bad hacks here with regard to its C FFI,
but it's likely fixable in some reasonable way. We should get more
eyes looking at it.
Rich

[Szabolcs Nagy]

Post by Gernot Reisinger
I did no extensive research how glibc executes these constructor calls. At
least the call stack indicates that they are partially executed in dynamic
linker context - _dl_start_user () in /lib64/ld-linux-x86-64.so
calling _dl_init.

the dynamic linker runs the
- preinit_array functions of the main executable,
- the init_array and DT_INIT functions of shared libraries.

then via __libc_start_main the _init and init_array functions
of the main executable are run by libc_nonshared.a code that
is linked into the executable.

so part of the initialization (main exe) does require entry
via __libc_start_main (but this is not an issue for go).

however this design can change when glibc introduces a new
symbol version for __libc_start_main, so i don't see how
go can rely on any of this.

[Gernot Reisinger]

Thanks a lot for this exhaustive explanation - helps a lot to understand
the different initialization stages. I agree, one should not assume a
specific execution sequence of these initialization routines.

Post by Rich Felker

Post by Gernot Reisinger
I did no extensive research how glibc executes these constructor calls.

At

Post by Gernot Reisinger
least the call stack indicates that they are partially executed in

dynamic

Post by Gernot Reisinger
linker context - _dl_start_user () in /lib64/ld-linux-x86-64.so
calling _dl_init.

the dynamic linker runs the
- preinit_array functions of the main executable,
- the init_array and DT_INIT functions of shared libraries.
then via __libc_start_main the _init and init_array functions
of the main executable are run by libc_nonshared.a code that
is linked into the executable.
so part of the initialization (main exe) does require entry
via __libc_start_main (but this is not an issue for go).
however this design can change when glibc introduces a new
symbol version for __libc_start_main, so i don't see how
go can rely on any of this.

[Rich Felker]

Post by Rich Felker

Post by Gernot Reisinger
I wonder if calling constructor functions from crt code is an intended musl
libc behavior. My personal - non expert - gut feeling considers glibc
behavior "more correct". Is there a chance that musl will change this
behavior?

The musl behavior here is intentional. For FDPIC targets, it's
impossible to run *any* application code, in the main application or
shared libraries, before the main application's crt1 has executed,
because there are (essentially -- the equivalent of) self-relocations
performed at that stage that the dynamic linker can't see. If any
ctors were invoked directly by the dynamic linker before passing
control the the main application's entry point, they would run without
these relocations in the main application having been performed,
possibly resulting in runaway-wrong execution.

For reference, this was initially done in commit
c87a52103399135d2f57a91a8bcc749d8cb2ca83. Of course these code paths
have changed significantly since then, but it gives some historical
context.

Rich

[Szabolcs Nagy]

Post by Gernot Reisinger
Hi,
I recently stumbled upon an issue with preloading a shared object into a Go
application (see related Go ticket https://github.com/golang/go/issues/28909
).
In short - Go comes with an internal linker which will not link crt code to
the application. The entry point will directly execute Go standard library

then calling into the c runtime later is undefined.

crt is required for the c runtime setup.

Post by Gernot Reisinger
code. As musl libc calls shared object constructors in crt code, the shared

this is not true, the crt code is a tiny stub that
calls the __libc_start_main setup code in libc.so,
where ctors are run (there are several mechanisms
to do ctors, but that's an elf thing: musl supports
both _init and initarry style initializers, former
is passed as argument to __libc_start_main the latter
requires begin/end symbols for the .initarray section,
in glibc is similar, but part of the initialization
happens in the dynamic linker and part of it in
libc_nonshared.a code which should be linked to the
application and not in libc.so. static linking is
another story but i assume you are using dynamic linking).

Post by Gernot Reisinger
objects constructors subsequently will never be invoked. Things will work
on glibc systems / processes. it It seems to be a subtle - but in this case
wide reaching - behavioral difference to glibc.

this is libc internal implementation detail that
callers should not try to guess or rely on.
(however it has to be abi stable within one libc
implementation because old crt1.o linked into
an executable must work with new libc.so, otoh
in glibc the abi can changed over time using
symbol versioning for backward compatibility,
and there were talks about doing exactly that
because the way it runs ctor code is a perfect
gadget for rop attacks and present in every
executable)

Post by Gernot Reisinger
I wonder if calling constructor functions from crt code is an intended musl
libc behavior. My personal - non expert - gut feeling considers glibc
behavior "more correct". Is there a chance that musl will change this

what made you think musl calls ctors from crt code?

Post by Gernot Reisinger
behavior?
br
Gernot

[Florian Weimer]

Post by Gernot Reisinger
I recently stumbled upon an issue with preloading a shared object into
a Go application (see related Go ticket
https://github.com/golang/go/issues/28909).

The bug here is that Go does not call __libc_start_main at all and does
not link crt1.o. That is a major toolchain bug. It's surprising that
this works at all.

Thanks,
Florian