Discussion:
Best place to discuss other lightweight libraries?
(too old to reply)
LM
2013-04-21 16:30:54 UTC
Permalink
Musl does a great job of replacing glibc. It's lightweight,
well-designed and written with friendly licensing. However, I'm
noticing there are a lot of other standard libraries and tools on a
typical system and often many of them are bloated or not so well
designed. I've been looking for a good forum to discuss using
alternative libraries and tools. I've checked out projects like
Beyond Linux from Scratch and suckless.org/Plan 9 and several other
options. I've even checked out some embedded systems mailing
lists/forums. Have yet to find a good forum to discuss topics like
these with interested developers. Does anyone know of a mailing list,
forum, etc. where one could discuss such topics. If so, please let me
know. If not, and if anyone on the list is interested in discussing
this sort of thing further, please e-mail me. At the moment, I'm
investigating finding a less bloated alternative to libintl. Did
notice gettext-tiny as part of Sabotage. I've also run across 3 other
options and am looking at pros and cons of the implementations.

Thanks.

Sincerely,
Laura
Isaac Dunham
2013-04-21 23:26:48 UTC
Permalink
On Sun, 21 Apr 2013 12:30:54 -0400
Post by LM
Musl does a great job of replacing glibc. It's lightweight,
well-designed and written with friendly licensing. However, I'm
noticing there are a lot of other standard libraries and tools on a
typical system and often many of them are bloated or not so well
designed. I've been looking for a good forum to discuss using
alternative libraries and tools. I've checked out projects like
Beyond Linux from Scratch and suckless.org/Plan 9 and several other
options. I've even checked out some embedded systems mailing
lists/forums. Have yet to find a good forum to discuss topics like
these with interested developers. Does anyone know of a mailing list,
forum, etc. where one could discuss such topics. If so, please let me
know. If not, and if anyone on the list is interested in discussing
this sort of thing further, please e-mail me.
FWIW, that's occasionally discussed on the Puppy Linux forums (programming subforum, some threads within Puppy Projects, and some discussion elsewhere...)
Some puplets are large, some are small, and most of the developers are after a way to build a small, fast, easy-to-use system (I know of several projects they have that target systems with less than 64 megs of RAM, and they're not tossing out hotplug)

This reminds me of this comment[1] from Flash, the moderator at the Puppy Linux forums:
You guys feel free to invite the developers to join our forum.
Explain to them how useful Puppy could be to a developer.

I begged off at the time, but should probably mention it...
Although I use Puppy rather infrequently now (read: haven't booted that in several months, maybe a year), the Puppy forum is a better community than most I've run across.

[1] http://www.murga-linux.com/puppy/viewtopic.php?p=627636#627636

HTH,
Isaac Dunham
Rob Landley
2013-04-21 20:24:38 UTC
Permalink
Post by LM
Musl does a great job of replacing glibc. It's lightweight,
well-designed and written with friendly licensing. However, I'm
noticing there are a lot of other standard libraries and tools on a
typical system and often many of them are bloated or not so well
designed. I've been looking for a good forum to discuss using
alternative libraries and tools.
Another place to ask would be:

http://lists.celinuxforum.org/mailman/listinfo/celinux-dev

Which is the mailing list for the elinux.org wiki. It used to be Tim
Bird's Consumer Electronic Linux Forum project, but unfortunately the
Linux Foundation gobbled up CELF and turned it into The Linux
Foundation Embedded Linux Conference By The Linux Foundation, and again
discussion on the list tailed off.

It does have the advantage that it's got developers for actual
electronics manufacturers (mostly japanese) subscribed to it...

Once upon a time busybox.net was collecting some of these:

http://busybox.net/tinyutils.html

But that largely seems to have rolled to a stop since I handed it off...

The group of guys on #edev on freenode are quite knowledgeable about
hardware stuff...

Rob
LM
2013-04-24 11:39:56 UTC
Permalink
Post by Rob Landley
http://busybox.net/tinyutils.html
Thanks for mentioning this. I've seen the page before, but this was an
opportune time to go back and revisit it. One thing on my todo list is to
find a non-interactive way to build Perl and microperl (as mentioned on the
page) appears to be just the solution I was looking for.

Would be curious if anyone's tried using the microperl makefile with musl.
I ran a search to see if I could dig up any more information on microperl.
I did see mention that the Perl developers weren't sure if they'd continue
to support it in the future. Also found a couple of build scripts. Both
install microperl executable and then create a link to perl using it. Both
install various .pm files, but they each installed different sets of
files. If anyone comes across documentation on installing microperl or has
any recommendations on which files are most useful to install and where to
put them, would be very interested in more details.

Thanks.

Sincerely,
Laura
http://www.distasis.com/cpp
Rob Landley
2013-04-25 19:30:31 UTC
Permalink
Post by LM
Post by Rob Landley
http://busybox.net/tinyutils.html
Thanks for mentioning this. I've seen the page before, but this was an
opportune time to go back and revisit it. One thing on my todo list is to
find a non-interactive way to build Perl and microperl (as mentioned on the
page) appears to be just the solution I was looking for.
The automated Linux From Scratch 6.8 build I did in Aboriginal Linux
builds perl without requiring user interaction.

wget http://landley.net/aboriginal/bin/system-image-mips.tar.bz2 (or
whichever)
tar xvjf system-image-mips.tar.bz2
cd system-image-mips
wget
http://landley.net/aboriginal/control-images/downloads/binaries/lfs-bootstrap.hdc
./native-build.sh lfs-bootstrap.hdc

Does require qemu to be installed on the host...

(I don't _think_ that lfs-bootstrap binary is too stale. The
system-image-mips is from the start of this month, when I finally got
the darn arm bugs ironed out of the 3.8 kernel. I've been meaning to
update to LFS 7.3 before cutting a new release, that one's 6.8. Plus a
new dropbear binary came out for the static-tools.hdc build control
image...)

However, I note all that currently builds against uClibc, not musl.
(Day job eats time...)
Post by LM
Would be curious if anyone's tried using the microperl makefile with musl.
Microperl is part of the normal perl build. The perl build system is
implemented _in_ perl (configure and make both), so first they build
microperl to run the rest of it.

It's not really an intentionally shipped product, just a side effect of
perl's head being up its own ass.
Post by LM
I ran a search to see if I could dig up any more information on microperl.
I did see mention that the Perl developers weren't sure if they'd continue
to support it in the future. Also found a couple of build scripts.
Both
install microperl executable and then create a link to perl using it. Both
install various .pm files, but they each installed different sets of
files. If anyone comes across documentation on installing microperl or has
any recommendations on which files are most useful to install and where to
put them, would be very interested in more details.
I'm under the vague impression that microperl is just a perl
interpreter with several of the default libraries bundled into it, so
it doesn't need to find anything in external search paths to run the
perl configure and build.

I remember that people using it outside the perl build originally came
as a surprise to the perl developers, but that was something like a
decade ago and I expect they're used to it by now. Doesn't mean they
put effort into supporting it...

Rob
Rob Landley
2013-04-21 20:17:05 UTC
Permalink
Post by LM
Musl does a great job of replacing glibc. It's lightweight,
well-designed and written with friendly licensing. However, I'm
noticing there are a lot of other standard libraries and tools on a
typical system and often many of them are bloated or not so well
designed. I've been looking for a good forum to discuss using
alternative libraries and tools. I've checked out projects like
Beyond Linux from Scratch and suckless.org/Plan 9 and several other
options. I've even checked out some embedded systems mailing
lists/forums. Have yet to find a good forum to discuss topics like
these with interested developers. Does anyone know of a mailing list,
forum, etc. where one could discuss such topics. If so, please let me
know. If not, and if anyone on the list is interested in discussing
this sort of thing further, please e-mail me. At the moment, I'm
investigating finding a less bloated alternative to libintl. Did
notice gettext-tiny as part of Sabotage. I've also run across 3 other
options and am looking at pros and cons of the implementations.
If you find one, I'm interested in hearing about it.

In theory the linux-embedded list might be a good place:

http://vger.kernel.org/vger-lists.html#linux-embedded

In practice, not much goes on there and the last post to that list was
January. But if people did start posting there, I could join in. :)

Rob
Rich Felker
2013-04-22 14:53:46 UTC
Permalink
Post by LM
Musl does a great job of replacing glibc. It's lightweight,
well-designed and written with friendly licensing. However, I'm
noticing there are a lot of other standard libraries and tools on a
typical system and often many of them are bloated or not so well
designed. I've been looking for a good forum to discuss using
alternative libraries and tools. I've checked out projects like
I have no objection to discussion on this list, at least for the time
being. If it becomes too much clutter, I might request later that we
move such discussions to a separate list, but for now, go ahead. We've
already had a few past discussions along these lines.

For what it's worth, I think poor design is really the core of the
problem. Bloat often stems from the effort needed to work around the
bad design. The major design issues I've seen in widespread libraries
are:

- Tacking on error handling as an afterthought or not at all. This
leads to ugly things like the caller having to setup a jmp_buf for
the library to bail out on error (likely leaking lots of memory in
the process) or just aborting the program on error. Sometimes the
hacks like longjmp are added in ways that they're not safe with
multiple users of the library or with threads.

- Multiple "portable runtime" layers to do things the standard library
already does, usually in less-portable or buggy ways.

- Making a big deal out of string handling. Ugly OO-in-C string types
with dynamic allocation and slow concatenation operations all over
the place. Basically, anything that's generating strings any way
other than snprintf/asprintf, unless it's performance-critical, is
broken.

- Gratuitous OO design. Good OO in C is having the complex or
long-lived objects your library deals with well-encapsulated by
library functions, and fully self-contained, without global state.
Bad OO in C is reimplementing the C++ STL in C, with things like
container objects, iterator objects, string objects, etc., as well
as trying to do any global-level inheritance. As a clarification,
things like implementing multiple video decoders that inherit an
interface from a base "class" is reasonable; making every single
object in your library derive from something like "GObject" is not.

- Gratuitous ugly typedefs, especially as part of the external API,
and especially when they duplicate standard types (INT, gint, etc.).

- Interfaces that encourage or require UB to use them. One example
that comes to mind is functions with signatures like posix_memalign
which take a void** argument to store the result of an allocation.
This encourages things like (void **)&ptr where ptr has type int*,
for example.

- Thread allergies, i.e. horribly over-complicating program logic to
avoid threads. The best examples I can think of are the added logic
needed to generalize a program that's reading from ordinary file
descriptors (e.g. connection sockets) in an event loop to support
SSL sockets or zlib-compressed streams. (Note: there are ways to
address this kind of problem more cleanly without threads too, but
nobody does it. I can elaborate if anybody's interested.)

- DBus.

- Use of global state. Even seemingly-harmless things like a global
registered log function are harmful, because two different libraries
(or the main program and a library) might be trying to use the
library with the global log destination, and clobbering each other's
choices.

- Designs based on shared libraries, especially lots of them. This
creates bloat and often interferes with the ability to use static
linking.

- Excessive dependence on external files and filesystem layout. This
conflicts with easily migrating the binaries to other machines.

- Dependency on any library with the above problems. :-)

Rich
Luca Barbato
2013-04-22 15:21:25 UTC
Permalink
Post by Rich Felker
- Thread allergies, i.e. horribly over-complicating program logic to
avoid threads. The best examples I can think of are the added logic
needed to generalize a program that's reading from ordinary file
descriptors (e.g. connection sockets) in an event loop to support
SSL sockets or zlib-compressed streams. (Note: there are ways to
address this kind of problem more cleanly without threads too, but
nobody does it. I can elaborate if anybody's interested.)
I'm interested to read about it.
Post by Rich Felker
- DBus.
Sadly nobody is pushing for a better local socket multicast abstraction
to send notifications back and forth in an efficient fashion.

I'm hoping for nanomsg once it is complete or Binder once it is
correctly documented ^^; (and thus implemented in more than few forks of
linux and maybe haiku)
Post by Rich Felker
- Use of global state. Even seemingly-harmless things like a global
registered log function are harmful, because two different libraries
(or the main program and a library) might be trying to use the
library with the global log destination, and clobbering each other's
choices.
For this there aren't solution that won't cause different problems I'm
afraid.
Post by Rich Felker
- Designs based on shared libraries, especially lots of them. This
creates bloat and often interferes with the ability to use static
linking.
Special mention to those that want to do clever stuff on the init
section (e.g. change a program global state from there)
Post by Rich Felker
- Dependency on any library with the above problems. :-)
And that kills everybody using glib? *runs and hides*

lu
LM
2013-04-22 16:40:37 UTC
Permalink
Post by Luca Barbato
Post by Rich Felker
- Dependency on any library with the above problems. :-)
And that kills everybody using glib? *runs and hides*
I guess that's one of the reasons I'm trying to eliminate glib from my
system and thus, at the moment, trying to find a gettext alternative that
doesn't need it.

Was reading about the Citrus Project ( http://citrus.bsdclub.org/ )
creating a BSD licensed gettext. Having trouble digging up latest source
code for it though. I also ran across Apache's version of iconv (
http://apr.apache.org/download.cgi ) but didn't see anything for gettext
mentioned at the site.

As time allows, I'm reworking sdcv which uses glib so that it doesn't need
it. Then, all I need to do is find some useable replacements for SciTE and
Sylpheed and I think I will have eliminated any major needs for glib in the
Open Source applications and libraries I prefer.
Daniel Cegiełka
2013-04-22 16:47:02 UTC
Permalink
Post by LM
Post by Luca Barbato
Post by Rich Felker
- Dependency on any library with the above problems. :-)
And that kills everybody using glib? *runs and hides*
I guess that's one of the reasons I'm trying to eliminate glib from my
system and thus, at the moment, trying to find a gettext alternative that
doesn't need it.
https://github.com/rofl0r/gettext-tiny

Daniel
Rich Felker
2013-04-22 22:07:23 UTC
Permalink
Post by Daniel Cegiełka
Post by LM
Post by Luca Barbato
Post by Rich Felker
- Dependency on any library with the above problems. :-)
And that kills everybody using glib? *runs and hides*
I guess that's one of the reasons I'm trying to eliminate glib from my
system and thus, at the moment, trying to find a gettext alternative that
doesn't need it.
https://github.com/rofl0r/gettext-tiny
I think the goal is to have gettext functionality, not just stubs.
Please correct me if I'm mistaken.

With that said, an old version of GNU gettext should work fine. I
wasn't even aware that the current version depends on glib; it
certainly didn't in the past.

I have certainly considered at times including a working gettext
implementation in musl or writing one for external use, but haven't
gotten around to it. There's no reason for it to be anything but tiny.
It doesn't do anything but search for and return strings out of
mmapped files.

Rich
LM
2013-04-23 12:50:29 UTC
Permalink
Post by Rich Felker
Post by Daniel Cegiełka
https://github.com/rofl0r/gettext-tiny
I think the goal is to have gettext functionality, not just stubs.
Please correct me if I'm mistaken.
With that said, an old version of GNU gettext should work fine. I
wasn't even aware that the current version depends on glib; it
certainly didn't in the past.
Correct, I was looking for a functional drop-in replacement that was better
designed or at least needed less dependencies. Gettext was looking for
libxml2, libcroco and glib or would happily use its own versions if not
supplied. It also has a circular reference with libiconv on some
platforms. Was hoping for something more lightweight or at least with less
dependencies.

Wasn't aware that older versions didn't have all those dependencies. That
definitely gives me one more alternative. I did find an old version of bsd
gettext (
http://www.postgresql.org/message-id/Pine.LNX.4.30.0105222006170.757-***@peter.localdomain)
that appears to be able to replace libintl and gettext. It doesn't
supply replacements for msgfmt, msgmerge, xgettext. Was checking if
gettext-tiny might replace them, but doesn't seem to supply the
functionality used by the Open Source program I was trying to compile.
Doesn't even get past the configure tests. Might look into the possibility
of combining the older bsd gettext with any current modifications that
might look useful from BSD Citrus project. Not sure if what I currently
have works properly for every language, but it at least appears to be
working for what I tested and it's a start.
Post by Rich Felker
https://github.com/pkgconf/pkgconf might be handy btw.
Was going to mention that yesterday as well. pkgconf is a great drop-in
replacement for pkgconfig and does not require glib or any circular library
references. The developers of pkgconf were also nice about adding support
and/or receiving patches. I absolutely cannot say the same thing about the
glib developers.
Post by Rich Felker
If you want the data structures, I think that means you should be using
C++, not C.

This may be very much a matter of taste, because you can write data
structures in any language, but that certainly sums up my preferences. I
find C++ a very useful language for expressing data structures.
Post by Rich Felker
Many moons ago I started a thread on here (or was it on freenode?) asking
about lightweight alternatives to stuff and the need for a wiki page
tracking them.

I try to document them as I have time at my own web site. I also add links
to the MinGW and OpenWatcom wikis if I find libraries or tools that might
be useful to developers. I know there are some die-hard Linux advocates on
this list. My own bias is toward cross-platform portability. I started
out and still am a cross-platform programmer. I prefer to be able to write
programs in such a way that they're easy to port to any machine. I don't
like to be locked into only being able to use one operating system.
Sometimes one doesn't have a choice at work which operating system one is
stuck with and it's nice to be able to have Open Source tools and utilities
to work on any system.

Here's what I've dug up for MinGW as far as various tools, utilities and
libraries:
http://www.mingw.org/wiki/Community_Supplied_Links
The links are definitely slanted toward working with MinGW, but as I
mentioned, my preference is for cross-platform, so many of the libraries
and some of the utilities links work on a variety of operating systems. A
few of them came from Linux/Unix environments and I had to send in patches
to port before they'd even work with MinGW.
Post by Rich Felker
I know we discussed more stuff (rxvt, xcfe and lxde...)
I'll throw in a mention of razor-qt and Equinox Desktop Environment as
lighter-weight desktop environment alternatives. Personally, I've never
found xfce efficient on my older machine. I currently prefer a lightweight
window manager and a few utilities over a desktop. I also like
rxvt-unicode. The ability to run multiple terminal windows with a daemon
in order to save memory is really nice. The other terminals with this
feature (like LXTerminal, lilyterm, evilvte and Sakura) all seem to require
VTE and don't seem as efficient in some ways.

Connochaetos has some helpful memory usage information for comparing
various desktop and X applications. Gives some statistics of rxvt-unicode
versus other terminals too in case anyone's curious.
http://www.connochaetos.org/wiki/devel:x-apps

Sincerely,
Laura
http://www.distasis.com/cpp/osrclist.htm
John Spencer
2013-04-23 14:40:48 UTC
Permalink
Post by LM
Post by Rich Felker
Post by Daniel Cegiełka
https://github.com/rofl0r/gettext-tiny
I think the goal is to have gettext functionality, not just stubs.
Please correct me if I'm mistaken.
With that said, an old version of GNU gettext should work fine. I
wasn't even aware that the current version depends on glib; it
certainly didn't in the past.
Correct, I was looking for a functional drop-in replacement that was better
designed or at least needed less dependencies.
[...]
Post by LM
I did find an old version of bsd
gettext (
that appears to be able to replace libintl and gettext. It doesn't
supply replacements for msgfmt, msgmerge, xgettext. Was checking if
gettext-tiny might replace them, but doesn't seem to supply the
gettext-tiny is currently working as a stub that bounces back the
original english message.
however msgfmt and msgmerge use a full blown .po parser that can easily
be adapted to do the translation thing.
you are welcome to do the work (as long as you dont use C++)!
Post by LM
functionality used by the Open Source program I was trying to compile.
Doesn't even get past the configure tests.
which program was that ?
the main idea behind gettext-tiny is that it should make all configure
scripts happy. so this is a bug, please report details here or open an
issue on the github repo.
Rich Felker
2013-04-23 14:58:38 UTC
Permalink
Post by John Spencer
Post by LM
functionality used by the Open Source program I was trying to compile.
Doesn't even get past the configure tests.
which program was that ?
the main idea behind gettext-tiny is that it should make all
configure scripts happy. so this is a bug, please report details
here or open an issue on the github repo.
Some programs need the gettext tools for .po/.mo files. I've found
that making an empty shell script serves as a suitable replacement for
them if you don't need message translation.

Rich
Luca Barbato
2013-04-22 19:31:17 UTC
Permalink
Post by LM
I guess that's one of the reasons I'm trying to eliminate glib from my
system and thus, at the moment, trying to find a gettext alternative that
doesn't need it.
Well eina from the enlightenment people isn't half bad if you need some
data structures but I doubt people would replace their working code
leveraging glib to use something else.
Post by LM
As time allows, I'm reworking sdcv which uses glib so that it doesn't need
it. Then, all I need to do is find some useable replacements for SciTE and
Sylpheed and I think I will have eliminated any major needs for glib in the
Open Source applications and libraries I prefer.
https://github.com/pkgconf/pkgconf might be handy btw.
Rob Landley
2013-04-22 23:24:06 UTC
Permalink
Post by LM
Post by Luca Barbato
Post by Rich Felker
- Dependency on any library with the above problems. :-)
And that kills everybody using glib? *runs and hides*
I guess that's one of the reasons I'm trying to eliminate glib from my
system and thus, at the moment, trying to find a gettext alternative that
doesn't need it.
Was reading about the Citrus Project ( http://citrus.bsdclub.org/ )
creating a BSD licensed gettext. Having trouble digging up latest source
code for it though. I also ran across Apache's version of iconv (
http://apr.apache.org/download.cgi ) but didn't see anything for gettext
mentioned at the site.
I use http://penma.de/code/gettext-stub/ which doesn't fix the problem,
but does avoid it for my use cases. An actual desktop should be
internationalizable, though.
Post by LM
As time allows, I'm reworking sdcv which uses glib so that it doesn't need
it. Then, all I need to do is find some useable replacements for SciTE and
Sylpheed and I think I will have eliminated any major needs for glib in the
Open Source applications and libraries I prefer.
Many moons ago I started a thread on here (or was it on freenode?)
asking about lightweight alternatives to stuff and the need for a wiki
page tracking them.

I believe that at the time, the musl wiki was insufficiently
combobulated, but it has since been fixed. The
http://busybox.net/tinyutils.html page is old is stale and never really
had good coverage, http://elinux.org/System_Size is sort of adjacent to
the topic. If we do come up with anything, can we put it on a wiki page?

Rob
Rich Felker
2013-04-22 23:31:10 UTC
Permalink
Post by Rob Landley
On Mon, Apr 22, 2013 at 11:21 AM, Luca Barbato
Post by Luca Barbato
Post by Rich Felker
- Dependency on any library with the above problems. :-)
And that kills everybody using glib? *runs and hides*
I guess that's one of the reasons I'm trying to eliminate glib from my
system and thus, at the moment, trying to find a gettext
alternative that
doesn't need it.
Was reading about the Citrus Project ( http://citrus.bsdclub.org/ )
creating a BSD licensed gettext. Having trouble digging up latest source
code for it though. I also ran across Apache's version of iconv (
http://apr.apache.org/download.cgi ) but didn't see anything for gettext
mentioned at the site.
I use http://penma.de/code/gettext-stub/ which doesn't fix the
problem, but does avoid it for my use cases. An actual desktop
should be internationalizable, though.
Agreed.
Post by Rob Landley
As time allows, I'm reworking sdcv which uses glib so that it
doesn't need
it. Then, all I need to do is find some useable replacements for SciTE and
Sylpheed and I think I will have eliminated any major needs for glib in the
Open Source applications and libraries I prefer.
Many moons ago I started a thread on here (or was it on freenode?)
asking about lightweight alternatives to stuff and the need for a
wiki page tracking them.
I believe that at the time, the musl wiki was insufficiently
combobulated, but it has since been fixed. The
http://busybox.net/tinyutils.html page is old is stale and never
really had good coverage, http://elinux.org/System_Size is sort of
adjacent to the topic. If we do come up with anything, can we put it
on a wiki page?
That would be a perfectly acceptable topic for the wiki.

Rich
Rob Landley
2013-04-23 00:54:55 UTC
Permalink
Post by Rich Felker
Post by Rob Landley
Many moons ago I started a thread on here (or was it on freenode?)
asking about lightweight alternatives to stuff and the need for a
wiki page tracking them.
I believe that at the time, the musl wiki was insufficiently
combobulated, but it has since been fixed. The
http://busybox.net/tinyutils.html page is old is stale and never
really had good coverage, http://elinux.org/System_Size is sort of
adjacent to the topic. If we do come up with anything, can we put it
on a wiki page?
That would be a perfectly acceptable topic for the wiki.
Obviously I'd nominate toybox. (And busybox.) If only so you have a
context for "ok, what do these NOT cover".

There's an existing "musl vs uClibc" page but we might want a sentence
or two on "musl: obviously the best", "uClibc: tried really hard but
buildroot squished it in 2005 and it never recovered". "dietlibc:
widely mocked for not-a-but-thats-a-feature disease", "klibc: official
libc of the postminimalist art movement".)

"There's always room for dropbear". And polarssl, and so on.

I know we discussed more stuff (rxvt, xcfe and lxde...)

Rob
Rich Felker
2013-04-23 01:46:40 UTC
Permalink
Post by Rob Landley
Post by Rich Felker
Post by Rob Landley
Many moons ago I started a thread on here (or was it on freenode?)
asking about lightweight alternatives to stuff and the need for a
wiki page tracking them.
I believe that at the time, the musl wiki was insufficiently
combobulated, but it has since been fixed. The
http://busybox.net/tinyutils.html page is old is stale and never
really had good coverage, http://elinux.org/System_Size is sort of
adjacent to the topic. If we do come up with anything, can we put it
on a wiki page?
That would be a perfectly acceptable topic for the wiki.
Obviously I'd nominate toybox. (And busybox.) If only so you have a
context for "ok, what do these NOT cover".
Of course. My proposed criteria would just be:

1. Must be free software.

2. Duplicates a significant portion of the usage cases of a program or
library that's widely perceived as bloated or otherwise problematic
for systems musl might be used on, in a way that fixes some or all of
these problems. Reasons other than bloat might be waking up every
second to eat battery (non-Busybox ntpd does this!), requiring dynamic
linking, etc.

3. Non-criterion: the software doesn't have to be perfect or lack any
bloat problems itself; it just has to be better than the mainstream
solution in at least one way that might be significant to our users.

4. Not by Lennart Poettering. :-)

Does this sound reasonable?
Post by Rob Landley
There's an existing "musl vs uClibc" page but we might want a
sentence or two on "musl: obviously the best", "uClibc: tried really
hard but buildroot squished it in 2005 and it never recovered".
"dietlibc: widely mocked for not-a-but-thats-a-feature disease",
"klibc: official libc of the postminimalist art movement".)
If such changes are going to be made, I think they should be done by
somebody who's not going to word them in that way... :-)
Post by Rob Landley
"There's always room for dropbear". And polarssl, and so on.
cyassl looked promising too. I would probably mention tomcrypt too
even though it's not sufficient to do SSL; it has the most slim,
clean, portable implementations of crypto algorithms I've seen.
Post by Rob Landley
I know we discussed more stuff (rxvt, xcfe and lxde...)
These are a bit more borderline, but I wouldn't call them
unacceptable.

Rich
Isaac Dunham
2013-04-23 05:04:30 UTC
Permalink
On Mon, 22 Apr 2013 21:46:40 -0400
Post by Rich Felker
Post by Rob Landley
"There's always room for dropbear". And polarssl, and so on.
cyassl looked promising too. I would probably mention tomcrypt too
even though it's not sufficient to do SSL; it has the most slim,
clean, portable implementations of crypto algorithms I've seen.
wpa_supplicant can use tomcrypt (external or internal) as fallback if no other encryption method (ie, openssl/gnutls) is configured, so I'd say it merits a mention.

I wonder if some notes should be put somewhere to point out that a network mangler on top of wpa_supplicant is not needed (the learning curve for configuring it is pretty steep, due to the need to find and understand the docs, but wpa_supplicant + wpa_cli -a script + wpa_cli in command mode can handle most situations, including dhcp).
I mention this because it seems to be "accepted wisdom" (but false) that you need wpa_supplicant as a tool and a network manager to make it useable. And most of the network managers I've encountered are bloat of the highest order:
NetworkManager, wicd, wifiradar...
But this might be better put somewhere else.
--
Isaac Dunham
Rich Felker
2013-04-23 13:47:24 UTC
Permalink
Post by Isaac Dunham
On Mon, 22 Apr 2013 21:46:40 -0400
Post by Rich Felker
Post by Rob Landley
"There's always room for dropbear". And polarssl, and so on.
cyassl looked promising too. I would probably mention tomcrypt too
even though it's not sufficient to do SSL; it has the most slim,
clean, portable implementations of crypto algorithms I've seen.
wpa_supplicant can use tomcrypt (external or internal) as fallback
if no other encryption method (ie, openssl/gnutls) is configured, so
I'd say it merits a mention.
In that case I don't even see why they bother including the code to
use openssl/gnutls...
Post by Isaac Dunham
I wonder if some notes should be put somewhere to point out that a
network mangler on top of wpa_supplicant is not needed (the learning
curve for configuring it is pretty steep, due to the need to find
and understand the docs, but wpa_supplicant + wpa_cli -a script +
wpa_cli in command mode can handle most situations, including dhcp).
I mention this because it seems to be "accepted wisdom" (but false)
that you need wpa_supplicant as a tool and a network manager to make
it useable. And most of the network managers I've encountered are
bloat of the highest order: NetworkManager, wicd, wifiradar... But
this might be better put somewhere else.
Well the accepted wisdom is "almost true": for practical use of mobile
wifi, you need not just wpa_supplicant but also some controlling
process that's capable of:

1. Choosing which network to connect to.
2. Managing keys.
3. Logic for what to do when signal is lost.
4. Automating nonsense click-through agreements on public wifi.
...

The existing solutions all manage the above very poorly. Respectively,
they have:

1. No way to manage network priority/preference order.
2. Annoying popups to ask for key rather than having it be part of the
configuration of the network, and storing the keys in obscure places.
3. Annoying network-hopping.
4. Minimal or no auto-click-through; even when it does work, you can
get burned if your web browser happens to attempt a load before it
succeeds. A correct one needs to encapsulate the connection somehow so
that no connection is exposed to the user at all until the
click-through succeeds.

Rich
Luca Barbato
2013-04-23 21:25:56 UTC
Permalink
Post by Rich Felker
Well the accepted wisdom is "almost true": for practical use of mobile
wifi, you need not just wpa_supplicant but also some controlling
1. Choosing which network to connect to.
2. Managing keys.
3. Logic for what to do when signal is lost.
4. Automating nonsense click-through agreements on public wifi.
...
The existing solutions all manage the above very poorly. Respectively,
1. No way to manage network priority/preference order.
2. Annoying popups to ask for key rather than having it be part of the
configuration of the network, and storing the keys in obscure places.
3. Annoying network-hopping.
4. Minimal or no auto-click-through; even when it does work, you can
get burned if your web browser happens to attempt a load before it
succeeds. A correct one needs to encapsulate the connection somehow so
that no connection is exposed to the user at all until the
click-through succeeds.
connman is the closest I found so far, it uses dbus and glib so isn't
exactly a fit in this list.

lu
Kurt H Maier
2013-04-23 21:50:23 UTC
Permalink
Post by Rich Felker
1. Choosing which network to connect to.
2. Managing keys.
3. Logic for what to do when signal is lost.
4. Automating nonsense click-through agreements on public wifi.
...
wpa_supplicant can do 1., unless you mean choosing between ethernet and
wifi. but it supports priority weighting on access points..

2. is pretty trivial since you can easily wpa_cli >> /etc/wpa.conf or
similar. I've never been sure why typing 'dhclient eth0' is seen as
more onerous than running a polling daemon to save you the trouble.

Can you elucidate more on 3? if the signal is lost, wpa_supplicant
rescans and connects to any configured network, or else sleeps and
rescans later.

4. will never be solved satisfactorally, since that garbage is not
predictable. the database of tedious TOS crap will never stop
expanding.
Post by Rich Felker
1. No way to manage network priority/preference order.
wpa_supplicant has priority= to do this.
Post by Rich Felker
2. Annoying popups to ask for key rather than having it be part of the
configuration of the network, and storing the keys in obscure places.
What is less obscure than the wpa_supplicant config file?
Post by Rich Felker
3. Annoying network-hopping.
this can also be fixed with priority=
Post by Rich Felker
4. Minimal or no auto-click-through; even when it does work, you can
get burned if your web browser happens to attempt a load before it
succeeds. A correct one needs to encapsulate the connection somehow so
that no connection is exposed to the user at all until the
click-through succeeds.
There are lots of useful things that can be done with this concept of an
encapsulated connection. I get burned by this on my work laptop, which
likes to spam VPN connection attempts back to corporate.

khm
Rich Felker
2013-04-24 02:37:39 UTC
Permalink
Post by Kurt H Maier
Post by Rich Felker
1. Choosing which network to connect to.
2. Managing keys.
3. Logic for what to do when signal is lost.
4. Automating nonsense click-through agreements on public wifi.
...
wpa_supplicant can do 1., unless you mean choosing between ethernet and
wifi. but it supports priority weighting on access points..
OK.
Post by Kurt H Maier
2. is pretty trivial since you can easily wpa_cli >> /etc/wpa.conf or
similar. I've never been sure why typing 'dhclient eth0' is seen as
more onerous than running a polling daemon to save you the trouble.
Because you don't have a keyboard, you have a 3-4" touchscreen. And
you want it to keep working as you move from place to place without
any interaction.
Post by Kurt H Maier
Can you elucidate more on 3? if the signal is lost, wpa_supplicant
rescans and connects to any configured network, or else sleeps and
rescans later.
I'm not sure what the ideal behavior is, but I know all the existing
ones have bad behavior.
Post by Kurt H Maier
4. will never be solved satisfactorally, since that garbage is not
predictable. the database of tedious TOS crap will never stop
expanding.
Agree, but it still needs to be solved, even if the solution requires
frequent updates to be fully effective. With decent heuristics though
I think it could be fully automated for most sites with just a few
exceptions for really weird ones..
Post by Kurt H Maier
Post by Rich Felker
4. Minimal or no auto-click-through; even when it does work, you can
get burned if your web browser happens to attempt a load before it
succeeds. A correct one needs to encapsulate the connection somehow so
that no connection is exposed to the user at all until the
click-through succeeds.
There are lots of useful things that can be done with this concept of an
encapsulated connection. I get burned by this on my work laptop, which
likes to spam VPN connection attempts back to corporate.
Agreed. I think really most users should _always_ be running in an
environment where only root sees the real network interfaces and
applications just see a virtual network routed through the real one.

Rich
Kurt H Maier
2013-04-24 04:43:06 UTC
Permalink
Post by Rich Felker
Because you don't have a keyboard, you have a 3-4" touchscreen. And
you want it to keep working as you move from place to place without
any interaction.
If you don't have a keyboard, you likely also don't have physical
ethernet via rj-45, and even if you do, there's no reason to spam
wakeups checking if it's plugged in. touch-only devices have touch-only
interfaces, which can easily include a 'switch to wired' button instead
of a daemon eating your battery.

setting physical links aside, wpa_supplicant can manage roaming fine. I
should clarify that I think wpa_supplicant could do with a massive
wrecking ball, but it has a lot of good functionality that obviates a
ton of overengineered networkmanager-type software. The trick will be
removing the garbage or implementing the good stuff in a slightly less
horrendous fashion.

In fact, I'm firmly of the opinion that complete signal loss is the
*only* time a system should monkey with the network; one of my least
favorite things is my phone aggressively dropping 3G so it can switch
to wifi, dumping my ssh sessions and filesystem mounts in the process.
Post by Rich Felker
Agree, but it still needs to be solved, even if the solution requires
frequent updates to be fully effective. With decent heuristics though
I think it could be fully automated for most sites with just a few
exceptions for really weird ones..
I think the ideal solution is for network administrators to stop
pretending hijacking sessions is acceptable, but until an automated
solution exists I enjoy all the hate they get from users.
Post by Rich Felker
Agreed. I think really most users should _always_ be running in an
environment where only root sees the real network interfaces and
applications just see a virtual network routed through the real one.
This doesn't necessarily solve anything from the user's standpoint
unless he's trained to use the feature appropriately, but it would
enable system designers to get uncomfortably clever in ways that can
make system behavior damn hard to predict. Having said that, there is
software available on plan 9 called aan - 'any available network' - that
works this way and can be very useful.

khm
Rich Felker
2013-04-24 13:37:14 UTC
Permalink
Post by Kurt H Maier
In fact, I'm firmly of the opinion that complete signal loss is the
*only* time a system should monkey with the network; one of my least
favorite things is my phone aggressively dropping 3G so it can switch
to wifi, dumping my ssh sessions and filesystem mounts in the process.
Ideally it would keep using both as long as there were some
"important" connections persisting on the old one, and there would be
a socket option for applications using unimportant persistent
connections to flag them unimportant.
Post by Kurt H Maier
Post by Rich Felker
Agree, but it still needs to be solved, even if the solution requires
frequent updates to be fully effective. With decent heuristics though
I think it could be fully automated for most sites with just a few
exceptions for really weird ones..
I think the ideal solution is for network administrators to stop
pretending hijacking sessions is acceptable, but until an automated
solution exists I enjoy all the hate they get from users.
Maybe once everyone finishes switching to https...then the hijacking
will cease to work, and to give a reasonable user experience, they'll
have to drop hijacking.
Post by Kurt H Maier
Post by Rich Felker
Agreed. I think really most users should _always_ be running in an
environment where only root sees the real network interfaces and
applications just see a virtual network routed through the real one.
This doesn't necessarily solve anything from the user's standpoint
unless he's trained to use the feature appropriately, but it would
The assumption is that the system software, possibly interacting with
the user if the user were allowed to change network settings, would
handle the status of the real connection, and expose it only though
the virtual interface through the user when it's actually working. For
semi-advanced users, this could allow transparent migration (even
keeping your ssh/chat/etc. sessions) if you integrate it with vpn.

Rich
i***@lavabit.com
2013-04-24 00:50:26 UTC
Permalink
Post by Rich Felker
Post by Isaac Dunham
On Mon, 22 Apr 2013 21:46:40 -0400
Post by Rich Felker
Post by Rob Landley
"There's always room for dropbear". And polarssl, and so on.
cyassl looked promising too. I would probably mention tomcrypt too
even though it's not sufficient to do SSL; it has the most slim,
clean, portable implementations of crypto algorithms I've seen.
wpa_supplicant can use tomcrypt (external or internal) as fallback
if no other encryption method (ie, openssl/gnutls) is configured, so
I'd say it merits a mention.
In that case I don't even see why they bother including the code to
use openssl/gnutls...
There are one or two features that need to be disabled to use tomcrypt.
I wish I could remember what they were. But upstream has provided many
options that only duplicate functionality with additional bloat.
(sockets and plain C, vs. DBUS + glib)
Post by Rich Felker
Post by Isaac Dunham
I wonder if some notes should be put somewhere to point out that a
network mangler on top of wpa_supplicant is not needed (the learning
curve for configuring it is pretty steep, due to the need to find
and understand the docs, but wpa_supplicant + wpa_cli -a script +
wpa_cli in command mode can handle most situations, including dhcp).
I mention this because it seems to be "accepted wisdom" (but false)
that you need wpa_supplicant as a tool and a network manager to make
it useable. And most of the network managers I've encountered are
bloat of the highest order: NetworkManager, wicd, wifiradar... But
this might be better put somewhere else.
Well the accepted wisdom is "almost true": for practical use of mobile
wifi, you need not just wpa_supplicant but also some controlling
1. Choosing which network to connect to.
Oh, like wpa_cli select_network ?
Post by Rich Felker
2. Managing keys.
wpa_cli [ passphrase | otp | password | new_password | pin | wps_pbc ]
(though figuring it out may be difficult, even with the help messages)
Post by Rich Felker
3. Logic for what to do when signal is lost.
wpa_supplicant reassociates on non-user-specified disconnects, and
wpa_cli -a <script>
allows configuration of the commands to run on CONNECTED and
DISCONNECTED events.
Post by Rich Felker
4. Automating nonsense click-through agreements on public wifi.
...
Nothing for this, as far as I know. (On the other hand, I tend to dislike
software that pretends that I agreed to something I never saw.
Weird, I know ;-). )
Post by Rich Felker
The existing solutions all manage the above very poorly...
What's worse is how some of them handle changing networks.

wpa_supplicant comes with wpa_cli for a reason: you need to be able to
tell the existing process to change its configuration.
The WRONG way to do things is to create a new config file, start a new
instance of wpa_supplicant using that config file, and leave the old
wpa_supplicant running.
(wicd, I hope you've figured that out by now.)

Of course, setting up wpa_supplicant so that wpa_cli works is not easy.
And while wpa_gui (the Qt interface that corresponds to wpa_cli) is
available, it needs as much preconfiguration as wpa_cli, and the UI
could use some improvement before it's easy to understand (I can follow
it readily, but that's after using wpa_cli without anything else for a year or
two).
A tool capable of producing a functional wpa_supplicant.conf and
providing a gui corresponding to wpa_cli in functionality would handle
most scenarios.
Unfortunately, the existing tools tend not to do that; I should see how
ceni works sometime-it's the only one I know of and haven't tried yet.

--
Isaac Dunham
Rob Landley
2013-04-24 06:11:54 UTC
Permalink
Post by Isaac Dunham
I wonder if some notes should be put somewhere to point out that a
network mangler on top of wpa_supplicant is not needed (the learning
curve for configuring it is pretty steep, due to the need to find and
understand the docs, but wpa_supplicant + wpa_cli -a script + wpa_cli
in command mode can handle most situations, including dhcp).
I mention this because it seems to be "accepted wisdom" (but false)
that you need wpa_supplicant as a tool and a network manager to make
it useable. And most of the network managers I've encountered are
NetworkManager, wicd, wifiradar...
But this might be better put somewhere else.
Split up the page when it gets big and unmanageable. Don't try to
categorize information that's not there yet.

And example shell scripts can be nice. You if you don't use them, it
documents "here's what you need to do".

Rob
Rich Felker
2013-04-22 21:52:48 UTC
Permalink
Post by Luca Barbato
Post by Rich Felker
- Thread allergies, i.e. horribly over-complicating program logic to
avoid threads. The best examples I can think of are the added logic
needed to generalize a program that's reading from ordinary file
descriptors (e.g. connection sockets) in an event loop to support
SSL sockets or zlib-compressed streams. (Note: there are ways to
address this kind of problem more cleanly without threads too, but
nobody does it. I can elaborate if anybody's interested.)
I'm interested to read about it.
Well the canonical multi-threaded way to deal with such cases would be
to replace your file descriptor with a pipe or socket file descriptor
connected to a thread that's doing all the translation work (SSL,
zlib, etc.). Then, your main event loop just sees a normal file
descriptor, so you don't have to invade your code that handles
streams/connections/whatever with a new abstraction framework around
file descriptors that also supports SSL or whatever. Not only are such
abstraction layers bloated; they also add a lot of places for bugs to
hide, and they're difficult to get right. For a great example from
just a few days ago, see Bitlbee bug #1046:

http://bugs.bitlbee.org/bitlbee/ticket/1046

But suppose you want to solve this problem without threads. Well, the
basic idea is to do the same thing, but put the translation layer in
your event loop rather than in a thread. That is, make the pipe or
socketpair like you would have done for the threaded solution, but
instead of creating a thread, register the file descriptors with an
event handler in your main event loop. For an example with SSL, when
data comes in on the SSL socket, first the SSL-processing even handler
gets fired, and writes the decrypted stream to a pipe or socketpair.
Then, the event handler for the other end of that pipe/socket runs and
handles it like an ordinary connection.

Obviously this has some degree of additional overhead compared to an
abstraction layer, so I wouldn't recommend doing it for applications
whose JOB is to handle connections or streaming data as efficiently as
possible (e.g. a webserver). But for something like an IRC client or
web browser, where socket performance is irrelevant compared to other
things, it's idiotic to design fancy abstraction layers for reading
and writing to connections when you could just do a design like what I
described above.
Post by Luca Barbato
Post by Rich Felker
- DBus.
Sadly nobody is pushing for a better local socket multicast abstraction
to send notifications back and forth in an efficient fashion.
I'm hoping for nanomsg once it is complete or Binder once it is
correctly documented ^^; (and thus implemented in more than few forks of
linux and maybe haiku)
Yes, Binder looks really promising, but I also think part of the
problem is that the need for this kind of interface is exaggerated...
Post by Luca Barbato
Post by Rich Felker
- Use of global state. Even seemingly-harmless things like a global
registered log function are harmful, because two different libraries
(or the main program and a library) might be trying to use the
library with the global log destination, and clobbering each other's
choices.
For this there aren't solution that won't cause different problems I'm
afraid.
Sure there are. I get the impression you can tell I was talking about
libav/ffmpeg's log interface. :-) The obvious solution is to bind log
contexts to the object you're acting on. See this nice talk:

http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/

If I remember right, part of the problem way back was that there were
deep function calls that had no context available to them, and that
didn't _need_ a context for anything but logging warnings or whatnot.
Really, the fact that they can fail and want to be able to report
failure means they _do_ need a context, but I can understand the
desire to cheat, especially if there's a performance cost to passing
the context all the way down. In that case, hidden non-global state,
namely thread-local storage, might be an appropriate solution. It's
still hidden state (and thus A Bad Thing), but at least it's no longer
global state.
Post by Luca Barbato
Post by Rich Felker
- Designs based on shared libraries, especially lots of them. This
creates bloat and often interferes with the ability to use static
linking.
Special mention to those that want to do clever stuff on the init
section (e.g. change a program global state from there)
Did whatever lib did that (OpenAL, was it?) ever fix their bugs?
Post by Luca Barbato
Post by Rich Felker
- Dependency on any library with the above problems. :-)
And that kills everybody using glib? *runs and hides*
Yes, basically. Dependency on glib means your library will impose
bloat and it will preclude robustness.

Rich
Luca Barbato
2013-04-22 22:42:01 UTC
Permalink
Post by Rich Felker
Post by Luca Barbato
For this there aren't solution that won't cause different problems I'm
afraid.
Sure there are. I get the impression you can tell I was talking about
libav/ffmpeg's log interface. :-) The obvious solution is to bind log
http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
If I remember right, part of the problem way back was that there were
deep function calls that had no context available to them, and that
didn't _need_ a context for anything but logging warnings or whatnot.
In the specific case yes. I tried to foster proper return error
propagation, so you get something more meaningful than EINVAL/-1 and
that is usually enough in those specific cases.

The general problem is that the library user wants to be the only one
having a say on what goes where so single point overrides are useful.

When you start using those libraries in situations in which you'd like
to have per-$situation logging then you start to scream.

(In the specific case above there is a quite borderline solution since
we can override the global logger and store per context loggers in
creative ways)
Post by Rich Felker
Really, the fact that they can fail and want to be able to report
failure means they _do_ need a context, but I can understand the
desire to cheat, especially if there's a performance cost to passing
the context all the way down. In that case, hidden non-global state,
namely thread-local storage, might be an appropriate solution. It's
still hidden state (and thus A Bad Thing), but at least it's no longer
global state.
Exactly. (bonus points if you do that in a void returning function...)
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
- Designs based on shared libraries, especially lots of them. This
creates bloat and often interferes with the ability to use static
linking.
Special mention to those that want to do clever stuff on the init
section (e.g. change a program global state from there)
Did whatever lib did that (OpenAL, was it?) ever fix their bugs?
Given recent binutils would strip away symbols if used that way, making
linking fail, I hope so.
Post by Rich Felker
Yes, basically. Dependency on glib means your library will impose
bloat and it will preclude robustness.
Yet glib gives you oh-so-many-features (I fell for it once), sadly there
aren't many utility libs that provide sort of useful data structures,
eventloops, threadpools and portable string mangling (e.g. strl*
functions) in a single widespread package.

Some lean/cleaned up alternative are cropping but usually they aren't as
known and widespread.

lu
Rich Felker
2013-04-22 23:06:28 UTC
Permalink
Post by Luca Barbato
Post by Rich Felker
Post by Luca Barbato
For this there aren't solution that won't cause different problems I'm
afraid.
Sure there are. I get the impression you can tell I was talking about
libav/ffmpeg's log interface. :-) The obvious solution is to bind log
http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
If I remember right, part of the problem way back was that there were
deep function calls that had no context available to them, and that
didn't _need_ a context for anything but logging warnings or whatnot.
In the specific case yes. I tried to foster proper return error
propagation, so you get something more meaningful than EINVAL/-1 and
that is usually enough in those specific cases.
The general problem is that the library user wants to be the only one
having a say on what goes where so single point overrides are useful.
The problem with your comment here is the phrase "library user". Who
is the library user? You may be thinking from a standpoint (in our
example) of MPlayer, but what if instead the application you were
looking at were a file manager that itself had no awareness of video
files, and only ended up processing them as part of a library pulled
in from a plugin for file previews? Obviously there is no way the app
can be aware of where the log messages should go since it's not aware
the library even exists. The user is the library that depends on the
library doing the logging, not the app, and it's very possible that
there is more than once such library. In which case, you have madness.
Post by Luca Barbato
When you start using those libraries in situations in which you'd like
to have per-$situation logging then you start to scream.
Keep in mind it might not even be "per-situation logging". It might be
something like one plugin needing to send the message back up to the
application as a popup message to display, and another plugin just
wanting to render the message text as a file preview in place of an
image...
Post by Luca Barbato
Post by Rich Felker
Yes, basically. Dependency on glib means your library will impose
bloat and it will preclude robustness.
Yet glib gives you oh-so-many-features (I fell for it once), sadly there
aren't many utility libs that provide sort of useful data structures,
If you want the data structures, I think that means you should be
using C++, not C.
Post by Luca Barbato
eventloops,
If your program is event-loop-performance-critical (think httpd), you
probably need a custom one for your application.

If it's not, you could probably write a much simpler program using
threads. It might even perform better too.
Post by Luca Barbato
threadpools
Similar situation. Threadpools are an optimization that shouldn't be
applied prematurely, and if you do need them, you probably want a
custom solution.
Post by Luca Barbato
and portable string mangling (e.g. strl*
functions) in a single widespread package.
strl* considered harmful, for 3 reasons:

1. Concatenation is a bad idiom. See
http://en.wikipedia.org/wiki/Schlemiel_the_Painter's_algorithm

2. It duplicates standard functionality in snprintf. Yes it's quicker
for some cases (pure copying), but in that case you probably should
just be using the original string in-place. On the other hand,
building the whole output in one step with snprintf makes it obvious
that your code is correct and avoids issue #1 above.

3. While returning the length that "would be needed" is useful if you
want to allocate and retry, it's harmful if your goal is to protect
against malicious inputs. For instance,

strlcpy(buf, some_500_gb_string, 10)

has to process the whole input string even though it's going to throw
away most of it. BTW this issue applies to snprintf too, but at least
it has something useful to be computing. With strl*, the return value
is always something that would be trivial to obtain yourself if you
needed it.

I'm aware some people like strl* and we have them in musl because it's
better to provide them than to deal with people rolling their own and
doing it in wrong and insecure ways. But I still would recommend
against using them in new code.

Rich
Luca Barbato
2013-04-23 00:26:21 UTC
Permalink
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
Post by Luca Barbato
For this there aren't solution that won't cause different problems I'm
afraid.
Sure there are. I get the impression you can tell I was talking about
libav/ffmpeg's log interface. :-) The obvious solution is to bind log
http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
If I remember right, part of the problem way back was that there were
deep function calls that had no context available to them, and that
didn't _need_ a context for anything but logging warnings or whatnot.
In the specific case yes. I tried to foster proper return error
propagation, so you get something more meaningful than EINVAL/-1 and
that is usually enough in those specific cases.
The general problem is that the library user wants to be the only one
having a say on what goes where so single point overrides are useful.
The problem with your comment here is the phrase "library user". Who
is the library user? You may be thinking from a standpoint (in our
example) of MPlayer, but what if instead the application you were
looking at were a file manager that itself had no awareness of video
files, and only ended up processing them as part of a library pulled
in from a plugin for file previews? Obviously there is no way the app
can be aware of where the log messages should go since it's not aware
the library even exists. The user is the library that depends on the
library doing the logging, not the app, and it's very possible that
there is more than once such library. In which case, you have madness.
Usually (at least that's what I do in those case) the global logger is
overridden to use the outer library logger then you -end-user- override
it as well and then everything goes where you want.

The other widespread solution is to eat stderr and if something appears
show to the user, crude but not so bad.
Post by Rich Felker
Post by Luca Barbato
When you start using those libraries in situations in which you'd like
to have per-$situation logging then you start to scream.
Keep in mind it might not even be "per-situation logging". It might be
something like one plugin needing to send the message back up to the
application as a popup message to display, and another plugin just
wanting to render the message text as a file preview in place of an
image...
Yeah, logging messages properly is terrible.
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
Yes, basically. Dependency on glib means your library will impose
bloat and it will preclude robustness.
Yet glib gives you oh-so-many-features (I fell for it once), sadly there
aren't many utility libs that provide sort of useful data structures,
If you want the data structures, I think that means you should be
using C++, not C.
C++ stock data structures are too runtime-dependent, crafting your own
means getting the worst of both words if you aren't extremely careful =\

Hopefully the new crop of system languages would try to capitalize on
the experience...
Post by Rich Felker
Post by Luca Barbato
eventloops,
If your program is event-loop-performance-critical (think httpd), you
probably need a custom one for your application.
If it's not, you could probably write a much simpler program using
threads. It might even perform better too.
event loops, coroutines and threads have their uses.
Post by Rich Felker
Post by Luca Barbato
threadpools
Similar situation. Threadpools are an optimization that shouldn't be
applied prematurely, and if you do need them, you probably want a
custom solution.
There is always a tradeoff, reimplementing the wheel is ok if only
square ones are available.
Post by Rich Felker
Post by Luca Barbato
and portable string mangling (e.g. strl*
functions) in a single widespread package.
1. Concatenation is a bad idiom. See
http://en.wikipedia.org/wiki/Schlemiel_the_Painter's_algorithm
scatter-gather lists aren't that widespread =)
Post by Rich Felker
2. It duplicates standard functionality in snprintf. Yes it's quicker
for some cases (pure copying), but in that case you probably should
just be using the original string in-place. On the other hand,
building the whole output in one step with snprintf makes it obvious
that your code is correct and avoids issue #1 above.
Using snprintf in those cases doesn't scream efficient if I recall
correctly.
Post by Rich Felker
I'm aware some people like strl* and we have them in musl because it's
better to provide them than to deal with people rolling their own and
doing it in wrong and insecure ways. But I still would recommend
against using them in new code.
That's more or less the annoying question: better to provide good
implementation of slightly dangerous to use code or let people bake
their own? (usually you have people using something not implemented
perfectly doesn't matter what, but again, tradeoffs...)

lu
Rob Landley
2013-04-23 02:14:27 UTC
Permalink
Post by Luca Barbato
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
Post by Luca Barbato
For this there aren't solution that won't cause different
problems I'm
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
Post by Luca Barbato
afraid.
Sure there are. I get the impression you can tell I was talking
about
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
libav/ffmpeg's log interface. :-) The obvious solution is to bind
log
http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
If I remember right, part of the problem way back was that there
were
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
deep function calls that had no context available to them, and
that
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
didn't _need_ a context for anything but logging warnings or
whatnot.
Post by Rich Felker
Post by Luca Barbato
In the specific case yes. I tried to foster proper return error
propagation, so you get something more meaningful than EINVAL/-1
and
Post by Rich Felker
Post by Luca Barbato
that is usually enough in those specific cases.
The general problem is that the library user wants to be the only
one
Post by Rich Felker
Post by Luca Barbato
having a say on what goes where so single point overrides are
useful.
Post by Rich Felker
The problem with your comment here is the phrase "library user". Who
is the library user? You may be thinking from a standpoint (in our
example) of MPlayer, but what if instead the application you were
looking at were a file manager that itself had no awareness of video
files, and only ended up processing them as part of a library pulled
in from a plugin for file previews? Obviously there is no way the
app
Post by Rich Felker
can be aware of where the log messages should go since it's not
aware
Post by Rich Felker
the library even exists. The user is the library that depends on the
library doing the logging, not the app, and it's very possible that
there is more than once such library. In which case, you have
madness.
Usually (at least that's what I do in those case) the global logger is
overridden to use the outer library logger then you -end-user-
override
it as well and then everything goes where you want.
The other widespread solution is to eat stderr and if something appears
show to the user, crude but not so bad.
Post by Rich Felker
Post by Luca Barbato
When you start using those libraries in situations in which you'd
like
Post by Rich Felker
Post by Luca Barbato
to have per-$situation logging then you start to scream.
Keep in mind it might not even be "per-situation logging". It might
be
Post by Rich Felker
something like one plugin needing to send the message back up to the
application as a popup message to display, and another plugin just
wanting to render the message text as a file preview in place of an
image...
Yeah, logging messages properly is terrible.
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
Yes, basically. Dependency on glib means your library will impose
bloat and it will preclude robustness.
Yet glib gives you oh-so-many-features (I fell for it once), sadly
there
Post by Rich Felker
Post by Luca Barbato
aren't many utility libs that provide sort of useful data
structures,
Post by Rich Felker
If you want the data structures, I think that means you should be
using C++, not C.
C++ stock data structures are too runtime-dependent, crafting your own
means getting the worst of both words if you aren't extremely careful =\
Hopefully the new crop of system languages would try to capitalize on
the experience...
What new crop of system languages?

C is a portable assembly language with minimal abstraction between the
programmer and what the hardware is actually doing. It uses static
typing, static memory allocation, and if you really care you can
explicitly specify integer sizes (uint16_t or LP64) and handle
endianness and alignment and so on down to memory mapped bitmasks. It
provides simple container types based on pointer math: arrays are
simple pointer arithmetic, and structs concatenate a group of variables
so each member name corresponds to a fixed offset and size (static,
determined at compile time) where the value is to be found relative to
the pointer to the start of the struct.

Scripting languages like python/lua/ruby have opaque abstractions where
you honestly don't need to know how it's implemented. They replace
poitners with references, and build garbage collection and dynamic
typing on top of that. Their built-in container types are resizeable,
including an array variant and a dictionary variant. The dictionaries
aggregate via keyword/value association, so you can add and remove
members on the fly.

In C, types are a property of pointers. In scripting languages, types
are a property of objects, meaning _references_have_no_type_. You
dereference to find out what type it is. So when you implement
functions, you find out what type it is when you try to use it, but
asking for a member and performing an operation on that member. If the
member isn't there, or doesn't support that operation, it throws an
exception. You can catch that exception and handle it however you like,
up to and including adjusting the object to add the member in question
so it _can_ succeed. But if you don't catch the exception locally, no
problem: it's all garbage collected. References that fall out of scope
are naturally freed by the system.

These are two fundamentally different ways of programming. scripting
languages are dynamic, everything interesting determineda t runtime, to
the point where they don't even have a compilation step. You set the
executable bit on your source code. (Is there a bytecode compilation
step at load time with an optimized interpreter doing batched code
translation with buffering that Sun's marketing department called "just
in time" or some such nonsense but which Apple's 68k emulator for the
PowerPC was already doing in 1994? Maybe. Again: it doesn't matter, the
abstractions are opaque, it all just works.)

So with C: pointers, everything statically compiled to machine
language, no abstraction. With scripting langauges: references,
interpreted at runtime, opaque abstraction and often multiple different
but compatible implementations (python/jython).

Then you have C++, which can't decide which it is. C is a local peak in
language design space. scripting languages are another. C++ is in the
valley between them, neither fish nor fowl, with the worst
characteristics of both. It's a static language like C, statically
typed and based on pointers, with thick layers of _leaky_ abstractions.
If anything goes wrong, you have to know all the implementation details
of the standard template library in order to debug it. Your global
constructors are called before main() and those have zeroed memory but
when you new() an object it doesn't have zeroed memory and you must
initialize every single member in the constructor and of coure you
can't memset(this, 0, sizeof(this)) because there's magic data in the
object for RTTI and virtual methods which you can't _see_ but which you
can trivially damage if you don't know the magic invisible
implementation details.

ALL of C++ is magic invisible implementation details. The only way to
safely use the language is to know enough about it you could have
written the compiler and all the libraries. Otherwise, it's going to
break and you won't know why, although following magic "design
patterns" from your local cargo cult leader may help shield you from
the wrath of the compiler for another day, if you're lucky and turn
widdershins twice every tuesday before noon but after having the
_right_ cup of coffee while wearing lucky socks.

C++ saw scripting languages and tried to ape their features
(Exceptions!) but doing dynamic typing at compile time is every bit as
stupid as doing dynamic memory management at compile time, and their
attempt (templates) is TURING COMPLETE AT COMPILE TIME meaning you can
write 10 lines of C++ that will keep the compiler busy until your hard
drive fills up, and detecting this is equivalent to solving the halting
problem. Even when it does NOT do that, a couple lines of C++ template
making your binary ten times larger is considered _normal_.

Note: Java is also in the no man's land between C and scripting
languages, but it's in the foothills of scripting languages instead of
the foothills of C: it did dynamic memory management but _kept_ static
typing, then realized how dumb taht was and punched holes in its type
system with "interfaces", and then made code generators to spit out
reams of interface code and designed new tools (Eclipse!) to handle
multi-million line code bases for 2 year old projects made by 3 people.
Alas, when Y2K happened and all that Cobol needed to be rewritten Java
was the hot new fad (Pogs! Beanie Babies! Bitcoin!) and looked good in
comparison to cobol, so it's the new mainframe punchcard language. Oh
well.

Steve Yegge eviscerated Java so I don't have to here:
http://steve-yegge.blogspot.com/2007/12/codes-worst-enemy.html

So back to the "new generation of system languages": C is a portable
assembly language. It's a category killer in that niche, the best there
is at what it does that's already killed off competitors like Pascal.
The only real survivors are derivatives of C whose main selling point
is that they CONTAIN THE WHOLE OF C, VERBATIM. (By that logic a mud pie
is a good beverage, because each mud pie contains a glass of water.)

Scripting langauges (even ugly ones like Javascript, Perl, and PHP)
rely on opaque abstractions independent of what the hardware is doing.
Java is the new Cobol.

Which direction is your new system language going in?
...
Post by Luca Barbato
Post by Rich Felker
I'm aware some people like strl* and we have them in musl because
it's
Post by Rich Felker
better to provide them than to deal with people rolling their own
and
Post by Rich Felker
doing it in wrong and insecure ways. But I still would recommend
against using them in new code.
Toybox has xstrncpy(): If the string doesn't fit in the buffer, kill
the program with an error message.

Rob
Strake
2013-04-23 19:07:26 UTC
Permalink
[Java] did dynamic memory management but _kept_ static
typing, then realized how dumb taht was
So they realized that it's a damn good idea, but somehow made a crap
language nevertheless.

Haskell, for example, does dynamic memory allocation with static
types, and quite often wins.
So back to the "new generation of system languages": C is a portable
assembly language. It's a category killer in that niche, the best there
is at what it does that's already killed off competitors like Pascal.
So on that note, I deem Haskell would be a categorical category killer (^_^)

Cheers,
Strake
Daniel Cegiełka
2013-04-23 19:24:57 UTC
Permalink
Post by Strake
So on that note, I deem Haskell would be a categorical category killer (^_^)
Cheers,
Strake
Haskell and musl - has anyone tried this combination? :) GHC is a
pretty big package.
Daniel
Szabolcs Nagy
2013-04-23 21:33:46 UTC
Permalink
Post by Daniel Cegiełka
Post by Strake
So on that note, I deem Haskell would be a categorical category killer (^_^)
Haskell and musl - has anyone tried this combination? :) GHC is a
pretty big package.
as far as i know haskell has no well defined semantics
about its interaction with the c runtime eventhough it
has an ffi that can use c libraries directly
(eg how haskell managed threads interact with c threads)

(the same is true for most high level languages that also
try to do system level things)

it would be nice to look at these issues systematically

in c it's reasonably clear what the runtime does and what
it doesnt do and how the application can interact with
the system through the c runtime

in a high level language you interact with the system
through high level abstractions (implemented by the
language runtime that almost always goes through the
c runtime) *and* through the c runtime when external
c libraries are used which interact with the system
as well

if you only go through the high level language runtime
then in theory it can shield you from ugly details of
the system (in practice it's not so: you can call
exec, fork, etc from python/php/perl.. and they dont
fully protect the abstractions of the language:
destructors are not called when you exit with
os.exec('/bin/true'))

when the high level language runtime is mixed with the
c runtime through ffi that's a problem: you need
'c level' guarantees about the implementation of the
language runtime and those are the things language
designers want to hide from you
Zvi Gilboa
2013-04-24 12:12:47 UTC
Permalink
Post by Daniel Cegiełka
Post by Strake
So on that note, I deem Haskell would be a categorical category killer (^_^)
Haskell and musl - has anyone tried this combination? :) GHC is a
pretty big package.
One aspect of language binding that often goes under the radar concerns
the method to create public API header files. For my own projects, I use
a simple PostgreSql database that stores the definitions of constants,
structures, functions, and also the public API header-tree, and then a
few shell/psql scripts to generate the entire set of public API headers.
As a matter of convenience, I normally enter the information into plain
text files, which are then processed from the command line and populate
the tables of the above database. In my experience, that kind of
framework not only makes it easy to add binding for new languages, but
also simplifies book-keeping tasks such as the splitting or joining of
libraries, API consistency checks, etc.
Luca Barbato
2013-04-23 21:34:59 UTC
Permalink
Post by Rob Landley
What new crop of system languages?
I could point Go and Rust as two that are trying too much for my taste.

There are few other that try to add just small syntactic features to C
(e.g. blocks/anonymous function) to make the language a bit more
expressive and yet compile to something abi/api compatible with C.
Post by Rob Landley
Toybox has xstrncpy(): If the string doesn't fit in the buffer, kill the
program with an error message.
That's fine for a program, a horrid sin for a library.
Post by Rob Landley
Rob
Daniel Cegiełka
2013-04-24 11:18:43 UTC
Permalink
Post by Luca Barbato
Post by Rob Landley
What new crop of system languages?
I could point Go and Rust as two that are trying too much for my taste.
rust - ok, but written in c++; llvm as dependency.

go - nice, only ed as a dependency but really stupid build system:

https://code.google.com/p/go/source/browse#hg%2Fsrc%2Fcmd%2Fdist

https://code.google.com/p/go/source/browse/src/cmd/dist/build.c

btw. has anyone used go with musl?

Daniel
Kurt H Maier
2013-04-24 11:48:52 UTC
Permalink
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on. It's
also not suitable as a system programming language and they dropped that
claim from their propaganda some time ago.

khm
Daniel Cegiełka
2013-04-24 12:32:21 UTC
Permalink
Post by Kurt H Maier
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on.
lib9, but are you sure about that?

# ldd /usr/bin/go
linux-vdso.so.1 => (0x00007fff50fff000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff215e4b000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff215abf000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff216068000)

Daniel
Rich Felker
2013-04-24 13:38:20 UTC
Permalink
Post by Daniel Cegiełka
Post by Kurt H Maier
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on.
lib9, but are you sure about that?
# ldd /usr/bin/go
linux-vdso.so.1 => (0x00007fff50fff000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff215e4b000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff215abf000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff216068000)
Is the go binary written in go? The point was that go links programs
against its own libc, not that the go compiler is linked against its
own libc.

Rich
Daniel Cegiełka
2013-04-24 13:55:25 UTC
Permalink
Post by Rich Felker
Post by Daniel Cegiełka
Post by Kurt H Maier
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on.
lib9, but are you sure about that?
# ldd /usr/bin/go
linux-vdso.so.1 => (0x00007fff50fff000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff215e4b000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff215abf000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff216068000)
Is the go binary written in go?
Good question! :) If so, it should be statically linked.

http://code.google.com/p/go/source/browse#hg%2Fsrc%2Fcmd%2Fgo
Post by Rich Felker
The point was that go links programs
against its own libc, not that the go compiler is linked against its
own libc.
I'm not sure if they really use __only__ their own libc (lib9). In my
opinion lib9 seems to refer to libc.

Daniel
Post by Rich Felker
Rich
John Spencer
2013-04-24 13:37:36 UTC
Permalink
Post by Kurt H Maier
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on. It's
also not suitable as a system programming language and they dropped that
claim from their propaganda some time ago.
correct, the go runtime is *very* heavy, and it's always linked statically.
this adds ~ 1.5MB to any binary (at least on x86_64).
that's about equivalent to the bloat imposed by the C++ stdlib.

on the suckless page, there's something written about plans to migrate
the coreutils functionality to go, this seems like an insane plan if
even dead-simple tools like cat will eat 1.5 MB of your RAM and storage
space.
Rich Felker
2013-04-24 13:39:48 UTC
Permalink
Post by John Spencer
Post by Kurt H Maier
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on. It's
also not suitable as a system programming language and they dropped that
claim from their propaganda some time ago.
correct, the go runtime is *very* heavy, and it's always linked statically.
this adds ~ 1.5MB to any binary (at least on x86_64).
that's about equivalent to the bloat imposed by the C++ stdlib.
on the suckless page, there's something written about plans to
migrate the coreutils functionality to go, this seems like an insane
plan if even dead-simple tools like cat will eat 1.5 MB of your RAM
and storage space.
Storage space, yes. RAM, no. It's read-only mapping so it doesn't
consume commit charge, and it won't consume physical memory either
except for the parts that get paged in. In reality it might still use
less RAM than a dynamic-linked program, especially one linked with
glibc or with multiple shared libraries.

Rich
Kurt H Maier
2013-04-24 16:33:56 UTC
Permalink
Post by John Spencer
correct, the go runtime is *very* heavy, and it's always linked statically.
this adds ~ 1.5MB to any binary (at least on x86_64).
that's about equivalent to the bloat imposed by the C++ stdlib.
on the suckless page, there's something written about plans to migrate
the coreutils functionality to go, this seems like an insane plan if
even dead-simple tools like cat will eat 1.5 MB of your RAM and storage
space.
Without getting into why none of that is true, the primary problem with
go as a systems language is the memory management.

khm
Szabolcs Nagy
2013-04-24 15:47:26 UTC
Permalink
Post by Kurt H Maier
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on. It's
also not suitable as a system programming language and they dropped that
claim from their propaganda some time ago.
go has its own independent world (own toolchain, syscall wrappers,
runtime, calling convention, stack management etc) but it can interact
with libc through cgo

so the question might be if anyone has tried cgo with musl
and i guess nobody tried but it should work since cgo does
not make much assumptions about the c runtime

go is special in this respect, most other language runtime
implementations build on top of libc so the interaction
between c and said language is less trivial

(there are some caveats in go as well: it does not call
__libc_start_main on startup nor exit on exit so eg atexit
handlers wont get called)
Rich Felker
2013-04-24 19:17:35 UTC
Permalink
Post by Szabolcs Nagy
Post by Kurt H Maier
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on. It's
also not suitable as a system programming language and they dropped that
claim from their propaganda some time ago.
go has its own independent world (own toolchain, syscall wrappers,
runtime, calling convention, stack management etc) but it can interact
with libc through cgo
so the question might be if anyone has tried cgo with musl
and i guess nobody tried but it should work since cgo does
not make much assumptions about the c runtime
go is special in this respect, most other language runtime
implementations build on top of libc so the interaction
between c and said language is less trivial
(there are some caveats in go as well: it does not call
__libc_start_main on startup nor exit on exit so eg atexit
handlers wont get called)
The idea of calling functions in libc without __libc_start_main ever
having been called sounds highly misguided and potentially dangerous.
In musl it might mostly work, but with glibc I don't see how it could
possibly work.

Rich
Szabolcs Nagy
2013-04-25 06:40:58 UTC
Permalink
Post by Rich Felker
Post by Szabolcs Nagy
(there are some caveats in go as well: it does not call
__libc_start_main on startup nor exit on exit so eg atexit
handlers wont get called)
The idea of calling functions in libc without __libc_start_main ever
having been called sounds highly misguided and potentially dangerous.
In musl it might mostly work, but with glibc I don't see how it could
possibly work.
it works because _init is called by the loader before the entry point
and that does enough setup

(when cgo is used then the interpreter is set to /lib/ld-linux.so.2
in the elf header and it calls _init when it loads libc which is
listed in the dynamic section

the entry point is the go runtime which checks if it is in cgo mode
and sets up a separate pthread with libc managed stack and runs c
code there

the elf binary is prepared by the go toolchain, the go world is
statically linked including the wrapper, but there is an object
in the wrapper that is compiled with gcc and references the
extern symbols and uses the c abi, with a trick the go toolchain
knows about all the extern symbols and needed libraries which get
into the dynamic section so the c world is dynamically linked

there are some other runtime details to allow calling from go
into c and then calling back to go from c, bridging the difference
in abi and calling convenion, but basically this is how cgo works)
Rob Landley
2013-04-25 19:37:36 UTC
Permalink
Post by Kurt H Maier
Post by Daniel Cegiełka
btw. has anyone used go with musl?
Go ships its own libc, which I'm fairly certain it depends on. It's
also not suitable as a system programming language and they dropped that
claim from their propaganda some time ago.
Ken Thompson is designing go, but Dennis Ritchie was really the guy who
made C into a systems language (and kept pestering Ken to try doing
bits of unix kernel in his latest version, then took the resulting
feedback and implemented solutions for it).

For the curious, here's the paper from Dennis Ritchie's HOPL II talk:

http://cm.bell-labs.com/cm/cs/who/dmr/chist.html

Rob
John Spencer
2013-04-24 13:28:48 UTC
Permalink
Post by Daniel Cegiełka
btw. has anyone used go with musl?
i tried to build gcc 4.7.2 with go support (--enable-languages=c,c++,go)
and that fails due to a lack of set/getcontext().
(see pkg/gcc472 in sabotage)

according to rich, adding that to musl requires a non-trivial amount of
arch specific asm.

the go runtime in the gcc tree should be fixed to have a fallback when
this functionality is missing (if possible),
so it maybe be needed to ask on the go mailing list.
Rich Felker
2013-04-24 13:42:44 UTC
Permalink
Post by John Spencer
Post by Daniel Cegiełka
btw. has anyone used go with musl?
i tried to build gcc 4.7.2 with go support (--enable-languages=c,c++,go)
and that fails due to a lack of set/getcontext().
(see pkg/gcc472 in sabotage)
according to rich, adding that to musl requires a non-trivial amount
of arch specific asm.
Yes, but it is a wanted feature, so I wouldn't mind it getting done.
It was even part of the standard prior to POSIX 2008, and the reason
for removing it was stupid. (The reason was that the makecontext
function's calling convention is bogus and impossible to support
properly, but they could have fixed this by deprecating the use of the
variadic arguments in any way except passing a single void* argument,
rather than deprecating the whole set of interfaces.)
Post by John Spencer
the go runtime in the gcc tree should be fixed to have a fallback
when this functionality is missing (if possible),
so it maybe be needed to ask on the go mailing list.
The only fallback is to use C11 or POSIX threads in place of
coroutines, or shipping their own set/getcontext code for each arch...

Rich
Christian Neukirchen
2013-04-24 14:06:33 UTC
Permalink
Post by Daniel Cegiełka
Post by Luca Barbato
Post by Rob Landley
What new crop of system languages?
I could point Go and Rust as two that are trying too much for my taste.
rust - ok, but written in c++; llvm as dependency.
If "stupid" means "not clever", yes. That's a feature.
--
Christian Neukirchen <***@gmail.com> http://chneukirchen.org
Daniel Cegiełka
2013-04-29 11:41:46 UTC
Permalink
Post by Christian Neukirchen
If "stupid" means "not clever", yes. That's a feature.
it's more then stupid.

1) __hardcoded__ glibc loader (!!!)

https://code.google.com/p/go/source/browse/src/cmd/6l/asm.c
https://code.google.com/p/go/source/browse/src/cmd/8l/asm.c

you can run go (force musl libc) with command:

/lib/libc.so /bin/go (...)

2) hardcoded bison (instead ${YACC}):

https://code.google.com/p/go/source/browse/src/cmd/gc/Makefile

3) hardcoded bash in build scripts (instead portable sh).

4) C based 'build script' instead Makefile.

btw. Go (Google) works with musl (sabotage).

Daniel
Post by Christian Neukirchen
--
John Spencer
2013-04-29 16:31:24 UTC
Permalink
Post by Daniel Cegiełka
btw. Go (Google) works with musl (sabotage).
oh ? which one ? the one that comes with gcc 4.7.2 doesn't.
Daniel Cegiełka
2013-04-29 16:44:00 UTC
Permalink
Post by John Spencer
Post by Daniel Cegiełka
btw. Go (Google) works with musl (sabotage).
oh ? which one ? the one that comes with gcc 4.7.2 doesn't.
http://code.google.com/p/go/

http://code.google.com/p/go/downloads/detail?name=go1.0.3.src.tar.gz&can=2&q=

But I'll have to review the code to clean up the hardcoded glibc
loader, bash, bison etc.

Best regards,
Daniel

Rob Landley
2013-04-23 00:31:03 UTC
Permalink
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
- Thread allergies, i.e. horribly over-complicating program logic
to
...
Post by Rich Felker
zlib, etc.). Then, your main event loop just sees a normal file
descriptor, so you don't have to invade your code that handles
streams/connections/whatever with a new abstraction framework around
file descriptors that also supports SSL or whatever. Not only are such
abstraction layers bloated; they also add a lot of places for bugs to
...
Post by Rich Felker
possible (e.g. a webserver). But for something like an IRC client or
web browser, where socket performance is irrelevant compared to other
things, it's idiotic to design fancy abstraction layers for reading
and writing to connections when you could just do a design like what I
described above.
If I might suggest: the general problem is unnecessary abstraction
layers that hide the details of what you're doing from _yourself_, and
which perform unnecessary work translating between your program and
itself. (Let's marshall data into structures! Let's copy it back out
again into _another_ structure! Let's do that five times in the same
darn call chain! Bonus points for passing a hardwired constant as the
only caller of a function that then checks that input for illegal
values!)

It's nice to pull in a packaged solution that automates away fiddly
bits and tedium with tested and working code (which is basically what
libc is), but pulling in 8 megabytes of shared library to beep the
speaker instead of writing to a /proc file? Very common, and not an
improvement. (Random current example: why does qemu need an extended
attribute library to build the virtfs server? It doesn't pull in a
libchmod.so to set file ownership and permissions...)
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
- DBus.
Sadly nobody is pushing for a better local socket multicast
abstraction
Post by Luca Barbato
to send notifications back and forth in an efficient fashion.
When I say that it always winds up being my job. (Belling the cat
protocol.)
Post by Rich Felker
Post by Luca Barbato
I'm hoping for nanomsg once it is complete or Binder once it is
correctly documented ^^; (and thus implemented in more than few
forks of
Post by Luca Barbato
linux and maybe haiku)
Yes, Binder looks really promising, but I also think part of the
problem is that the need for this kind of interface is exaggerated...
Is anybody collecting patches to remove it from packages that currently
require it? If it's easy to Not Do That, it should be possible to
demonstrate...
Post by Rich Felker
Post by Luca Barbato
Post by Rich Felker
- Use of global state. Even seemingly-harmless things like a
global
Post by Luca Barbato
Post by Rich Felker
registered log function are harmful, because two different
libraries
Post by Luca Barbato
Post by Rich Felker
(or the main program and a library) might be trying to use the
library with the global log destination, and clobbering each
other's
Post by Luca Barbato
Post by Rich Felker
choices.
For this there aren't solution that won't cause different problems
I'm
Post by Luca Barbato
afraid.
Sure there are. I get the impression you can tell I was talking about
libav/ffmpeg's log interface. :-) The obvious solution is to bind log
http://misko.hevery.com/2008/11/21/clean-code-talks-global-state-and-singletons/
Was that the java one?
Post by Rich Felker
If I remember right, part of the problem way back was that there were
deep function calls that had no context available to them, and that
didn't _need_ a context for anything but logging warnings or whatnot.
Really, the fact that they can fail and want to be able to report
failure means they _do_ need a context, but I can understand the
desire to cheat, especially if there's a performance cost to passing
the context all the way down. In that case, hidden non-global state,
namely thread-local storage, might be an appropriate solution. It's
still hidden state (and thus A Bad Thing), but at least it's no longer
global state.
There are contexts between global and local. There have always been
contexts between global and local. This is why objects were invented,
to provide a better syntax for contexts between global and local.

But another issue is visibility of contexts. For example, if you put
stuff in environment variables, you can view and edit the environment
variables. So if you want it to stop doing something, or see whether or
not it's doing something, it's easy to dump state with any number of
tools.

Unix had reasonable solutions for this stuff in the 1970's. Java
reinvented the wheel and thought "hexagonal" was a good idea, and it
_sort_ of works, and they've been filing bits off and gluing other bits
on ever since. (Alas, mostly they've just been making their wheel
_thicker_. Oh well.)

Rob
Continue reading on narkive:
Loading...