Planet Gentoo

Quite some weeks ago, Markos stated asking for a more thorough testing of ebuilds to check whether they support LDFLAGS; he then described why that is important which can be summarised in “if LDFLAGS are ignored, setting --as-needed by default is pointless”. He’s right on this of course, but there are a few more tricks to consider.

You might remember that I described an alternative approach to --as-needed that I call “forced --as-needed”, by editing the GCC specifications. The idea is that by forcing the flag in through this method, packages ignoring LDFLAGS and packages using unpatched libtool won’t simply ignore it. This was one of the things I had in mind when I suggested that approach, but alas, as it happens, I wasn’t listened to.

Right now there are 768 bugs reported blocking the tracker of which 404 are open and, in the total, 635 were reported by me (mostly through the tinderbox but not limited to). And I’m still opening bugs on a daily basis for those.

But let’s say that these are all the problems there are, and that in two weeks time no package is reporting ignored LDFLAGS at all. Will that mean that all packages will properly build with --as-needed at that point? Not really.

Unfortunately, LDFLAGS-ignored warnings have both false positives (prebuild binary packages, packages linking with non-GNU linkers) and false partially-negatives. To understand that you have to understand how the current warning works. A few binutils versions ago, a new option was introduced, called --hash-style; this option is used to change the way the symbol table is hashed for faster loading by the runtime linker/loader (ld.so); the classic hashing algorithm (SysV) is not excessively good for very long, similar symbols that are way too common when using C++, so there has been some work to replace that with a better GNU-specific algorithm. I’ve followed most of the related development closely at the time, since Michael Meeks (of OpenOffice.org fame) actually came asking Gentoo for some help testing things out; it was that work that actually got me interested in linkers and the ELF format.

At any rate, while the original hash table was created in the .hash section, the new hash table, being incompatible, is in .gnu.hash. The original patch only replaced one with the other, but what actually landed in binutils was slightly different (it allows to choose between SysV, GNU, or both hash tables), and the default has been to enable both hash tables, so that older systems can load .hash and the new ones can load .gnu.hash; win win. On the other hand, on Gentoo (and many other distributions) where excessively old GLIBC versions are not supported at all, there is enough of a reason to use --hash-style=gnu to disable the generation of the SysV table entirely.

Now, the Portage warning is derived by this situation: if you add -Wl,--hash-style=gnu to your LDFLAGS, it will be checking the generated ELF files and warn if it finds the SysV .hash section. Unfortunately this does not work for non-Linux profiles (as far as I know, FreeBSD does not support the GNU hashing style yet; uClibc does), and will obviously report all the prebuilt binaries coming from proprietary packages. In those cases, you don’t want to strip .hash because it might be the only hash table preventing ld.so from doing a linear search.

So, what is the problem with this test? Well, let’s note one big difference between --as-needed and --hash-style flags: the former is positional, the second is not. That means that --as-needed, to work as intended, needs to be added before the actual libraries are listed, while --hash-style can come at any point in the command line. Unfortunately this means that if any package has a linking line such as

It won’t be hitting the LDFLAGS warning from Portage, but (basic) --as-needed would fail — OTOH, my forced --as-needed method would work just fine. So there is definitely enough work to do here for the next… years?

Some of you might already know that I am moving to Manchester/UK this year ( at least ) to attend a MSc course at the University of Manchester. The Computer Science Department offers a variety of courses hance I am not sure which pathway I am gonna choose yet. The Computer System Engineering MSc looks rather good but I will have to wait for the 1st week of lectures before I make my choice.  In any case, moving to UK is quite an experience, and I am really excited about that :)

Even though the motivation about Gentoo development is anything but high, I was quite worried about the fact that I couldn’t bring my two “dev” machines[1][2] with me so all I have is my laptop and my netbook, but lets face it, you can’t do much development on such hardware :-/.  I could easily forward a dozen of ports on my router in order to remotely administer all my services but, when it comes to security, this is a serious drawback. This really left me with one choice: Openvpn :).  Thanks to this guide, the whole setup was easier than I originally thought. Now, I am able to access my local network from anywhere. Furthermore, tmux[3] makes the whole /remote/development process much easier :). What a great app \o/

Just for stable amd64 keywording and Server services

[2] Linux Mystical 2.6.35-zen2-dark-jedi #1

This is actually a *good* dev machine. The 6-core power combined with 4GB@1066 RAM

is an ideal platform for extensive development and compilation 24/7



* dev-db/mysql-5.1.50-r1: Merge time 2 hours, 7 minutes, 45 seconds ( tests enabled )

* sys-devel/gcc-4.4.4-r1: Merge time 15 minutes, 36 seconds

* sys-libs/glibc-2.12.1-r1: Merge time 11 minutes, 5 seconds

This devbox hosts 6 chroors to test all the Qt4 live versions and one  1 gcc-4.5.X chroot

[3]

• Blog this on Blogger
• Subscribe to the comments for this post?
• Share this on del.icio.us
• Share this on Facebook
• Email this via Gmail
• Add this to Google Reader
• Post this to Identica
• Tweet This!

A foreword: some people might think that I’m writing this just to banter about what I did; my sincere reason to write, though, is to point out an example of why I dislike 5-minutes fixes as I wrote last December. It’s also an almost complete analysis of my process of ebuild maintenance so it might be interesting for others to read.

For a series of reasons that I haven’t really written about at all, I need Quagga in my homemade Celeron router running Gentoo — for those who don’t know, Quagga is a fork of an older project called Zebra, and provides a few daemons for route advertisement protocols (such as RIP and BGP). Before yesterday, the last version of Quagga in Portage was 0.99.15 (and the stable is an old 0.98 still), but there was recently a security bug that required a bump to 0.99.17.

I was already planning on getting Quagga a bump to fix a couple of personal pet peeves with it on the router; since Alin doesn’t have much time, and also doesn’t use Quagga himself, I’ve added myself to the package’s metadata; and started polishing the ebuild and its support files. The alternative would have been for someone to just pick up the 0.99.15 ebuild, update the patch references, and push it out with the 0.99.17 version, which would have categorized for a 5-minutes-fix and wouldn’t have solved a few more problems the ebuild had.

Now, the ebuild (and especially the init scripts) make a point that they were contributed by someone working for a company that used Quagga; this is a good start, from one point: the code is supposed to work since it was used; on the other hand companies don’t usually care for the Gentoo practices and policies, and tend to write ebuilds that could be polished a bit further to actually be compliant to our guidelines. I like them as a starting point, and I got used to do the final touches in those cases. So if you have some ebuilds that you use internally and don’t want to spend time maintaining it forever, you can also hire me to clean them up and merge in tree.

So I started from the patches; the ebuild applied patches from a tarball, three unconditionally and two based on USE flags; both of those had URLs tied to them that pointed out that they were unofficial feature patches (a lot of networking software tend to have similar patches). I set out to check the patches; one was changing the detection of PCRE; one was obviously a fix for --as-needed, one was a fix for an upstream bug. All five of them were on a separate patchset tarball that had to be fetched from the mirrors. I decided to change the situation.

First of all, I checked the PCRE patch; actually the whole PCRE logic, inside configure is long winded and difficult to grok properly; on the other hand, a few comments and the code itself shows that the libpcreposix library is only needed non non-GNU systems, as GLIBC provides the regcomp/regexec functions. So instead of applying the patch and have a pcre USE flag, I changed to link the use or not of PCRE depending on the elibc_glibc implicit USE flag; one less patch to apply.

Second patch I looked at was the --as-needed-related patch that changed the order of libraries link so that the linker wouldn’t drop them out; it wasn’t actually as complete as I would have made. Since libtool handles transitive dependencies fine, if the libcap library is used in the convenience library, it only has to be listed there, not also in the final installed library. Also, I like to take a chance to remove unused definitions in the Makefile while I’m there. So I reworked the patch on top of the current master branch in their GIT, and sent it upstream hoping to get it merged before next release.

The third patch is a fix for an upstream bug that hasn’t been merged in a few releases already, so I kept it basically the same. The two feature patches had new versions released, and the Gentoo version seems to have gone out of sync with the upstream ones a bit; for the sake of reducing Gentoo-specific files and process, I decided to move to use the feature patches that the original authors release; since they are only needed when their USE flags are enabled, they are fetched from the original websites conditionally. The remaining patches are too small to be part of a patchset tarball, so I first simply put them in files/ are they were, with mine a straight export from GIT. Thinking about it a bit more, I decided today to combine them in a single file, and just properly handle them on Gentoo GIT (I started writing a post detailing how I manage GIT-based patches).

Patches done, the next step is clearing out the configuration of the program itself; the ipv6 USE flag handles the build and installation of a few extra specific daemons for for the IPv6 protocol; the rest are more or less direct mappings from the remaining flags. For some reason, the ebuild used --libdir to change the installation directory of the libraries, and then later installed an env.d file to set the linker search path; which is generally a bad idea — I guess the intention was just to follow that advice, and not push non-generic libraries into the base directory, but doing it that way is mostly pointless. Note to self: write about how to properly handle internal libraries. My first choice was to see if libtool set rpath properly, and in that case leave it to the loader to deal with it. Unfortunately it seems like there is something bad in libtool, and while rpath worked on my workstation, it didn’t work on the cross-build root for the router though; I’m afraid it’s related to the lib vs lib64 paths, sigh. So after testing it out on the production router, I ended up revbumping the ebuild already to unhack it — if libtool can handle it properly, I’ll get that fixed upstream so that the library is always installed, by default, as a package-internal library, in the mean time it gets installed vanilla as upstream wrote it. It makes even more sense given that there are headers installed that suggest the library is not an internal library after all.

In general, I found the build system of quagga really messed up and in need of an update; since I know how many projects are sloppy about build systems, I’d probably take a look. But sincerely, before that I have to finish what I started with util-linux!

While I was at it, I fixed the installation to use the more common emake DESTDIR= rather than the older einstall (which means that it now installs in parallel as well); and installed the sample files among the documentation rather than in /etc (reasoning: I don’t want to backup sample files, nor I want to copy them to the router, and it’s easier to move them away directly). I forgot the first time around to remove the .la files, but I did so afterwards.

What remains is the most important stuff actually; the init scripts! Following my own suggestions the scripts had to be mostly rewritten from scratch; this actually was also needed because the previous scripts had a non-Gentoo copyright owner and I wanted to avoid that. Also, there were something like five almost identical init scripts in the package, where almost is due to the name of the service itself; this means also that there had to be more than one file without any real reason. My solution is to have a single file for all of them, and symlink the remaining ones to that one; the SVCNAME variable is going to define the name of the binary to start up. The one script that differs from the other, zebra (it has some extra code to flush the routes) I also rewrote to minimise the differences between the two (this is good for compression, if not for deduplication). The new scripts also take care of creating the /var/run directory if it doesn’t exist already, which solves a lot of trouble.

Now, as I said I committed the first version trying it locally, and then revbumped it last night after trying it on production; I reworked that a bit harder; beside the change in libraries install, I decided to add a readline USE flag rather than force the readline dependency (there really isn’t much readline-capable on my router, since it’s barely supposed to have me connected), this also shown me that the PAM dependency was strictly related to the vtysh optional component; and while I looked at PAM, (Updated) I actually broke it (and fixed it back in r2); the code is calling pam_start() with a capital-case “Quagga” string; but Linux-PAM puts it in all lower case… I didn’t know that, and I was actually quite sure that it was case sensitive. Turns out that OpenPAM is case-sensitive, Linux-PAM is not; that explains why it works with one but not the other. I guess the next step in my list of things to do is check out if it might be broken with Turkish locale. (End of update)

Another thing that I noticed there is that by default Quagga has been building itself as a Position Independent Executable (PIE); as I have written before using PIE on a standard kernel, without strong ASLR, has very few advantages, and enough disadvantages that I don’t really like to have it around; so for now it’s simply disabled; since we do support proper flags passing, if you’re building a PIE-complete system you’re free to; and if you’re building an embedded-enough system, you have nothing else to do.

The result is a pretty slick ebuild, at least in my opinion, less files installed, smaller, Gentoo-copyrighted (I rewrote the scripts practically entirely). It handles the security issue but also another bunch of “minor” issues, it is closer to upstream and it has a maintainer that’s going to make sure that the future releases will have an even slicker build system. It’s nothing exceptional, mind you, but it’s what it is to fix an ebuild properly after a few years spent with bump-renames. See?

Afterword: a few people, seemingly stirred up by a certain other developer, seems to have started complaining that I “write too much”, or pretend that I actually have an uptake about writing here. The main uptake I have is not having to repeat myself over and over to different people. Writing posts cost me time, and keeping the blog running, reachable and so on so forth takes me time and money, and running the tinderbox costs me money. Am I complaining? Not so much; Flattr is helping, but trust me that it doesn’t even cover the costs of the hosting, up to now. I’m just not really keen on the slandering because I write out explanation of what I do and why. So from now on, you bother me? Your comments will be deleted. Full stop.

A Chromium security stabilisation requires Cups 1.4 stable, which gives architecture teams some headaches. It is a major update and several stuff can break, tinderboxing it by building packages that depend on it is one side of the task, but maybe something breaks on runtime. On my own systems CUPS 1.4 runs fine for some months in mostly stable environments, but maybe there are more issues out there that we need to be aware of.  Please point out any issues that might be there on stable systems (maybe upgrade your stable system right now and tell us).

Hi folks,

last week I have been attending to the LinuxCon Brazil 2010, that was the first edition of this event here in Brazil.

All the talks were pretty good and the event was a great oportunity to meet nice people, good projects and learn some cool stuff (and get an autograph of Linus Torvalds as well :)

Unfortunately the Gentoo presence in the event was null. Well, not really null because I was there, but it seems that the Brazilian users are not very interested in Gentoo and I'm not sure if someone will be able to change this some day.

Maybe I'll write more stuff about the LinuxCon soon.

Stay tuned! ;)

I have written last week of the good feeling of merging patches upstream – even though since then I don’t think I got anything else merged … well, beside the bti fixes that I sent Greg – this week, let’s start with the opposite problem: how can you handle backports sanely, and have a quick way to check what was merged upstream? Well, the answer, at least for the software that is managed upstream with GIT, is quite easy to me.

Note: yes this is a more comprehensive rehashing of what I posted last December so if you’ve been following my blog for a long time you might not be extremely surprised by the content.

So let’s start with two ideas: branches and tags; for my system to work out properly, you need upstream to have tagged their releases properly; so if the foobar project just released version 1.2.3, we need to have a tag available that is called foobar-1.2.3, v1.2.3, or something along these lines. From that, we’ll start out a new “scratch branch”; it is important to note that it’s a scratch branch, because it means that it can be force-pushed and might require a complete new checkout to work properly. So we have something like the following:

This gives us the 1.2.3-gentoo branch as the scratch branch, and we’ll see how that behave in a moment. If upstream fails to provide tags you can also try to track down which exact commit a release corresponds to – it is tricky but not unfeasible – and replace origin/v1.2.3 with the actual SHA hash of the commit or, even better as you’ll guess by the end of the post, tag it yourself.

The idea of using a scratch branch, rather than an actual “gentoo branch” is mostly out of simplicity to me; most of the time, I make more than a couple of changes to a project if I’m packaging it – mostly because I find it easier to just fix possible autotools minor issues before they actually spread throughout the package and other packages as well – but just the actual fixes I want to apply to the packaged version; cleanups, improvements and optimisations I send upstream and wait for the next release. I didn’t always do it this way, I admit.. I changed my opinion when I started maintaining too many packages to follow all of them individually. For this reason I usually have either a personal or a “gentoo” branch where I make changes to apply to master branch, which get sent upstream and merged, and a scratch branch to handle patches. It also makes it no different to add a custom patch or a backport to a specific version (do note, I’ll try to use the word “backport” whenever possible to stress the important of getting the stuff merged upstream so that it will be present in the future, hopefully).

So we know that in the upstream repository there have been a few commits to fix corner case crashers that, incidentally, seem to always apply on Gentoo (don’t laugh, it happens more often than you can think). The commits have the shorthashes 1111111 2222222 3333333 — I have no fantasy for hashes, so sue me.

Now you have a branch with three commits, cherry-picked copies (with different hashes) of the commits you need. At this point, what I usually do, is tagging the current state (and in a few paragraphs you’ll understand why), so that we can get the data out properly; at this point, the way you name the tag depends vastly on how you will release the backport, so I’ll get to that right away.

The most common way to apply patches in Gentoo, for good or bad, is adding them to the files/ subdirectory of a package; to be honest this is my least preferred way unless they are really trivial stuff, because it means that the patches will be sent down the mirrors to all users, no matter whether they use the software or not; also, given the fact that you can use GIT for patch storage and versioning, it’s also duplicating the effort. With GIT-stored patches, it’s usually the easiest to create a files/${PV}/ subdirectory and store there the patches as exported by git format-patch — easy, yes; nice nope: given that, as I’ll say, you’ll be picking the patches again when a new version is released, they’ll always have different hashes, and thus the files will always differ, even if the patch itself is the same patch. This not only wastes time, it makes it non-deduplicable and also gets around the duplicated-files check. D’oh!

A more intelligent way to handle these trivial patches is to use a single, combined patch; while patchutils has a way to combine patches, it’s not really smart; on the other hand, GIT, like most other source control managers, can provide you with diffs between arbitrary points in the repository’s history… you can thus use git diff to export a combined, complete patch in a single file (though lacking history, attribution and explanation). This helps quite a lot when you have a few, or a number, of very small patches, one or two hunks each, that would cause too much overhead in the tree. Combining this way bigger patches can also work, but you’re more likely to compress it and upload it to the mirrors, or to some storage area and add it to SRC_URI.

A third alternative, which is also requiring you to have a storage area for extra distfiles, is using a so-called “patchset tarball”; as a lot of packages already do. The downside of this is that if you have a release without any patch tarball at all, it becomes less trivial to deal with it. At any rate, you can just put in a compressed tar archive the files created, once again, by git format-patch; if you add them as a subdirectory such as patches/ you can then use the epatch function from eutils.eclass to apply them sequentially, simply pointing it at the directory. You can then use the EPATCH_EXCLUDE variable to remove one patch without re-rolling the entire tarball.

Note: epatch itself was designed to use a slightly different patchset tarball format, that included the use of a specification of the architecture, or all to apply to all architectures. This was mostly because its first users were the toolchain-related packages, where architecture-dependent patches are very common. On the other hand, using conditional patches is usually discouraged, and mostly frown upon, for the rest of the software. Reason being that’s quite more likely to make a mistake when conditionality is involved; and that’s nothing new since it was the topic of an article I wrote over five years ago.

If you export the patches as multiple files in filesdir/, you’re not really going to have to think much about naming the tag; for both other cases you have multiple options: tie the name to the ebuild release, tie it to the CVS revision indication, and so on. My personal preferred choice is that of using a single incremental, non-version-specific number for patch tarballs and patches, and mix that with the upstream release version in the tag; in the example above, it would be 1.2.3-gentoo+1. This is, though, just a personal preference.

The reason is simple to explain and I hope it makes sense for others than me; if you tie it to the release of the ebuild (i.e. ${PF}), like the Ruby team did before, you end up in trouble when you want to add a build-change-only patch – take for instance the Berkeley DB 5.0 patch; it doesn’t change what is already installed on a system built with 4.8; it only allows to build anew with 5.0; given that, bumping the release in tree is going to waste users’ time – while using the CVS revision will create quite a few jumps (if you use the revision of the ebuild, that is) as many times you change the ebuild without changing the patches. Removing the indication of the upstream version is also useful, albeit rarely, when upstream does not merge any of your patches, and you could simply reuse the same patchset tarball as previous release; it’s something that comes handy especially when security releases are done.

At this point, as a summary you can do something like this:

• mkdir patches; pushd patches; git format-patch v1.2.3..; popd; tar jcf foobar-gentoo-1.tar.bz2 patches — gets you a patchset tarball with the patches (similarly you can prepare split patches to run add to the tree);
• git diff v1.2.3.. > foobar-gentoo-1.patch — creates the complete patch that you can either compress, or upload to mirrors or (if very very little) put it on the tree.

Now, let’s say upstream releases version 1.2.4, and integrates one of our patches. Redoing the patches is quick with GIT as well.

If there are compatible changes, the new patches will be applied just fine, and updated to not apply with fuzz; any patch that was applied already will count as “empty” and will be simply removed from the branch. At that point, you can just reiterate the export as said above.

When pushing to the repository, remember to push explicitly the various gentoo branches, and make sure to push --tags as well. If you’re a Gentoo developer, you can host such repository on git.overlays.gentoo.org host a few of them already; lxc, libvirt, quagga …); probably contributors, even not developers, can ask for similar repositories to be hosted there.

I hope this can help out other developers dealing with GIT-bound upstreams to ease their overweight.

What's new in sys-apps/portage-2.1.9? Here is the portage-2.1.9 section of the release notes:

• The emerge "world" set now includes separate "selected" and "system" sets, where the "selected" set includes packages listed in /var/lib/portage/world.
• Package set names in emerge arguments have to be prefixed with @ (exceptions: 'world' and 'system' can be used without the prefix).
• Configuration files now support atoms with wildcards inside the category and package name parts of the atoms.
• The functionality of the autounmask program is emulated by the new emerge --autounmask option, which outputs required configuration changes for package.accept_keywords and package.use.
• The new emerge --exclude option allows packages to be excluded from the dependency resolution. Doing so might result in a fatal error. See the emerge(1) man page for details.
• Per-package environment variables can be set with the new package.env configuration file in /etc/portage/. See the portage(5) man page for details.
• Support for per-package bashrc files in /etc/portage/env. See the portage(5) man page for details.
• The package.keywords configuration file in /etc/portage/ is now deprecated. Instead use the package.accept_keywords file which has the same format and behavior. See the portage(5) man page for details.
• FEATURES="fixlafiles" (enabled by default): Rewrites newly installed .la files in the same way dev-util/lafilefixer does. Note that this won't fix your installed .la files.

Original post blogged on b2evolution.

Well, I spent half of today rebuilding my system, and upgrading mysql.

I figured that I might use the opportunity of my newly-found spare time while running revdep-rebuild to perhaps put out a general plea for developers to make use of the news feature in portage.

Upgrading to mysql 5.1 requires doing a full dump of your databases, some manual cleanup, an upgrade, and then some manual restore steps. Oh, and that dump has to be done BEFORE the upgrade or you end up having to revert back to 5.0 (which I ended up doing). Usually mysql upgrades are relatively painless, but jumps between major versions (0.1 level) are often not.

The upgrade also breaks anything that links to libmysql, which is quite a bit on a system that runs any number of services (mail, mythtv, ulog, etc).

It might have been nice if a news item were published a day or two before stabilizing mysql 5.1 so that users might have some advance warning and could plan accordingly.

Now, this upgrade didn’t rise to the level of some of the past breakages that actually were very painful to recover from and could result in unbootable systems/etc. Still, it never hurts to give users notice. The beauty of news items is that they only pester users who will actually be impacted by them. I don’t think anybody running mysql would mind a reminder that an upcoming upgrade requires careful planning – this is far more relevant to users than half the stuff we put in elogs/etc.

On the other hand, I do appreciate the mysql upgrade guide on the gentoo website (might not hurt to update it a tiny bit), and Peter Davies’s blog entry from 1.5 years ago was very helpful. If these had been pointed out before stabilizing the build the stable experience would have been a little smoother.

So, at the moment I start writing this (and that’s unlikely to be the time I actually post this, given that I see now it could use some drawings) it’s early in the morning in Italy and I haven’t slept yet – a normal condition for me especially lately – but I have spent a bit bouncing ideas around with ferringb, Ramereth and Craig for what concerns Linux Containers (LXC). Given that, I’d like to point point out a couple of things regarding networking and LXC that might not have been extremely obvious before.

First of all, of the four networking types supported by LXC, I only could try two, for obvious reasons: phys is used to assign a particular physical device to the container, and only works if you have enough physical devices to work with, vlan requires a device able to do vlan tagging. This leaves us with veth (virtual ethernet), and macvlan (mac-address based virtual lan tagging). The former is the most simple setup, and the one I’ve been using; it creates a pair of devices, one of which is assigned within the container, and the other which is assigned to the host; you can then manage that device exactly like any other device you have on your system, and in my case that means it’s added to the br0 bridge where KVM instances are also joined. LXC allows for defining the bridge to join directly in the configuration file.

The macvlan mode is supposed to have smaller overhead because the kernel knows the mac address assigned to the single interfaces beforehand; on the other hand setting it up is slightly harder; in particular, there is one further mode parameter that can be set, in either vepa (Virtual Ethernet Port Aggregator) or bridge mode; the former isolates the container, like they were a number of different hosts connected over to the network segment, but disallows the various containers from talking with one another; on the other hand the latter mode actually creates a special bridge (not to be confused with the Linux bridge used above with virtual ethernet devices), that allows all the containers to talk with one another.. but isolates them from the host system

You end up having to choose between the performance of network-to-container and that of host-to-container: in the first case you can choose macvlan, reducing the work the kernel has to do, but requiring you to route your own traffic to the container with an outside router; in the second case you use veth and make the kernel handle the bridge itself. In my case, since the containers are mostly used for local testing, and the workstation will still be using the in-kernel bridge anyway, the choice is obvious for veth.

Now, when I decided to seal the tinderbox I wondered about one thing, that LXC cannot do and that I would like to find the time to send upstream. As it is, I want to disallow any access from the tinderbox to the outside, minus the access to the RSync service and the non-caching Squid proxy. To achieve that I dropped IPv4 connectivity (so I don’t run any DHCP client at all), and limited myself to autoconfigured IPv6 addresses; then I set in /etc/hosts the static address for yamato.home.flameeyes.eu, and used that as hostname for the two services. Using iptables to firewall the access to any other thing had unfortunate results before (the kernel locked up without actually any panic happening); while I have to investigate that again, I don’t think much changed in that regard. There is no access to the outside network or from the outside network, since the main firewall is set to refuse talking at all with the tinderbox, but that’s not generally a good thing (I would like, at some point in the future, to allow access to the tinderbox to other developers), and does not ensures isolation between that and the other boxes on the network, which is a security risk (remember: the tinderbox builds and execute a lot of code that for me is untrusted).

Now, assuming that the iptables kernel problem happens only with the bridge enabled (I would be surprised if it failed that badly on a pure virtual ethernet device!), my solution was actually kinda easy: I would just have used the link-local IPv6 address, and relied on Yamato as a jump-host to connect to the tinderbox. Unfortunately, while LXC allows you to set a fixed hardware address for the interface created inside the container, it provides you no way to do the same for the host-side interface (which also get a random name such as veth8pUZr), so you cannot simply use iptables to enforce the policy as easily.

But up to this, it’s just a matter of missing configuration interfaces, so it shouldn’t be much of a problem, no? Brian pointed out a chance of safety issue there though, and I went on to check it out. Since when you use virtual ethernet devices it is the kernel’s bridge that takes care of identifying where to send the packages based on STP there is no checking of the hardware address used by the container; just like the IP settings you have there, any root user inside the container can add and remove IP addresses and change the mac address of it altogether. D’uh!

I’m not sure whether this would work better with macvlan, but as it is, there is enough work to be done with the configuration interface, and – over an year after the tinderbox started using LXC to run – it’s still not ready for production use — or at least not for the kind of production use where you actually allow third parties to access a “virtualised” root.

For those interested, the original SVG file of the (bad) network diagrams used in the article, is here and is drawn using Inkscape.

This work by Diego Elio Pettenò is licensed under Creative Commons Attribution-ShareAlike 3.0 Unported License.
Based on a work at blog.flameeyes.eu.
Permissions beyond the scope of this license may be available at http://blog.flameeyes.eu/pages/licensing-terms.

I was looking around for desktop search frameworks today, specifically something with a gtk frontend and that required the fewest resources to run.

I discovered Pinot, a dbus-based file index/monitor/search tool. It even comes with a minimal gtk+ interface. I found few reviews on Pinot, and even fewer recent reviews comparing it to other search frameworks like Strigi, Tracker, and Beagle. I also discovered Catfish, a lightweight frontend to several different search services. There's not much out there on integrating Catfish and Pinot, so I forged ahead and wrote my own code, then did some trial-and-error experiments.

All ebuilds are available on my overnight overlay. Instructions for adding the overlay are on the wiki.

Writing the ebuilds

The only ebuild I found for Pinot is sadly out-of-date, and is completely incorrect. Also, it depends on libtextcat, and I never found an ebuild for that.

So, I wrote my own ebuilds for the latest versions of Pinot and libtextcat.

Not content with Pinot's minimal gtk+ interface, I decided to try Catfish, a PyGtk frontend for several different search engines, including Pinot. Catfish is made by the same developer of Midori, a well-respected lightweight WebKit browser. While Catfish's development has been stalled for two years, I figured it was worth a shot, since its user interface is friendlier than Pinot's.

Catfish, like Pinot and libtextcat, is not in Portage, but there is an open bug for its inclusion. However, the ebuild for the latest version needed updating, as it didn't include Strigi or Pinot. So I rewrote it and added descriptive metadata.xml entries for Catfish's and Pinot's USE flags.

There's still a bit of work left on the Catfish ebuild, since there's a QA warning about not leaving precompiled Python object files in /usr/share/catfish. However, the application itself works perfectly. Just need to clean up the install process so that the bytecode doesn't clutter up the filesystem.

Pinot

On first run, Pinot will take a long time to index your files. I pointed it at my user's /home/ directory, which contains 51,000+ files, totaling 9.3GB on a Reiser3 filesystem with tail enabled. That operation took probably half an hour, and that's on a fast SSD! All of Pinot's indexes and databases take up 455MB, bringing my total /home/ usage to about 9.7GB. Pinot typically used about 50% of my CPU while doing so, sometimes dropping down to the 20s and 40s.

However, since Pinot is on a fast SSD, and it's running off a 2.3Ghz dual-core Athlon backed by 4GB RAM, I didn't notice any performance hit while indexing. I'm not running any special kernels or schedulers (like BFS) either; just vanilla-source-2.6.35.4. There was no noticeable lag or slowdown, despite viewing two Thunar windows, working with four terminals, and browsing nine Firefox tabs. My system was only laggy when compiling Pinot and its dependencies.

Once my /home/ was indexed, I searched around. Queries were pretty much instantaneous. There's no easy way to measure the speed of each query, since it's much too fast to time with a stopwatch. That's probably mostly because of the SSD -- as it is, without a desktop indexer/search app, most similar queries take less than a second. Once the initial filesystem index is complete, Pinot drops back to just monitoring directories if you've told it to do so, relying on the inotify feature in the kernel. That drops CPU and memory usage to zero, as near as I can tell. Nice!

Pinot's greatest advantage on my system, at least, is not its speed, but its usefulness for easily finding deeply buried files and folders.

Interestingly, even though Pinot by default is not supposed to index Git, CVS, or SVN repositories, it seems to ignore that setting. Searching for "catfish" turns up a document named catfish tricks and all the ebuilds and git logs that have "catfish" in the title. Apparently Pinot's regex filter isn't very reliable. I probably need to add in another asterisk to disable searching or indexing of any files within a git directory.

Catfish

Catfish mostly works as expected, though it defaults to using "find" rather than "pinot" as its search engine. I haven't yet found a way to set it to use Pinot as the default search provider. Catfish is quick to load, and its layout is fairly intuitive. Sometimes, however, it will just stop working with Pinot, and even though Pinot has indexed my entire home directory, Catfish won't return any search results, though I can get those results by using Pinot's interface. The rest of the time it works great.

Besides offering a friendlier UI for searches, Catfish's real strengths are its useful options, both for presentation and for tying in with my desktop's filemanager. With a couple of commandline switches, Catfish can display thumbnails of various filetypes, use larger icons in search results, use various wrappers for opening and working with files, or even use powerful regex search methods. No, it won't have the awesome preview capabilities of Gloobus, but you also don't have to install all of Gnome to get similar features.

Right out of the box, Catfish will allow you to open files and folders obtained from your search results just by clicking them. I don't know if that works for all filemanagers, but it works with Thunar, which is all I ask.

I like to use Catfish in combination with another powerful feature of Thunar: custom actions. Since Thunar lacks a built-in search bar (aside from a rudimentary go-to alphabetical list when you press a key), how do you integrate a search utility? One way is by adding search functions to the right-click menu.

1. Open a Thunar window, and go to Edit -> Configure custom actions.
2. Click the plus icon: +. Give the action a helpful title, description, and icon. "Search" is pretty standard among icon sets, so there should always be one available even when you change themes.
3. Add the action command: catfish --path=%F
4. Now go to the Appearance Conditions tab. I left the file pattern as * and checked all boxes, so that no matter where I browse or click, I can launch a Catfish search.
5. Save the new action and exit Thunar. The next Thunar window you launch will let you right-click anywhere in the browser to open a Catfish search.

You can add any commandline switch you like to the catfish command; just run catfish --help to see the available options.

Thunar's custom action feature is pretty nifty; there are all kinds of things you can put in the context menu. It comes with an example to open a terminal in the current directory. You can create actions to launch applications with a root prompt, convert one image type into another, play media, print or email documents, and more. If you can script it, you can write a trigger for it and stick it in the context menu. Just read the custom actions documentation for many more examples of what you can do with Thunar. Neat!

Looking forward

So, will I keep using Pinot and Catfish? Possibly. While I am leery of any process like Pinot that writes so often to my SSD, and I'm not at all happy with its database size compared to my actual directory size, I do like that it's fast, and responsive. It doesn't seem to have the huge memory leaks or lag that Strigi/Nepomuk do in KDE. In fairness, KDE is trying to get us to believe in the power of the "semantic desktop," while Pinot and Catfish just want to create an easy frontend for finding stuff, without worrying about associating them with various files or activities.

As long as the database doesn't get too much larger, or the indexing/monitoring services use too many resources, I'll keep it around. I've got five+ years of accumulated files in various folders, with more constantly being loaded to and from offline backups. Pinot and Catfish can help with my hard drive spring cleaning, and help me locate stuff that I've just plain forgotten about. The older you get, the less you remember, right?

What I'd really like is a search bar built-in to Thunar, maybe in the upper right corner, backed by Pinot. That'd place everything I need right up front, without having to drill down through right-click menus.

* * *

Speaking of Thunar:

Do you use Thunar? Do you use Dropbox? Xfce developer Mike Massonnet posted a message to the xfce-dev list this morning with a link to a new project: Thunar Dropbox. It integrates the Dropbox service right into your favorite lightweight filemanager. No longer do you have to run Nautilus just to use Dropbox easily. Now you can use it within Thunar.

Original post from Planet Gentoo.

Part of today’s KDE Team meeting:

KDE 4.5 status and plans to put it in Portage

We agreed that KDE 4.5.1 is suffering of some important bugs, and after a long discussion we decided to put it in portage, but it will never make it to stable branch. We are mentioning the upstream bugs, as we think that users should be aware of them before updating:

• https://bugs.kde.org/show_bug.cgi?id=247144
• https://bugs.kde.org/show_bug.cgi?id=246931
• https://bugs.kde.org/show_bug.cgi?id=230247 <-The most important

Also, keep in mind that KDE SC 4.5 lacks the KDEPIM suite, so users should use KDEPIM 4.4.5 instead, which is also stable in portage tree.

In case of an update it should be smooth.

The whole summary and log can be found at the KDE project space.

The Qt Team also had a meeting one our later, summary and logs at the Qt project space

=-=-=-=-=
Powered by Blogilo

What: Larry the cow t-shirt graphics contest ...

Where: irc.freenode.net, #gentoo-artwork

Starts: Wed, Sep 01, 2010

Ends: Sun, Oct 31, 2010

Introduction:

Larry the cow is our prized mascot and he is not getting the respect he deserves. Just about every other Linux distribution, their mascot is adorned on t-shirts and other types of schwag, but not our Larry, he cannot even decide his gender. Please help up fix this terrible injustice by entering our contest and provide Larry the graphics he deserves. For more information visit this Forum Post.

The top 3 contestants will receive three t-shirts in their design, donated by the Gentoo Foundation. The judging panel includes Dawid Węgliński, Alex Legler, and your host David Abbott.

Requirments:

Preferred format is scalable vector graphics (SVG) but any of the industry standard vector graphics formats will be accepted.

How to Enter?

Please email your graphics to artwork@g.o

1. By making a submission to the contest, you agree to give unlimited, non-exclusive rights to the Gentoo Foundation for your submission.
2. This may include relicensing or trademarking the content the if required to protect it from unfair commercial usage.
3. You may, at your own discretion, also provide the content to any other party under some form of open license (including, but not limited to the full range of Creative Commons licenses, including re-sampling)
4. The Foundation needs to be informed of any future licensing.

In this Podcast comprookie attempts to explain some tips on staying motivated.

Larry the cow is Gentoo's prized mascot and he is not getting the respect he deserves. Just about every other Linux distribution, their mascot is adorned on t-shirts and other types of schwag, but not Larry, he cannot even decide his gender. Please help up fix this terrible injustice by entering the Gentoo Larry Graphic's Contest and provide Larry the graphics he deserves.

LINKS:
Motivation
http://gentoo-pr.org/node/29

Larry Graphics Contest
http://www.gentoo.org/news/20100901-larry-contest-announcement.xml

OpenRC
http://www.gentoo.org/proj/en/base/openrc/index.xml

Gentoo Live DVD
http://www.gentoo.org/proj/en/pr/releases/10.0/faq.xml

Download

Today I discovered another linux kernel patchset/fork that attempts to bring many features not in mainline together, called pf-kernel.

Quoting from its website:

“pf-kernel is another Linux kernel fork, that provides you with new useful features, that are not merged into mainline. It’s not based on any existing Linux fork or patchset, but some parts of Zen kernel may be merged if there’s no official release of needed patch. The name of this fork is not connected with BSD Packet Filter. «pf» means «post-factum» in the short form.”

At this point the latest patchset includes the following:
* mainline update: 2.6.35.4
* -ck patchset (BFS included)
* BFQ
* TuxOnIce
* LinuxIMQ

Its main advantage over other forks seems to be that it’s regularly updated

I’m using it on my main system without any issues, so I added it to Gentoo’s tree:

* sys-kernel/pf-sources
Available versions:
(2.6.31_p9) (~)2.6.31_p9!b!s
(2.6.32_p16) (~)2.6.32_p16!b!s
(2.6.33_p4) (~)2.6.33_p4!b!s
(2.6.34_p7) (~)2.6.34_p7!b!s
(2.6.35_p7) (~)2.6.35_p7!b!s
{build deblob symlink}
Homepage: http://pf-kernel.org.ua/
Description: Linux kernel fork with new features, including the -ck patchset (BFS), BFQ, TuxOnIce and LinuxIMQ

Many thanks to Oleksandr for taking the time to create and maintain it

What ties in reckless glibc unmasking GTK+ 2.20 issues Ruby 1.9 porting and --as-needed failures all together? Okay the title is a dead giveaway for the answer: undefined symbols.

Before deepening within the topic I first have to tell you about symbols I guess; and to do so, and to continue further, I’ll be using C as the base language for everyone of my notes. When considering C, then, a symbol is any function or data (constant or variable) that is declared extern; that is anything that is neither static or defined in the same translation unit (that is, source file, most of the time).

Now, what nm shows as undefined (U code) is not really what we’re concerned about; for object files (.o, just intermediate) will report undefined symbols for any function or data element used that is not in the same translation unit; most of those get resolved at the time all the object files get linked in to form a final shared object or executable — actually, it’s a lot more complex than this, but since I don’t care about describing here symbolic resolution, please accept it like it was true.

The remaining symbols will be keeping the U code in the shared object or executable, but most of them won’t concern us: they will be loaded from the linked libraries, when the dynamic loader actually resolve them. So for instance, the executable built from the following source code, will have the printf symbol “undefined” (for nm), but it’ll be resolved by the dynamic linker just fine:

I have explicitly avoided using the fprintf function, mostly because that would require a further undefined symbol, so…

Why do I say that undefined symbols are our worst enemy? Well, the problem is actually with undefined, unresolved symbols after the loader had its way. These are either symbols for functions and data that is not really defined, or is defined in libraries that are not linked in. The former case is what you get with most of the new-version compatibility problems (glibc, gtk, ruby); the latter is what you get with --as-needed.

Now, if you have a bit of practice with development and writing simple commands, you’d be now wondering why is this a kind of problem; if you were to mistype the function above into priltf – a symbol that does not exist, at least in the basic C library – the compiler will refuse to create an executable at all, even if the implicit declaration was only treated as a warning, because the symbol is, well, not defined. But this rule only applies, by default, to final executables, not to shared objects (shared libraries, dynamic libraries, .so, .dll or .dylib files).

For shared objects, you have to explicitly ask to refuse linking them with undefined reference, otherwise they are linked just fine, with no warning, no error, no bothering at all. The way you can tell the linker to refuse that kind of linkage is passing the -Wl,--no-undefined flag; this way if there is even a single symbol that is not defined in the current library or any of its dependencies the linker will refuse to complete the link. Unfortunately, using this by default is not going to work that well.

There are indeed some more or less good reasons to allow shared objects to have undefined symbols, and here come a few:

Multiple ABI-compatible libraries: okay this is a very far-fetched one, simply for the difficulty to have ABI-compatible libraries (it’s difficult enough to have them API-compatible!), but it happens; for instance on FreeBSD you – at least used to – have a few different implementations of the threading libraries, and have more or less the same situation for multiple OpenGL and mathematical libraries; the idea behind this is actually quite simply; if you have libA1 and libA2 providing the symbols, then libB linking to libA1, and libC linking to libA2, an executable foo linking to libB and libC would get both libraries linked together, and creating nasty symbol collisions.

Nowadays, FreeBSD handles this through a libmap.conf file that allows to link always the same library, but then switch at load-time with a different one; a similar approach is taken by things like libgssglue that allows to switch the GSSAPI implementation (which might be either of Kerberos or SPKM) with a configuration file. On Linux, beside this custom implementation, or hacks such as that used by Gentoo (eselect opengl) to handle the switch between different OpenGL implementations, there seem to be no interest in tackling the problem at the root. Indeed, I complained about that when --as-needed was softened to allow this situation although I guess it at least removed one common complain about adopting the option by default.

Plug-ins hosted by a standard executable: plug-ins are, generally speaking, shared objects; and with the exception of the most trivial plugins, whose output is only defined in terms of their input, they use functions that are provided by the software they plug. When they are hosted (loaded and used from) by a library, such as libxine, they are linked back to the library itself, and that makes sure that the symbols are known at the time of creating the plugin object. On the other hand, when the plug-ins are hosted by some software that is not a shared object (which is the case of, say, zsh), then you have no way to link them back, and the linker has no way to discern between undefined symbols that will be lifted from the host program, and those that are bad, and simply undefined.

Plug-ins providing symbols for other plug-ins : here you have a perfect example in the Ruby-GTK2 bindings; when I first introduced --no-undefined in the Gentoo packaging of Ruby (1.9 initially, nowadays all the three C-based implementations have the same flag passed on), we got reports of non-Portage users of Ruby-GTK2 having build failures. The reason? Since all the GObject-derived interfaces had to share the same tables and lists, the solution they chose was to export an interface, unrelated to the Ruby-extension interface (which is actually composed of a single function, bless them!), that the other extensions use; since you cannot reliably link modules one with the other, they don’t link to them and you get the usual problem of not distinguish between expected and unexpected undefined symbols.

Note: this particular case is not tremendously common; when loading plug-ins with dlopen() the default is to use the RTLD_LOCAL option, which means that the symbols are only available to the branch of libraries loaded together with that library or with explicit calls to dlsym(); this is a good thing because it reduces the chances of symbol collisions, and unexpected linking consequences. On the other hand, Ruby itself seems to go all the way against the common idea of safety: they require RTLD_GLOBAL (register all symbols in the global procedure linking table, so that they are available to be loaded at any point in the whole tree), and also require RTLD_LAZY, which makes it more troublesome if there are missing symbols — I’ll get later to what lazy bindings are.

Finally, the last case I can think of where there is at least some sense into all of this trouble, is reciprocating libraries, such as those in PulseAudio. In this situation, you have two libraries, each using symbols from one another. Since you need the other to fully link the one, but you need the one to link the other, you cannot exit the deadlock with --no-undefined turned on. This, and the executable-plugins-host, are the only two reasons that I find valid for not using --no-undefined by default — but unfortunately are not the only two used.

So, what about that lazy stuff? Well, the dynamic loader has to perform a “binding” of the undefined symbols to their definition; binding can happen in two modes, mainly: immediate (“now”) or lazy, the latter being the default. With lazy bindings, the loader will not try to find the definition to bind to the symbol until it’s actually needed (so until the function is called, or the data is fetched or written to); with immediate bindings, the loader will iterate over all the undefined symbols of an object when it is loaded (eventually loading up the dependencies). As you might guess, if there are undefined, unresolved symbol, the two binding types have very different behaviours. An immediately-loaded executable will fail to start, and a loaded library would fail dlopen(); a lazily-loaded executable will start up fine, and abort as soon as a symbol is hit that cannot be resolved; and a library would simply make its host program abort at the same way. Guess what’s safer?

With all these catches and issues, you can see why undefined symbols are a particularly nasty situation to deal with. To the best of my knowledge, there isn’t a real way to post-mortem an object to make sure that all its symbols are defined. I started writing support for that in Ruby-Elf but the results weren’t really… good. Lacking that, I’m not sure how we can proceed.

It would be possible to simply change the default to be --no-undefined, and work around with --undefined the few that require the undefined symbols to be there (we decided to proceed that way with Ruby); but given the kind of support I’ve received before in my drastic decisions, I don’t expect enough people to help me tackle that anytime soon — and I don’t have the material time to work on that, as you might guess.

Observant viewers may notice that I’ve begun uploading some of the stage tarballs. The first of the stages built is the stage 1 tarball for MIPS-I little-endian. Stage 2 is building as I type this, hopefully by this weekend it’ll be uploaded and stage 3 will be on its way.

I’ll let you know when all the MIPS-I little-endian builds are done, that way you won’t wind up downloading a partially uploaded file. That said, should you jump the gun, you should be able to resume the download to fetch the remainder. I’ve started with MIPS-I since that’s the lowest-common-denominator … MIPS-III will be next, followed by MIPS-IV.

I plan to get onto the big-endian port shortly. SGI owners have not been forgotten, just I’ve been busy.

Greetings! Today we have an excellent interview of two fine gentlemen, Mr. David Gallo and Mr. Cassius Adams from SevenL Networks Inc., conducted by our very own Joshua Saddler.

For those of you that have been around awhile, you will recognize the company represented in this interview. SevenL Networks Inc. has been an infrastructure sponsor for over six years now. David and Cassius took some time to fill us in on their use of Gentoo throughout their extensive operations. We hope you enjoy reading the complete interview which you will find here.

Discuss this!