GNU Hurd/ Contributing

Welcome to... ... the GNU Hurd!

So, you are interested in contributing to the GNU Hurd project? Welcome! Every single contribution is very much encouraged.

There are various ways to contribute; read up on contributing to...

  1. Improve GNU Hurd Running on GNU Mach
    1. Small hack entries
    2. Porting Packages
    3. TODO List
    4. Open Issues
    5. Instant Development Environment
  2. Design / Research: GNU Hurd on a Modern Microkernel
  3. Documentation
    1. Technical Writer
    2. Web Pages
  4. Final Words -- Difficulties

If you are lurking around here and would like to contribute, but feel you would do so better under formal mentoring: please contact us, or just speak up at one of the regular IRC meetings!

We also have a list of open issues and one for more elaborate project ideas - the latter originally written for the Google Summer of Code, but not exclusively. Even just investigating open issues, without being able to fix them, can be useful, because a issue that has been tracked down often becomes obvious to address for people who know the stuff -- but these people typically don't have the time that is needed to track down the issues.

Improve GNU Hurd Running on GNU Mach

The GNU Hurd running on the GNU Mach microkernel is what is commonly meant when people are talking about GNU/Hurd systems.

This system has mostly been designed and implemented in the '90s. It works and is usable. For example, these web pages have been rendered on a GNU/Hurd system.

You can try it out for yourself: for getting access, installing Debian GNU/Hurd will probably be the easiest and most feature-complete solution. If you don't have spare hardware to use for doing so, you can also get a shell account on a public Hurd machine. Depending on the things you're going to work on (and on your internet connection), this may be an easy way of getting used to Hurd systems. Installing in a virtual machine is another possibility, see the page about running a Hurd system for the full story. In particular, running a Debian GNU/Hurd QEMU image may be a viable alternative.

Then you can either play around and eventually strive to do something useful or -- if you want -- ask us to assign something to you, depending on the skills you have and the resources you intend to invest.

Please spend some time with thinking about the items in this questionnaire.

Before you can significantly contribute to the operating system itself, you'll need to take some time to learn about the system, for example: microkernels for beginners, Mach's concepts, Hurd's concepts, the critique. Until you can understand and do the basic exercises listed there, you won't be able to significantly contribute to the Hurd.

You can also have a look at the starting guide talk.

In terms of building and hacking on software, the easiest way to avoid having to understand the whole picture from the start is install the Debian distribution, and patch over the Debian source code. Installing from upstream source is much more complexe since you would need to know which piece fits where. Building and installing patched packages is much more simple.

For more reading resources, please see these web pages, for example, Hurd documentation and Mach documentation for links to a bunch of documents.

Small hack entries

Here is a list of small hacks, which can serve as entries into the Hurd code for people who would like to dive into the code but just lack a "somewhere to begin with". Make sure to check out the most up-to-date version on https://darnassus.sceen.net/~hurd-web/contributing

  • Teach rsync to use *getxattr and friends on GNU/Hurd too, to enable the -X option, so as to preserve translator entries.
  • Avoid GCC trampolines: as discussed in https://gcc.gnu.org/onlinedocs/gccint/Trampolines.html these happen when we pass the address of a nested function to another function. This can be seen by running readelf -S file.o | grep GNU-stack | grep X, for instance that happens in libdiskfs/file-exec.c, libdiskfs/io-revoke.c. We can't really use -fno-trampoline, we should instead add void *data parameters to iterators such as ports_class_iterate or fshelp_exec_reauth, so that the nested functions can be made mere static functions that get their information from the void *data parameter.
  • Implement pthread_setschedparam and sched_setscheduler in glibc by calling mach's thread_policy and thread_priority.
  • Strengthen httpfs: it should append '/' to URL automatically, it should not fallback index.html itself, etc. probably a lot more small easy issues.
  • Create a Wiki page with all presentations about the Hurd. Many are referenced here in the Wiki, but they are not easy to find. (open issue documentation)
  • Some translators do not support fsysopts, i.e. support for the file_get_fs_options and fsys_set_options RPCs.
  • Extend device_read/device_write into supporting > 2TiB disk sizes.
  • Make host_get_time much more precise by using the TSC.
  • Add NX / SMEP / SMAP protection support to GNU Mach.
  • Add use of PCID in GNU Mach.
  • Fix 64bit instruction set disassembling in GNU Mach's /i386/i386/db_disasm.c db_disasm function and tables.
  • Write a partfs translator, to which one gives a disk image, and which exposes the partitions of the disk image, using parted, and the parted-based storeio (settrans -c foos1 /hurd/storeio -T typed part:1:file:/home/samy/tmp/foo). This would be libnetfs-based.
  • Write virtio drivers for KVM.
  • Move the mount/umount logic from utils/{,u}mount.c into glibc.
  • Add a tool to trace system calls, by using gnumach's Syscall-Emulation, see http://www.gnu.org/software/hurd/gnumach-doc/Syscall-Emulation.html
  • Improve our FUSE library.
  • Fix our symlink translator.
  • Add gnumach support for EFI memory areas report through GetMemoryMap instead of the BIOS E820.
  • Implement SA_NOCLDWAIT. It means adding an RPC to proc to implement it, and then making glibc detect when setting SIG_IGN on SIGCLD, or setting the SA_NOCLDWAIT flag, and in that case call into proc, similarly to the S_proc_mod_stopchild RPC. proc's S_proc_wait shall then wait for all children and return ECHILD.
  • Implement lsof. One can get inspiration from libshouldbeinlibc/portinfo.c for the details.
  • Add VSTATUS support to term. Essentially in term/munge.c, input_character, just like the VINTR, VQUIT, VSUSP, collect a few stats from the system, and put them into the output queue.
  • Make mig use the access function attribute to properly express accesses in arrays, e.g. for device_read/write_inband.
  • Add a limitation in the number of a process that a given uid can run. That would be in the proc translator. That will allow to avoid crashes when an application goes crazy with forking. Setting a hardcoded limitation would thus already be useful.
  • Complete BPF program validation in libbpf. For instance, for now if BPF_MOD or BPF_XOR are used in a filter, it is accepted, but the matching always fails. We should pre-refuse any unknown instruction (and of course then implement BPF_MOD and BPF_XOR)

Porting Packages

Please contact us before spending a lot of time on the following porting tasks: some work may already have been done that you can base your work upon.

For guidelines, please have a look at the dedicated porting page.

Debian GNU/Hurd

Along with the official Debian "jessie" release (but not as an official Debian release), in April 2015 the Debian GNU/Hurd team released Debian GNU/Hurd 2015. There is a goal of getting Debian GNU/Hurd into shape for a technology preview for integration as a proper Debian release candidate.

The to do list is on http://wiki.debian.org/Debian_GNU/Hurd.

The following missing packages/missing functionality block a lot of other packages, and are thus good candidates for porting, in order to increase archive coverage:

  • umount functionality in busybox

Here is a list of packages that need porting.

You can also just install Debian GNU/Hurd and find what doesn't work or suit you and try to improve that.

Or, you can pick one from the list of failing packages.

TODO List

This is the list of tasks that we want to address soon, starting with the most pressing:

  • Add amd64 support to gdb, see Flavio patch pending commit
  • Fix shell output pipe replacement issue on amd64, see discussion.
    • This means adding an i386x_float_state and i386_XFLOAT_STATE thread status, that glibc would be able to use along i386_REGS_SEGS_STATE and i386_FLOAT_STATE in _hurd_setup_sighandler and sigtreturn.c. The structure would contain the fp_save_kind. That'll actually be needed both on i386 and x86_64 actually, to fix SSE use against signals in general.
  • Compare testsuite results of python on hurd-i386 and hurd-amd64, to fix regressions between the former and the latter.
  • Check the packages build failures differences between hurd-i386 and hurd-amd64: they are failing on hurd-amd64 but are successful on hurd-i386. Possibly it's just a mere missing s/hurd-i386/hurd-any/ in the debian/ directory, or a new bug that actually also affects hurd-i386 if you rebuild the package there now, but possibly it's a more profound issue in the amd64 port.
  • On amd64, fix memcpy (> 16 bytes) from /dev/mem (makes hurd-console crash)
  • On amd64, fix crash-core
  • On amd64, fix running posixtestsuite (not necessarily fixing the tests themselves, but at least make sure it doesn't crash the box ; it could still be useful to compare the output on 32bit and 64bit are the same with the same gnumach/hurd/glibc).
  • Settle CI for mig+gnumach+hurd+glibc.
  • Port dhcpcd, see call for help
  • Make sure that lwip supports configuration by DHCP.
  • Extensively test (e.g. running testsuites of glibc, perl, curl, rust-mio) and fix the lwip-based TCP/IP stack, to be sure we don't get regressions by switching to it.
  • Fix swapping with rumpdisk.
  • Fix the GPLv2 vs GPLv3 licence incompatibility between ext2fs and libparted:
    • For /hurd/ext2fs, use a different libstore that does not include the parted module.
    • For /hurd/ext2fs.static in the bootstrap chain (e.g. to access wd0s1), add a storeio translator before it, and have ext2fs open it, use file_get_storage_info and access the underlying device. That wouldn't need any code modification if we were using an initial ramfs exposing that storeio on /dev/wd0s1.
  • Prevent duplicate instances of rumpdisk from competing for the disk PCI cards (e.g. when a second one gets started from a chroot), otherwise mayhem happens.
  • Fix the memory consumption of rumpdisk.
  • Plug acpi shutdown event.
  • Add overcommit limitation support to gnumach (RLIMIT_AS): limit the virtual size of processes to half of the memory + swap size. Unless MAP_NORESERVE is passed to mmap.
  • Integrate rumpusbdisk with the rest of the disk translators etc.
  • Fix tmpfs losing files, see discussion.
  • Port libasan/lsan/ubsan/libtasn so we can use these sanitizers (youpi did some of it, pending clean/submit).
  • Finish moving pthread_ symbols from libpthread to libc, see for instance some moves, synchronize with Guy-Fleury Iteriteka.
  • Rewrite pthread_cond_*, pthread_rwlock_*, pthread_barrier_* to use gsync, like pthread_mutex_* do (also see the nptl implementations, possibly just share with them).
  • Improve rumpdisk's asynchronism, see end of hurd/rumpdisk/block-rump.c.
  • Check performance of rumpdisk against the in-gnumach drivers.
  • Make ext2fs use xattr by default to store translators (see use_xattr_translator_records) after making sure the upgrade path works fine.
  • Finish glib's file monitoring (see merge request draft and Debian bug)
  • Extend ext2fs to support 64bit time.
  • Fix the git testsuite (just a few tests failing, used to pass).
  • Fix the subversion testsuite (just a few tests failing).
  • Fix the vim testsuite (just a few tests failing, used to pass).
  • Fix building mesa.
  • Fix building wayland.
  • Port python-procps.
  • Implement a rumpnet.
  • Implement a rumpfs.
  • Fix SMP support.

Open Issues

There is a list of open issues. This list includes everything from bug reports to open-ended research questions.

Instant Development Environment

This is a very brief guide to get your development environment set up. Pester ArneBab @ irc.freenode.net on IRC if something does not work :) (open issue documentation)

First run the hurd in qemu

After you have a Hurd vm set up and running:

  • apt update
  • apt install -y git mercurial emacs vim
  • apt build-dep -y hurd gnumach
  • git clone git://git.savannah.gnu.org/hurd/hurd.git
  • git clone git://git.savannah.gnu.org/hurd/gnumach.git
  • git clone git://git.savannah.gnu.org/hurd/incubator.git
  • You can connect through ssh with ssh root@localhost -p 2222
  • Optionally if you connect to the Hurd running on another local machine, then you might want to set up the terrible-mdns-responder.
  • Get more from the repo list.
  • Read the docs on these pages.
  • Start hacking.
  • For shutting down, use reboot, then press c in grub and issue halt (to avoid filesystem corruption). Adding --no-reboot to the qemu line should help, too.

Design / Research: GNU Hurd on a Modern Microkernel

Developers have identified a number of problem with the Hurd on Mach system. Problems, that can not easily be fixed by bug-fixing the existing code base, but which require design changes -- deep going ones actually.

As such systems (as the desired one) are not in common use, but are -- if at all -- research projects, this new Hurd on a modern microkernel project itself is more a research project than a sit down and implement/code/hack project.

If you're interested in contributing in this area, knowing the Hurd on Mach system (see above) nevertheless is a prerequisite. At least have a deep look at the documentation pointers. Also read through the HurdNG section.

Please send email to the l4-hurd mailing list for discussing this post-Mach system design.

Documentation

Technical Writer

Our hackers (programmers) typically do what their kind always does: they code. What they don't like too much is documenting their wonderful achievements. On the other hand, there are people (you?) who enjoy documenting technical matters, so don't hesitate to contact us if technical documentation shall be your contribution to GNU Hurd development.

A good start is probably to just start using the Hurd, and play with the translators. In the process you will probably find that some of the documentations are missing some details, are outdated, etc. That is were you can start contributing for instance.

As an advice: do not start yet another documentation from scratch. There are already a lot of tutorials in the wilds, and they are almost all completely outdated. Rather contribute to the existing official documentation: this wiki, the documentation in the Hurd source, the Debian Hurd port pages.

Web Pages

Please read about how to contribute to these web pages.

Final Words -- Difficulties

Please note that doing substantial contributions to a project as big and as encompassing as the GNU Hurd is not a trivial task. For working on the GNU Hurd's inner guts and getting useful work done, you have to plan for a many-months learning experience which will need sufficient self-motivation. Working on an advanced operating system kernel isn't something you can do in a few free minutes -- even less so without any previous kernel hacking experience.

Likewise, the Linux kernel maintainers are stating the exactly same difficulties, which is well presented by Jonathan Corbet in his 2010 Linux Kernel Summit report for the opening sessions about welcoming of newcomers.

But of course, none of this is meant to be dismissive, or to scare you away -- on the contrary: just start using the GNU Hurd, and either notice yourself what's not working as expected, or have a look at one of the Open Issues, and we shall see if you'll evolve to be the next core Hurd hacker! You'll just have to get excited about it!