diff options
Diffstat (limited to 'open_issues')
42 files changed, 1433 insertions, 259 deletions
diff --git a/open_issues/anatomy_of_a_hurd_system.mdwn b/open_issues/anatomy_of_a_hurd_system.mdwn new file mode 100644 index 00000000..e1d5c9d8 --- /dev/null +++ b/open_issues/anatomy_of_a_hurd_system.mdwn @@ -0,0 +1,73 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!taglink open_issue_documentation]] + +A bunch of this should also be covered in other (introductionary) material, +like Bushnell's Hurd paper. All this should be unfied and streamlined. + +IRC, freenode, #hurd, 2011-03-08 + + <foocraft> I've a question on what are the "units" in the hurd project, if + you were to divide them into units if they aren't, and what are the + dependency relations between those units(roughly, nothing too pedantic + for now) + <antrik> there is GNU Mach (the microkernel); there are the server + libraries in the Hurd package; there are the actual servers in the same; + and there is the POSIX implementation layer in glibc + <antrik> relations are a bit tricky + <antrik> Mach is the base layer which implements IPC and memory management + <foocraft> hmm I'll probably allocate time for dependency graph generation, + in the worst case + <antrik> on top of this, the Hurd servers, using the server libraries, + implement various aspects of the system functionality + <antrik> client programs use libc calls to use the servers + <antrik> (servers also use libc to communicate with other servers and/or + Mach though) + <foocraft> so every server depends solely on mach, and no other server? + <foocraft> s/mach/mach and/or libc/ + <antrik> I think these things should be pretty clear one you are somewhat + familiar with the Hurd architecture... nothing really tricky there + <antrik> no + <antrik> servers often depend on other servers for certain functionality + +--- + +IRC, freenode, #hurd, 2011-03-12 + + <dEhiN> when mach first starts up, does it have some basic i/o or fs + functionality built into it to start up the initial hurd translators? + <antrik> I/O is presently completely in Mach + <antrik> filesystems are in userspace + <antrik> the root filesystem and exec server are loaded by grub + <dEhiN> o I see + <dEhiN> so in order to start hurd, you would have to start mach and + simultaneously start the root filesystem and exec server? + <antrik> not exactly + <antrik> GRUB loads all three, and then starts Mach. Mach in turn starts + the servers according to the multiboot information passed from GRUB + <dEhiN> ok, so does GRUB load them into ram? + <dEhiN> I'm trying to figure out in my mind how hurd is initially started + up from a low-level pov + <antrik> yes, as I said, GRUB loads them + <dEhiN> ok, thanks antrik...I'm new to the idea of microkernels, but a + veteran of monolithic kernels + <dEhiN> although I just learned that windows nt is a hybrid kernel which I + never knew! + <rm> note there's a /hurd/ext2fs.static + <rm> I belive that's what is used initially... right? + <antrik> yes + <antrik> loading the shared libraries in addition to the actual server + would be unweildy + <antrik> so the root FS server is linked statically instead + <dEhiN> what does the root FS server do? + <antrik> well, it serves the root FS ;-) + <antrik> it also does some bootstrapping work during startup, to bring the + rest of the system up diff --git a/open_issues/binutils.mdwn b/open_issues/binutils.mdwn index 81fafaca..ca7496f0 100644 --- a/open_issues/binutils.mdwn +++ b/open_issues/binutils.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -9,7 +9,7 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!tag open_issue_binutils]] +[[!tag stable_URL open_issue_binutils]] Here's what's to be done for maintaining GNU Binutils. @@ -30,15 +30,14 @@ though, as explained below. # Configuration -Last reviewed up to the [[Git mirror's e347ef3b343fc42ed312d5125047d59ae15df795 -(2010-12-20) sources|source_repositories/binutils]]. +Last reviewed up to the [[Git mirror's a446ef2f3862fb5f89c669b34a2b6a2ab943ff96 +(2011-02-10) sources|source_repositories/binutils]]. * Globally * a.out, COFF, PE image support and 64 bit support are not interesting. - * In the [[testsuite]]s, `.exp` and `.d` files very likely should not - only + * In the testsuites, `.exp` and `.d` files very likely should not only care for `*-*-linux*`, but also `*-*-gnu*`. (If the need to be conditionalized like this at all.) @@ -96,7 +95,7 @@ Last reviewed up to the [[Git mirror's e347ef3b343fc42ed312d5125047d59ae15df795 * `*-*-gnu*` TODO: resolve `crt0.o` vs. `crt1.o` issue. [[Testsuite - failures|testsuite#static]]. + failures|binutils#static]]. * `configure.tgt` @@ -109,7 +108,7 @@ Last reviewed up to the [[Git mirror's e347ef3b343fc42ed312d5125047d59ae15df795 # Build Here's a log of a binutils build run; this is from our [[Git -repository's 245f62b817ee31135a190793dddb340f04ac95e6 (2010-12-20) +repository's e8052e7548e0d5523f1764b7d3896ca000bfaed7 (2011-02-10) sources|source_repositories/binutils]], run on kepler.SCHWINGE and grubber. $ export LC_ALL=C @@ -121,7 +120,7 @@ sources|source_repositories/binutils]], run on kepler.SCHWINGE and grubber. (kepler.SCHWINGE defaults to using /bin/sh for libtool, grubber to /bin/bash; thus harmonized.) -On grubber, this takes roughly one hour. +On grubber, this needs roughly one hour, and takes up around 100 MiB. ## Analysis @@ -145,7 +144,7 @@ GNU/Linux defining `-DTRAD_CORE`, `-DHAVE_i386linux_vec` (kepler.SCHWINGE defaults to using /bin/sh, grubber to /bin/bash; thus harmonized.) -On grubber, this needs roughly 15 minutes, and takes up around 0.7 GiB. +On grubber, this needs roughly 5 minutes, and takes up around 60 MiB. ## Analysis @@ -170,12 +169,12 @@ On grubber, this takes roughly one hour. Comparing the results files, [[sum_linux]] to [[sum_hurd]]: $ diff -u -F ^Running open_issues/binutils/sum_linux open_issues/binutils/sum_hurd - --- open_issues/binutils/sum_linux 2010-12-20 19:01:06.000000000 +0100 - +++ open_issues/binutils/sum_hurd 2010-12-20 19:01:20.000000000 +0100 + --- open_issues/binutils/sum_linux 2011-02-10 19:01:56.000000000 +0100 + +++ open_issues/binutils/sum_hurd 2011-02-10 20:27:17.000000000 +0100 @@ -1,5 +1,5 @@ - -Test Run By thomas on Mon Dec 20 11:34:53 2010 + -Test Run By thomas on Thu Feb 10 18:57:42 2011 -Native configuration is i686-pc-linux-gnu - +Test Run By tschwinge on Mon Dec 20 11:35:47 2010 + +Test Run By tschwinge on Thu Feb 10 18:58:16 2011 +Native configuration is i686-unknown-gnu0.3 === binutils tests === @@ -184,9 +183,9 @@ Comparing the results files, [[sum_linux]] to [[sum_hurd]]: # of expected passes 83 # of unsupported tests 2 - -Test Run By thomas on Mon Dec 20 11:35:19 2010 + -Test Run By thomas on Thu Feb 10 18:58:10 2011 -Native configuration is i686-pc-linux-gnu - +Test Run By tschwinge on Mon Dec 20 11:44:29 2010 + +Test Run By tschwinge on Thu Feb 10 19:06:15 2011 +Native configuration is i686-unknown-gnu0.3 === ld tests === @@ -232,21 +231,21 @@ Comparing the results files, [[sum_linux]] to [[sum_hurd]]: PASS: ELF DSO small bar (size) PASS: ELF DSO foo with small bar (size) PASS: ELF DSO big bar (size) - @@ -873,13 +873,14 @@ Running [...]/hurd/master/ld/testsuite/l + @@ -882,13 +882,14 @@ Running [...]/hurd/master/ld/testsuite/l === ld Summary === - -# of expected passes 618 + -# of expected passes 626 -# of expected failures 8 - +# of expected passes 608 + +# of expected passes 616 +# of unexpected successes 1 +# of expected failures 17 # of untested testcases 6 - /media/data[...]/hurd/master.build/ld/ld-new 2.21.51.20101220 + /media/data[...]/hurd/master.build/ld/ld-new 2.21.51.20110210 - -Test Run By thomas on Mon Dec 20 11:34:59 2010 + -Test Run By thomas on Thu Feb 10 18:57:49 2011 -Native configuration is i686-pc-linux-gnu - +Test Run By tschwinge on Mon Dec 20 11:38:03 2010 + +Test Run By tschwinge on Thu Feb 10 19:00:16 2011 +Native configuration is i686-unknown-gnu0.3 === gas tests === @@ -255,7 +254,7 @@ Comparing the results files, [[sum_linux]] to [[sum_hurd]]: ## Analysis - * <a name="static">`FAIL: static [...]`</a> + * <a name="static"><!-- stable_URL -->`FAIL: static [...]`</a> The testsuite isn't prepared for using `crt0.o` instead of `crt1.o` depending on whether a static or dynamic executable is created. Documented @@ -269,7 +268,7 @@ Comparing the results files, [[sum_linux]] to [[sum_hurd]]: weakness|performance/io_system/binutils_ld_64ksec]]), so assuming some system load variation, the testsuite's timeout may trigger. - * <a name="weak">`FAIL: ELF weak [...]`</a> + * <a name="weak"><!-- stable_URL -->`FAIL: ELF weak [...]`</a> [[I|tschwinge]] suppose this is due to us having an override w.r.t. weak symbol handling in glibc, needed for our external [[/libpthread]]. TODO: diff --git a/open_issues/binutils/log_build-diff b/open_issues/binutils/log_build-diff index 802d510c..3408d97d 100644 --- a/open_issues/binutils/log_build-diff +++ b/open_issues/binutils/log_build-diff @@ -1,5 +1,5 @@ ---- /dev/fd/63 2010-12-20 11:34:03.204493002 +0100 -+++ /dev/fd/62 2010-12-20 11:34:03.208493002 +0100 +--- /dev/fd/63 2011-02-10 17:33:04.738225001 +0100 ++++ /dev/fd/62 2011-02-10 17:33:04.738225001 +0100 @@ -1,6 +1,6 @@ -checking build system type... i686-pc-linux-gnu -checking host system type... i686-pc-linux-gnu @@ -530,7 +530,7 @@ checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... (cached) yes checking for gcc option to accept ISO C89... (cached) none needed -@@ -2450,28 +2429,28 @@ +@@ -2453,28 +2432,28 @@ checking for BSD- or MS-compatible name lister (nm)... nm checking the name lister (nm) interface... BSD nm checking whether ln -s works... yes @@ -566,23 +566,37 @@ checking how to hardcode library paths into programs... immediate checking whether stripping libraries is possible... yes checking if libtool supports shared libraries... yes -@@ -2555,13 +2534,13 @@ +@@ -2486,11 +2465,11 @@ + checking whether the g++ linker (ld) supports shared libraries... yes + checking for g++ option to produce PIC... -fPIC -DPIC + checking if g++ PIC flag -fPIC -DPIC works... yes +-checking if g++ static flag -static works... yes ++checking if g++ static flag -static works... no + checking if g++ supports -c -o file.o... yes + checking if g++ supports -c -o file.o... (cached) yes + checking whether the g++ linker (ld) supports shared libraries... yes +-checking dynamic linker characteristics... (cached) GNU/Linux ld.so ++checking dynamic linker characteristics... gnu0.3 ld.so + checking how to hardcode library paths into programs... immediate + checking whether NLS is requested... yes + checking for catalogs to be installed... bg da es fi fr ga id ja sv tr vi zh_CN zh_TW +@@ -2570,13 +2549,13 @@ /bin/bash ../../master/ld/../ylwrap ../../master/ld/ldgram.y y.tab.c ldgram.c y.tab.h ldgram.h y.output ldgram.output -- bison -y -d updating ldgram.h (echo "/* This file is automatically generated. DO NOT EDIT! */";\ -- for f in `echo " " eelf_i386.o ei386linux.o "" \ +- for f in `echo " " eelf_i386.o ei386linux.o eelf32_x86_64.o "" \ + for f in `echo " " eelf_i386.o "" \ | sed -e 's/ e/ ld/g' -e 's/ ld/ /g' -e 's/[.]o//g'`; do \ echo "extern ld_emulation_xfer_type ld_${f}_emulation;"; \ done;\ echo "";\ echo "#define EMULATION_LIST \\";\ -- for f in `echo " " eelf_i386.o ei386linux.o "" \ +- for f in `echo " " eelf_i386.o ei386linux.o eelf32_x86_64.o "" \ + for f in `echo " " eelf_i386.o "" \ | sed -e 's/ e/ ld/g' -e 's/ ld/ /g' -e 's/[.]o//g'`; do \ echo " &ld_${f}_emulation, \\"; \ done;\ -@@ -2650,8 +2629,8 @@ +@@ -2665,8 +2644,8 @@ mv -f .deps/ldctor.Tpo .deps/ldctor.Po gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT ldmain.o -MD -MP -MF .deps/ldmain.Tpo -c -o ldmain.o \ -DDEFAULT_EMULATION='"elf_i386"' \ @@ -593,7 +607,7 @@ ../../master/ld/ldmain.c mv -f .deps/ldmain.Tpo .deps/ldmain.Po gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT ldwrite.o -MD -MP -MF .deps/ldwrite.Tpo -c -o ldwrite.o ../../master/ld/ldwrite.c -@@ -2665,7 +2644,7 @@ +@@ -2680,7 +2659,7 @@ gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT ldmisc.o -MD -MP -MF .deps/ldmisc.Tpo -c -o ldmisc.o ../../master/ld/ldmisc.c mv -f .deps/ldmisc.Tpo .deps/ldmisc.Po gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT ldfile.o -MD -MP -MF .deps/ldfile.Tpo -c -o ldfile.o \ @@ -602,19 +616,22 @@ ../../master/ld/ldfile.c mv -f .deps/ldfile.Tpo .deps/ldfile.Po gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT ldcref.o -MD -MP -MF .deps/ldcref.Tpo -c -o ldcref.o ../../master/ld/ldcref.c -@@ -2673,14 +2652,11 @@ +@@ -2688,17 +2667,11 @@ gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT plugin.o -MD -MP -MF .deps/plugin.Tpo -c -o plugin.o ../../master/ld/plugin.c mv -f .deps/plugin.Tpo .deps/plugin.Po cp ../../master/ld/emultempl/astring.sed stringify.sed --LIB_PATH='' /bin/bash ../../master/ld/genscripts.sh "../../master/ld" "[...]/hurd/master.build.install/lib" "[...]/hurd/master.build.install" "[...]/hurd/master.build.install" i686-pc-linux-gnu i686-pc-linux-gnu i686-pc-linux-gnu "elf_i386" "/usr/local/lib /lib /usr/lib" no yes elf_i386 "i686-pc-linux-gnu" +-LIB_PATH='' /bin/bash ../../master/ld/genscripts.sh "../../master/ld" "[...]/hurd/master.build.install/lib" "[...]/hurd/master.build.install" "[...]/hurd/master.build.install" i686-pc-linux-gnu i686-pc-linux-gnu i686-pc-linux-gnu "elf_i386 elf32_x86_64" "/usr/local/lib /lib /usr/lib" no yes elf_i386 "i686-pc-linux-gnu" +LIB_PATH='' /bin/bash ../../master/ld/genscripts.sh "../../master/ld" "[...]/hurd/master.build.install/lib" "[...]/hurd/master.build.install" "[...]/hurd/master.build.install" i686-unknown-gnu0.3 i686-unknown-gnu0.3 i686-unknown-gnu0.3 "elf_i386" "/usr/local/lib /lib /usr/lib" no yes elf_i386 "i686-unknown-gnu0.3" gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT eelf_i386.o -MD -MP -MF .deps/eelf_i386.Tpo -c -o eelf_i386.o eelf_i386.c mv -f .deps/eelf_i386.Tpo .deps/eelf_i386.Po --LIB_PATH='' /bin/bash ../../master/ld/genscripts.sh "../../master/ld" "[...]/hurd/master.build.install/lib" "[...]/hurd/master.build.install" "[...]/hurd/master.build.install" i686-pc-linux-gnu i686-pc-linux-gnu i686-pc-linux-gnu "elf_i386" "/usr/local/lib /lib /usr/lib" no yes i386linux "i686-pc-linux-gnuaout" +-LIB_PATH='' /bin/bash ../../master/ld/genscripts.sh "../../master/ld" "[...]/hurd/master.build.install/lib" "[...]/hurd/master.build.install" "[...]/hurd/master.build.install" i686-pc-linux-gnu i686-pc-linux-gnu i686-pc-linux-gnu "elf_i386 elf32_x86_64" "/usr/local/lib /lib /usr/lib" no yes i386linux "i686-pc-linux-gnuaout" -gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT ei386linux.o -MD -MP -MF .deps/ei386linux.Tpo -c -o ei386linux.o ei386linux.c -mv -f .deps/ei386linux.Tpo .deps/ei386linux.Po --/bin/bash ./libtool --tag=CC --mode=link gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -o ld-new ldgram.o ldlex-wrapper.o lexsup.o ldlang.o mri.o ldctor.o ldmain.o ldwrite.o ldexp.o ldemul.o ldver.o ldmisc.o ldfile.o ldcref.o plugin.o eelf_i386.o ei386linux.o ../bfd/libbfd.la ../libiberty/libiberty.a -lz -ldl --libtool: link: gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -o ld-new ldgram.o ldlex-wrapper.o lexsup.o ldlang.o mri.o ldctor.o ldmain.o ldwrite.o ldexp.o ldemul.o ldver.o ldmisc.o ldfile.o ldcref.o plugin.o eelf_i386.o ei386linux.o ../bfd/.libs/libbfd.a ../libiberty/libiberty.a -lz -ldl +-LIB_PATH='' /bin/bash ../../master/ld/genscripts.sh "../../master/ld" "[...]/hurd/master.build.install/lib" "[...]/hurd/master.build.install" "[...]/hurd/master.build.install" i686-pc-linux-gnu i686-pc-linux-gnu i686-pc-linux-gnu "elf_i386 elf32_x86_64" "/usr/local/lib /lib /usr/lib" no yes elf32_x86_64 "i686-pc-linux-gnu" +-gcc -DHAVE_CONFIG_H -I. -I../../master/ld -I. -I../../master/ld -I../bfd -I../../master/ld/../bfd -I../../master/ld/../include -g -O2 -DENABLE_PLUGINS -DLOCALEDIR="\"[...]/hurd/master.build.install/share/locale\"" -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -MT eelf32_x86_64.o -MD -MP -MF .deps/eelf32_x86_64.Tpo -c -o eelf32_x86_64.o eelf32_x86_64.c +-mv -f .deps/eelf32_x86_64.Tpo .deps/eelf32_x86_64.Po +-/bin/bash ./libtool --tag=CC --mode=link gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -o ld-new ldgram.o ldlex-wrapper.o lexsup.o ldlang.o mri.o ldctor.o ldmain.o ldwrite.o ldexp.o ldemul.o ldver.o ldmisc.o ldfile.o ldcref.o plugin.o eelf_i386.o ei386linux.o eelf32_x86_64.o ../bfd/libbfd.la ../libiberty/libiberty.a -lz -ldl +-libtool: link: gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -o ld-new ldgram.o ldlex-wrapper.o lexsup.o ldlang.o mri.o ldctor.o ldmain.o ldwrite.o ldexp.o ldemul.o ldver.o ldmisc.o ldfile.o ldcref.o plugin.o eelf_i386.o ei386linux.o eelf32_x86_64.o ../bfd/.libs/libbfd.a ../libiberty/libiberty.a -lz -ldl +/bin/bash ./libtool --tag=CC --mode=link gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -o ld-new ldgram.o ldlex-wrapper.o lexsup.o ldlang.o mri.o ldctor.o ldmain.o ldwrite.o ldexp.o ldemul.o ldver.o ldmisc.o ldfile.o ldcref.o plugin.o eelf_i386.o ../bfd/libbfd.la ../libiberty/libiberty.a -lz -ldl +libtool: link: gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Werror -g -O2 -o ld-new ldgram.o ldlex-wrapper.o lexsup.o ldlang.o mri.o ldctor.o ldmain.o ldwrite.o ldexp.o ldemul.o ldver.o ldmisc.o ldfile.o ldcref.o plugin.o eelf_i386.o ../bfd/.libs/libbfd.a ../libiberty/libiberty.a -lz -ldl touch ld.1 diff --git a/open_issues/binutils/log_install-diff b/open_issues/binutils/log_install-diff index 83c8d7b6..00496f8b 100644 --- a/open_issues/binutils/log_install-diff +++ b/open_issues/binutils/log_install-diff @@ -1,5 +1,5 @@ ---- /dev/fd/63 2010-12-20 19:00:16.368493004 +0100 -+++ /dev/fd/62 2010-12-20 19:00:16.368493004 +0100 +--- /dev/fd/63 2011-02-10 18:56:20.086225001 +0100 ++++ /dev/fd/62 2011-02-10 18:56:20.086225001 +0100 @@ -68,7 +68,6 @@ libtool: install: /usr/bin/install -c .libs/libbfd.a [...]/hurd/master.build.install/lib/libbfd.a libtool: install: chmod 644 [...]/hurd/master.build.install/lib/libbfd.a diff --git a/open_issues/binutils/sum_hurd b/open_issues/binutils/sum_hurd index 96dd0cb2..15d225f9 100644 --- a/open_issues/binutils/sum_hurd +++ b/open_issues/binutils/sum_hurd @@ -1,4 +1,4 @@ -Test Run By tschwinge on Mon Dec 20 11:35:47 2010 +Test Run By tschwinge on Thu Feb 10 18:58:16 2011 Native configuration is i686-unknown-gnu0.3 === binutils tests === @@ -114,7 +114,7 @@ Running [...]/hurd/master/binutils/testsuite/binutils-all/x86-64/x86-64.exp ... # of expected passes 83 # of unsupported tests 2 -Test Run By tschwinge on Mon Dec 20 11:44:29 2010 +Test Run By tschwinge on Thu Feb 10 19:06:15 2011 Native configuration is i686-unknown-gnu0.3 === ld tests === @@ -640,6 +640,8 @@ PASS: ld-ifunc/ifunc-1-local-x86 PASS: ld-ifunc/ifunc-1-x86 PASS: ld-ifunc/ifunc-10-i386 PASS: ld-ifunc/ifunc-11-i386 +PASS: ld-ifunc/ifunc-12-i386 +PASS: ld-ifunc/ifunc-13-i386 PASS: ld-ifunc/ifunc-2-i386 PASS: ld-ifunc/ifunc-2-local-i386 PASS: ld-ifunc/ifunc-3a-x86 @@ -669,6 +671,8 @@ Running [...]/hurd/master/ld/testsuite/ld-m68k/m68k.exp ... Running [...]/hurd/master/ld/testsuite/ld-mep/mep.exp ... Running [...]/hurd/master/ld/testsuite/ld-mips-elf/mips-elf-flags.exp ... Running [...]/hurd/master/ld/testsuite/ld-mips-elf/mips-elf.exp ... +Running [...]/hurd/master/ld/testsuite/ld-misc/defsym.exp ... +PASS: ld-misc/defsym1 Running [...]/hurd/master/ld/testsuite/ld-mmix/mmix.exp ... Running [...]/hurd/master/ld/testsuite/ld-mn10300/mn10300.exp ... Running [...]/hurd/master/ld/testsuite/ld-pe/pe-compile.exp ... @@ -705,6 +709,7 @@ Running [...]/hurd/master/ld/testsuite/ld-scripts/alignof.exp ... PASS: ALIGNOF Running [...]/hurd/master/ld/testsuite/ld-scripts/assert.exp ... PASS: ASSERT +PASS: ld-scripts/assert2 Running [...]/hurd/master/ld/testsuite/ld-scripts/crossref.exp ... PASS: NOCROSSREFS 1 PASS: NOCROSSREFS 2 @@ -720,6 +725,8 @@ Running [...]/hurd/master/ld/testsuite/ld-scripts/defined.exp ... PASS: DEFINED (PRMS 5699) PASS: ld-scripts/defined2 PASS: ld-scripts/defined3 +PASS: ld-scripts/defined4 +PASS: ld-scripts/defined5 Running [...]/hurd/master/ld/testsuite/ld-scripts/dynamic-sections.exp ... PASS: dynamic sections Running [...]/hurd/master/ld/testsuite/ld-scripts/empty-address.exp ... @@ -735,6 +742,8 @@ Running [...]/hurd/master/ld/testsuite/ld-scripts/empty-orphan.exp ... PASS: ld-scripts/empty-orphan Running [...]/hurd/master/ld/testsuite/ld-scripts/expr.exp ... PASS: ld-scripts/expr1 +PASS: ld-scripts/expr2 +PASS: ld-scripts/sane1 Running [...]/hurd/master/ld/testsuite/ld-scripts/extern.exp ... PASS: EXTERN Running [...]/hurd/master/ld/testsuite/ld-scripts/include.exp ... @@ -873,13 +882,13 @@ Running [...]/hurd/master/ld/testsuite/ld-xtensa/xtensa.exp ... === ld Summary === -# of expected passes 608 +# of expected passes 616 # of unexpected successes 1 # of expected failures 17 # of untested testcases 6 -/media/data[...]/hurd/master.build/ld/ld-new 2.21.51.20101220 +/media/data[...]/hurd/master.build/ld/ld-new 2.21.51.20110210 -Test Run By tschwinge on Mon Dec 20 11:38:03 2010 +Test Run By tschwinge on Thu Feb 10 19:00:16 2011 Native configuration is i686-unknown-gnu0.3 === gas tests === @@ -961,8 +970,8 @@ PASS: CFI common 2 PASS: CFI common 3 PASS: CFI common 4 PASS: CFI common 5 -PASS: CFI common 7 PASS: CFI common 6 +PASS: CFI common 7 Running [...]/hurd/master/gas/testsuite/gas/cr16/cr16.exp ... Running [...]/hurd/master/gas/testsuite/gas/cr16/pic.exp ... Running [...]/hurd/master/gas/testsuite/gas/cris/cris.exp ... @@ -1001,6 +1010,7 @@ PASS: section flags PASS: DWARF2 1 PASS: DWARF2 2 PASS: DWARF2 3 +PASS: DWARF2 4 PASS: Check bad section flag Running [...]/hurd/master/gas/testsuite/gas/fr30/allinsn.exp ... Running [...]/hurd/master/gas/testsuite/gas/fr30/fr30.exp ... @@ -1094,8 +1104,10 @@ PASS: i386 -mtune=i686 nops 3 PASS: i386 nops 4 PASS: i386 nops -mtune=i386 4 PASS: i386 -mtune=i686 nops 4 +PASS: i386 -march=i686+nop nops 4a PASS: i386 nops 5 PASS: i386 -march=i686 nops 5 +PASS: i386 nops 6 PASS: i386 16-bit addressing in 32-bit mode. PASS: i386 32-bit addressing in 16-bit mode. PASS: i386 SSE4.1 @@ -1176,6 +1188,10 @@ PASS: i386 FMA scalar insns (Intel disassembly) PASS: i386 FMA4 PASS: i386 LWP PASS: i386 XOP +PASS: i386 BMI insns +PASS: i386 BMI insns (Intel disassembly) +PASS: i386 TBM +PASS: i386 TBM insns (Intel disassembly) PASS: i386 F16C PASS: i386 F16C (Intel disassembly) PASS: i386 FSGSBase @@ -1217,6 +1233,10 @@ PASS: i386 list-1 PASS: i386 list-2 PASS: i386 list-3 PASS: DWARF2 debugging information 1 +Running [...]/hurd/master/gas/testsuite/gas/i386/ilp32/cfi/ilp32.exp ... +Running [...]/hurd/master/gas/testsuite/gas/i386/ilp32/elf/ilp32.exp ... +Running [...]/hurd/master/gas/testsuite/gas/i386/ilp32/ilp32.exp ... +Running [...]/hurd/master/gas/testsuite/gas/i386/ilp32/lns/ilp32.exp ... Running [...]/hurd/master/gas/testsuite/gas/i860/i860.exp ... Running [...]/hurd/master/gas/testsuite/gas/ia64/ia64.exp ... Running [...]/hurd/master/gas/testsuite/gas/ieee-fp/x930509a.exp ... @@ -1317,6 +1337,6 @@ Running [...]/hurd/master/gas/testsuite/gas/z8k/z8k.exp ... === gas Summary === -# of expected passes 319 -../as-new 2.21.51.20101220 +# of expected passes 326 +../as-new 2.21.51.20110210 diff --git a/open_issues/binutils/sum_linux b/open_issues/binutils/sum_linux index c2dae925..49cf53fb 100644 --- a/open_issues/binutils/sum_linux +++ b/open_issues/binutils/sum_linux @@ -1,4 +1,4 @@ -Test Run By thomas on Mon Dec 20 11:34:53 2010 +Test Run By thomas on Thu Feb 10 18:57:42 2011 Native configuration is i686-pc-linux-gnu === binutils tests === @@ -114,7 +114,7 @@ Running [...]/hurd/master/binutils/testsuite/binutils-all/x86-64/x86-64.exp ... # of expected passes 83 # of unsupported tests 2 -Test Run By thomas on Mon Dec 20 11:35:19 2010 +Test Run By thomas on Thu Feb 10 18:58:10 2011 Native configuration is i686-pc-linux-gnu === ld tests === @@ -640,6 +640,8 @@ PASS: ld-ifunc/ifunc-1-local-x86 PASS: ld-ifunc/ifunc-1-x86 PASS: ld-ifunc/ifunc-10-i386 PASS: ld-ifunc/ifunc-11-i386 +PASS: ld-ifunc/ifunc-12-i386 +PASS: ld-ifunc/ifunc-13-i386 PASS: ld-ifunc/ifunc-2-i386 PASS: ld-ifunc/ifunc-2-local-i386 PASS: ld-ifunc/ifunc-3a-x86 @@ -669,6 +671,8 @@ Running [...]/hurd/master/ld/testsuite/ld-m68k/m68k.exp ... Running [...]/hurd/master/ld/testsuite/ld-mep/mep.exp ... Running [...]/hurd/master/ld/testsuite/ld-mips-elf/mips-elf-flags.exp ... Running [...]/hurd/master/ld/testsuite/ld-mips-elf/mips-elf.exp ... +Running [...]/hurd/master/ld/testsuite/ld-misc/defsym.exp ... +PASS: ld-misc/defsym1 Running [...]/hurd/master/ld/testsuite/ld-mmix/mmix.exp ... Running [...]/hurd/master/ld/testsuite/ld-mn10300/mn10300.exp ... Running [...]/hurd/master/ld/testsuite/ld-pe/pe-compile.exp ... @@ -705,6 +709,7 @@ Running [...]/hurd/master/ld/testsuite/ld-scripts/alignof.exp ... PASS: ALIGNOF Running [...]/hurd/master/ld/testsuite/ld-scripts/assert.exp ... PASS: ASSERT +PASS: ld-scripts/assert2 Running [...]/hurd/master/ld/testsuite/ld-scripts/crossref.exp ... PASS: NOCROSSREFS 1 PASS: NOCROSSREFS 2 @@ -720,6 +725,8 @@ Running [...]/hurd/master/ld/testsuite/ld-scripts/defined.exp ... PASS: DEFINED (PRMS 5699) PASS: ld-scripts/defined2 PASS: ld-scripts/defined3 +PASS: ld-scripts/defined4 +PASS: ld-scripts/defined5 Running [...]/hurd/master/ld/testsuite/ld-scripts/dynamic-sections.exp ... PASS: dynamic sections Running [...]/hurd/master/ld/testsuite/ld-scripts/empty-address.exp ... @@ -735,6 +742,8 @@ Running [...]/hurd/master/ld/testsuite/ld-scripts/empty-orphan.exp ... PASS: ld-scripts/empty-orphan Running [...]/hurd/master/ld/testsuite/ld-scripts/expr.exp ... PASS: ld-scripts/expr1 +PASS: ld-scripts/expr2 +PASS: ld-scripts/sane1 Running [...]/hurd/master/ld/testsuite/ld-scripts/extern.exp ... PASS: EXTERN Running [...]/hurd/master/ld/testsuite/ld-scripts/include.exp ... @@ -873,12 +882,12 @@ Running [...]/hurd/master/ld/testsuite/ld-xtensa/xtensa.exp ... === ld Summary === -# of expected passes 618 +# of expected passes 626 # of expected failures 8 # of untested testcases 6 -/media/data[...]/hurd/master.build/ld/ld-new 2.21.51.20101220 +/media/data[...]/hurd/master.build/ld/ld-new 2.21.51.20110210 -Test Run By thomas on Mon Dec 20 11:34:59 2010 +Test Run By thomas on Thu Feb 10 18:57:49 2011 Native configuration is i686-pc-linux-gnu === gas tests === @@ -960,8 +969,8 @@ PASS: CFI common 2 PASS: CFI common 3 PASS: CFI common 4 PASS: CFI common 5 -PASS: CFI common 7 PASS: CFI common 6 +PASS: CFI common 7 Running [...]/hurd/master/gas/testsuite/gas/cr16/cr16.exp ... Running [...]/hurd/master/gas/testsuite/gas/cr16/pic.exp ... Running [...]/hurd/master/gas/testsuite/gas/cris/cris.exp ... @@ -1000,6 +1009,7 @@ PASS: section flags PASS: DWARF2 1 PASS: DWARF2 2 PASS: DWARF2 3 +PASS: DWARF2 4 PASS: Check bad section flag Running [...]/hurd/master/gas/testsuite/gas/fr30/allinsn.exp ... Running [...]/hurd/master/gas/testsuite/gas/fr30/fr30.exp ... @@ -1093,8 +1103,10 @@ PASS: i386 -mtune=i686 nops 3 PASS: i386 nops 4 PASS: i386 nops -mtune=i386 4 PASS: i386 -mtune=i686 nops 4 +PASS: i386 -march=i686+nop nops 4a PASS: i386 nops 5 PASS: i386 -march=i686 nops 5 +PASS: i386 nops 6 PASS: i386 16-bit addressing in 32-bit mode. PASS: i386 32-bit addressing in 16-bit mode. PASS: i386 SSE4.1 @@ -1175,6 +1187,10 @@ PASS: i386 FMA scalar insns (Intel disassembly) PASS: i386 FMA4 PASS: i386 LWP PASS: i386 XOP +PASS: i386 BMI insns +PASS: i386 BMI insns (Intel disassembly) +PASS: i386 TBM +PASS: i386 TBM insns (Intel disassembly) PASS: i386 F16C PASS: i386 F16C (Intel disassembly) PASS: i386 FSGSBase @@ -1216,6 +1232,10 @@ PASS: i386 list-1 PASS: i386 list-2 PASS: i386 list-3 PASS: DWARF2 debugging information 1 +Running [...]/hurd/master/gas/testsuite/gas/i386/ilp32/cfi/ilp32.exp ... +Running [...]/hurd/master/gas/testsuite/gas/i386/ilp32/elf/ilp32.exp ... +Running [...]/hurd/master/gas/testsuite/gas/i386/ilp32/ilp32.exp ... +Running [...]/hurd/master/gas/testsuite/gas/i386/ilp32/lns/ilp32.exp ... Running [...]/hurd/master/gas/testsuite/gas/i860/i860.exp ... Running [...]/hurd/master/gas/testsuite/gas/ia64/ia64.exp ... Running [...]/hurd/master/gas/testsuite/gas/ieee-fp/x930509a.exp ... @@ -1316,6 +1336,6 @@ Running [...]/hurd/master/gas/testsuite/gas/z8k/z8k.exp ... === gas Summary === -# of expected passes 319 -../as-new 2.21.51.20101220 +# of expected passes 326 +../as-new 2.21.51.20110210 diff --git a/open_issues/binutils_gold.mdwn b/open_issues/binutils_gold.mdwn index f9008154..aa6843a3 100644 --- a/open_issues/binutils_gold.mdwn +++ b/open_issues/binutils_gold.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -11,3 +11,177 @@ License|/fdl]]."]]"""]] [[!tag open_issue_binutils]] Have a look at GOLD / port as needed. + + +# teythoon's try / `mremap` issue + +IRC, #hurd, 2011-01-12 + + <teythoon> I've been looking into building gold on hurd and it built fine + with one minor tweak + <teythoon> and it's working fine according to its test suite + <teythoon> the only problem is that the build system is failing to detect + the hurdish mremap which lives in libmemusage + <teythoon> on linux it is in the libc so the check succeeds + <teythoon> any hints on how to fix this properly? + <antrik> hm... it's strange that it's a different library on the Hurd + <antrik> are the implementations compatible? + <teythoon> antrik: it seems so, though the declarations differ slightly + <antrik> I guess the best thing is to ask on the appropriate list(s) why + they are different... + <teythoon> teythoon@ganymede:~/build/gold/binutils-2.21/gold$ grep -A1 + mremap /usr/include/sys/mman.h + <teythoon> extern void *mremap (void *__addr, size_t __old_len, size_t + __new_len, int __flags, ...) __THROW; + <teythoon> vs + <antrik> of course it would be possible to modify the configure script to + check for the Hurd variant too; but first we should establish whether + here is actually any reason for being different, or it's just some + historical artefact that should be fixed... + <teythoon> teythoon@ganymede:~/build/gold/binutils-2.21/gold$ fgrep 'extern + void *mremap' mremap.c + <teythoon> extern void *mremap (void *, size_t, size_t, int, ...); + <teythoon> the problem is that the test fails to link due to the fact that + mremap isn't in the libc on hurd + <antrik> yeah, it would be possible for the configure script to check + whether it works when the hurdish extra library is added explicitely + <antrik> but again, I don't see any good reason for being different here in + the first place... + <teythoon> so should I create a patch to move mremap? + <antrik> if it's not too complicated, that would be nice... it's always + easier to discuss when you already have code :-) + <antrik> OTOH, asking first might spare you some useless work if it turns + out there *is* some reason for being different after all... + so where is the right place to discuss this? + <antrik> bug-hurd mailing list and/or glibc mailing list. not sure which + one is better -- I guess it doesn't hurt to crosspost... + +[[mailing_lists/libc-alpha]] is the correct list, and cross-posting to +[[mailing_lists/bug-hurd]] would be fine, too. + + <teythoon> antrik: some further digging revealed that mremap belongs to + /lib/libmemusage.so on both hurd and linux + <teythoon> the only difference is that on linux there is a weak reference + to that function in /lib/libc-2.11.2.so + <teythoon> $ objdump -T /lib/libc-2.11.2.so | fgrep mremap + <teythoon> 00000000000cf7e0 w DF .text 0000000000000028 GLIBC_2.2.5 + mremap + <antrik> ah, it's probably simply a bug that we don't have this weak + reference too + <antrik> IIRC we had similar bugs before + <antrik> teythoon: can you provide a patch for that? + <teythoon> antrik: unfortunately I have no idea how that weak ref ended up + there + + <guillem> teythoon: also the libmemusage.s seems to be just a debugging + library to be used by LD_PRELOAD or similar + <guillem> which override those memory functions + <guillem> the libc should provide actual code for those functions, even if + the symbol is declared weak (so overridable) + <guillem> teythoon: are you sure that's the actual problem? can you paste + somewhere the build logs with the error? + <teythoon> guillem: sure + <teythoon> http://paste.debian.net/104437/ + <teythoon> that's the part of config.log that shows the detection (or the + failure to detect it) of mremap + <teythoon> this results in HAVE_MREMAP not being defined + <teythoon> as a consequence it is declared in gold.h and this declaration + conflicts with the one from sys/mman.h http://paste.debian.net/104438/ + <teythoon> on linux the test for mremap succeeds + <guillem> teythoon: hmm oh I guess it's just what that, mremap is linux + specific so it's not available on the hurd + <guillem> teythoon: I just checked glibc and seems to confirm that + <braunr> CONFORMING TO This call is Linux-specific, and should not be used + in programs intended to be portable. + <teythoon> ah okay + <teythoon> so I guess we shouldn't ship an header with that declaration... + <guillem> teythoon: yeah :/ good luck telling that to drepper :) + <guillem> teythoon: I guess he'll suggest that everyone else needs to get + our own copy of sys/mman.h + <guillem> s/our/their/ + <teythoon> hm, so how should I proceed? + <braunr> what's your goal ? + <braunr> detecting mremap ? + <teythoon> making binutils/gold compile ootb on hurd + <teythoon> I picked it from the open issues page ;) + <braunr> well, if there is no mremap, you need a replacement + <teythoon> gold has a replacement + <braunr> ok + <braunr> so your problem is fixing the detection of mremap right ? + <teythoon> yes + <braunr> ok, that's a build system question then :/ + <braunr> you need to ask an autotools guy + <teythoon> well, actually the build system correctly detects the absence of + mremap + <braunr> (gold does use the autotools right ?) + <teythoon> yes + <braunr> oh, i'm lost now (i admit i didn't read the whole issue :/) + <teythoon> it is just that the declaration in sys/mman.h conflicts with + their own declaration + <braunr> ah + <braunr> so in the absence of mremap, they use their own builtin function + <teythoon> yes + <teythoon> and according to the test suite it is working perfectly + <teythoon> gold that is + <teythoon> the declaration in mman.h has an extra __THROW + <guillem> a workaround would be to rename gold's mremap to something else, + gold_mremap for example + <braunr> that's really the kind of annoying issue + <braunr> you either have to change glibc, or gold + <guillem> yeah + <braunr> you'll face difficulty changing glibc, as guillem told you + <guillem> the correct solution though IMO is to fix glibc + <braunr> but this may be true for gold too + <braunr> guillem: i agree + <antrik> maybe it would be easiest actually to implement mremap()?... + <braunr> but as this is something quite linux specific, it makes sense to + use another internal name, and wrap that to the linux mremap if it's + detected + <braunr> antrik: i'm nto sure + <antrik> braunr: I don't think using such workarounds is a good + idea. clearly there would be no issue if the header file wouldn't be + incorrect on Hurd + <braunr> antrik: that's why i said i agree with guillem when he says "the + correct solution though IMO is to fix glibc" + <teythoon> what exactly is the problem with getting a patch into glibc? + <braunr> the people involved + <guillem> teythoon: and touching a generic header file + <braunr> but feel free to try, you could be lucky + <teythoon> but glibc is not an linux specific piece of software, right? + <braunr> teythoon: no, it's not + <guillem> erm... + <braunr> teythoon: but in practice, it is + <guillem> supposedly not :) + <antrik> braunr: BTW, by "easiest" I don't mean coding alone, but + coding+pushing upstream :-) + <guillem> so the problem is, misc/sys/mman.h should be a generic header and + as such not include linux specific parts, which are not present on hurd, + kfreebsd, etc etc + <braunr> antrik: yes, that's why guillem and i suggested the workaround + thing in gold + <antrik> that also requires pushing upstream. and quite frankly, if I were + the gold maintainer, I wouldn't accept it. + <guillem> but the easiest (and wrong) solution in glibc to avoid maintainer + conflict will probably be copying that file under hurd's glibc tree and + install that instead + <braunr> antrik: implementing mremap could be relatively easy to do + actually + <braunr> antrik: IIRC, vm_map() supports overlapping + <antrik> well, actually the easiest solution would be to create a patch + that never goes upstream but is included in Debian, like many + others... but that's obviously not a good long-term plan + <antrik> braunr: yes, I think so too + <antrik> braunr: haven't checked, but I have a vague recollection that the + fundamentals are pretty much there + <antrik> teythoon: so, apart from an ugly workaround in gold, there are + essentially three options: 1. implement mremap; 2. make parts of mman.h + conditional; 3. use our own copy of mman.h + <antrik> 1. would be ideal, but might be non-trivial; 2. would might be + tricky to get right, and even more tricky to get upstream; 3. would be + simple, but a maintenance burden in the long term + <teythoon> looking at golds replacement code (mmap & memcpy) 1 sounds like + the best option performance wise + +[[!taglink open_issue_glibc]]: check if it is possible to implement `mremap`. +[[I|tschwinge]] remember some discussion about this, but have not yet worked on +locating it. [[Talk to me|tschwinge]] if you'd like to have a look at this. diff --git a/open_issues/code_analysis.mdwn b/open_issues/code_analysis.mdwn index ad59f962..21e09089 100644 --- a/open_issues/code_analysis.mdwn +++ b/open_issues/code_analysis.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -40,7 +40,7 @@ analysis|performance]], [[formal_verification]], as well as general * <http://blog.llvm.org/2010/04/whats-wrong-with-this-code.html> - * [[Valgrind]] + * [[community/gsoc/project_ideas/Valgrind]] * [Smatch](http://smatch.sourceforge.net/) diff --git a/open_issues/crash_server.mdwn b/open_issues/crash_server.mdwn index d97f5458..7ed4afbf 100644 --- a/open_issues/crash_server.mdwn +++ b/open_issues/crash_server.mdwn @@ -1,4 +1,5 @@ -[[!meta copyright="Copyright © 2009, 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2010, 2011 Free Software Foundation, +Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -187,3 +188,8 @@ one... /home/tschwinge/tmp/gnumach/gnumach-1-branch-Xen-branch.build/../gnumach-1-branch-Xen-branch/kern/ipc_kobject.c:76 mach_msg_trap /home/tschwinge/tmp/gnumach/gnumach-1-branch-Xen-branch.build/../gnumach-1-branch-Xen-branch/ipc/mach_msg.c:1367 + +--- + +If someone is working in this area, they may want to have a look at +[[GDB_gcore]], and port <http://code.google.com/p/google-coredumper/>, too. diff --git a/open_issues/debian_cross_toolchain.mdwn b/open_issues/debian_cross_toolchain.mdwn new file mode 100644 index 00000000..e0665466 --- /dev/null +++ b/open_issues/debian_cross_toolchain.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +Have a look at the Debian *Cross Toolchain* project, +<https://alioth.debian.org/projects/crosstoolchain/>, +<http://wiki.debian.org/ToolChain/Cross>. diff --git a/open_issues/debugging.mdwn b/open_issues/debugging.mdwn index e66a086f..e5fbf7a0 100644 --- a/open_issues/debugging.mdwn +++ b/open_issues/debugging.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -42,7 +42,10 @@ We have debugging infrastructure. For example: Continues: <http://lwn.net/Articles/414264/>, which introduces <http://dmtcp.sourceforge.net/>. - * [[locking]] + * [[crash_server}}, [[GDB_gcore]], + <http://code.google.com/p/google-coredumper/> + + * [[community/gsoc/project_ideas/libdiskfs_locking]] * <http://lwn.net/Articles/415728/>, or <http://lwn.net/Articles/415471/> -- just two examples; there's a lot of such stuff for Linux. diff --git a/open_issues/ext2fs_page_cache_swapping_leak.mdwn b/open_issues/ext2fs_page_cache_swapping_leak.mdwn new file mode 100644 index 00000000..0ace5cd3 --- /dev/null +++ b/open_issues/ext2fs_page_cache_swapping_leak.mdwn @@ -0,0 +1,23 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, OFTC, #debian-hurd, 2011-03-24 + + <youpi> I still believe we have an ext2fs page cache swapping leak, however + <youpi> as the 1.8GiB swap was full, yet the ld process was only 1.5GiB big + <pinotree> a leak at swapping time, you mean? + <youpi> I mean the ext2fs page cache being swapped out instead of simply + dropped + <pinotree> ah + <pinotree> so the swap tends to accumulate unuseful stuff, i see + <youpi> yes + <youpi> the disk content, basicallyt :) diff --git a/open_issues/sudo_date_crash.mdwn b/open_issues/file_system_exerciser.mdwn index 53303abc..4277e5e7 100644 --- a/open_issues/sudo_date_crash.mdwn +++ b/open_issues/file_system_exerciser.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -8,9 +8,8 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] -[[!tag open_issue_gnumach]] +[[!tag open_issue_hurd]] -IRC, unknown channel, unknown date. +Test our file system implementations with the File System Exerciser. - <grey_gandalf> I did a sudo date... - <grey_gandalf> and the machine hangs + * <http://codemonkey.org.uk/projects/fsx/> diff --git a/open_issues/gdb_gcore.mdwn b/open_issues/gdb_gcore.mdwn index 7d4980f1..69211ac0 100644 --- a/open_issues/gdb_gcore.mdwn +++ b/open_issues/gdb_gcore.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -21,3 +21,6 @@ GDB's `gcore` command doesn't work / needs to be implemented / ported in GDB: /media/data/home/tschwinge/core.cA0ICY:2: Error in sourced command file: Undefined command: "gcore". Try "help". gcore: failed to create core.8371 + +If someone is working in this area, they may want to port +<http://code.google.com/p/google-coredumper/>, too. diff --git a/open_issues/gdb_noninvasive_mode_new_threads.mdwn b/open_issues/gdb_noninvasive_mode_new_threads.mdwn new file mode 100644 index 00000000..9b3992f4 --- /dev/null +++ b/open_issues/gdb_noninvasive_mode_new_threads.mdwn @@ -0,0 +1,15 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gdb]] + +Debugging a translator. `gdb binary`. `set noninvasive on`. `attach [PID]`. +Translator does some work. GDB doesn't notice new threads. `detach`. `attach +[PID]` -- now new threads are visible. diff --git a/open_issues/gdb_thread_ids.mdwn b/open_issues/gdb_thread_ids.mdwn index c31a9967..c04a10ee 100644 --- a/open_issues/gdb_thread_ids.mdwn +++ b/open_issues/gdb_thread_ids.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2008, 2009, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,8 +6,8 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] [[!meta title="GDB: thread ids"]] @@ -23,3 +23,9 @@ GNU GDB's Pedro Alves: Also see [[thread numbering of ps and GDB]]. + +--- + +`attach` to a multi-threaded process. See threads 1 to 5. `detach`. `attach` +again -- thread numbers continue where they stopped last time: now they're +threads 6 to 10. diff --git a/open_issues/gnumach_console_timestamp.mdwn b/open_issues/gnumach_console_timestamp.mdwn new file mode 100644 index 00000000..b36b47b9 --- /dev/null +++ b/open_issues/gnumach_console_timestamp.mdwn @@ -0,0 +1,27 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach]] + +IRC, freenode, #hurd, 2011-02-17 + + <azeem> task 39011c10 deallocating an invalid port 349, most probably a + bug. + <azeem> kernel: Page fault (14), code=6 + <azeem> Stopped at 0x28b9c7: orb %bh,0(%ecx,%edi,2) + <azeem> db> + [...] + <antrik> tschwinge: I doubt the deallocating warning is related to the + later fault + <tschwinge> antrik: YOu may be right. + <tschwinge> Perhaps it'd be a good idea to add some sort of timestamp to + Mach messages. + <tschwinge> Like in Linux' dmesg. + <tschwinge> Or just RDTSC (internal processor counter). diff --git a/open_issues/hurd_build_without_parted.mdwn b/open_issues/hurd_build_without_parted.mdwn new file mode 100644 index 00000000..06ecf56d --- /dev/null +++ b/open_issues/hurd_build_without_parted.mdwn @@ -0,0 +1,16 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +Seen with `cross-gnu`. + +If the *parted* libraries aren't available, we explicitly have to set +`--without-parted` or the build will fail. diff --git a/open_issues/implementing_hurd_on_top_of_another_system.mdwn b/open_issues/implementing_hurd_on_top_of_another_system.mdwn index 23512aa9..95b71ebb 100644 --- a/open_issues/implementing_hurd_on_top_of_another_system.mdwn +++ b/open_issues/implementing_hurd_on_top_of_another_system.mdwn @@ -78,3 +78,40 @@ IRC, #hurd, 2010-12-28 <antrik> though I must say that I'm more and more convinced running the Hurd on top of a monolithic kernel would actually be a useful approach for the time being... + +--- + +IRC, #hurd, 2011-02-11 + + <neal> marcus and I were discussing how to add Mach to Linux + <neal> one could write a module to implement Mach IPC + <neal> and another to implement Mach VM + <neal> the big thing missing with Mach VM is the ability for a tracing + process to easily map or unmap an inferior process's memory + <antrik> neal: why would a tracing process need to map the inferior's + memory? + <neal> the simple answer is that is how it is done on Mach + <antrik> neal: is it? not sure we are talking about the same thing + here. GDB uses vm_read()/vm_write() to access the inferior's memory AFAIK + <neal> on linux? + <neal> I think it use /proc/pid/mem + <antrik> on Hurd + <neal> I'm talking about adding Mach to Linux + <neal> by adding some functionality to Linux + <neal> and then implementing a bunch in user space + <antrik> yeah, but I don't understand the point about mapping inferior's + memory :-( + <antrik> what would be in user space? + <neal> there are a number of different cut points + <neal> one could imagine just using Linux's device drivers, CPU scheduler, + memory management, etc. + <neal> another possibility would be something higher where Hurd processes + just use some Hurdish servers + <antrik> neal: yeah, these are all options I have been considering... too + bad I wasn't able to come to FOSDEM -- I'd love to have participated in + this discussion :-( + <antrik> neal: BTW, were you just discussing this as a hypothetical idea, + or something you are seriously considering? + <neal> I'm unlikely to work on it, sorry + <antrik> didn't really expect that :-) + <antrik> would be nice though if you could write up your conclusions... diff --git a/open_issues/inotify_file_notice_changes.mdwn b/open_issues/inotify_file_notice_changes.mdwn new file mode 100644 index 00000000..a30cd3d3 --- /dev/null +++ b/open_issues/inotify_file_notice_changes.mdwn @@ -0,0 +1,42 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +IRC, freenode, #hurd, 2011-03-28 + +[[!tag open_issue_hurd]] + + <barrucadu> I've been going through the xmlfs code, and plan to have it + monitor the backing store (xml file) for changes and update the presented + directory hierarchy when something is changed; is there a better way to + check a file for changes beyond checking its modification time every few + minutes? + <tschwinge> barrucadu: In the Hurd spirit, you'd use file_notice_changes + (fs.defs). + <barrucadu> thanks + <youpi> we should manage to work out an inotify interface around it, btw + <pochu> like gamin? + <pinotree> imho making gamin work should gain all the fam-using + applications + <pochu> (which, looking at the sources, seems to support hurd already, not + sure why it's not built) + <pinotree> pochu: the hurd_notify of gamin does not build OOTB + <pochu> > /build/buildd/gamin-0.1.10/./libgamin/gam_data.c:476: error: + 'PTHREAD_MUTEX_RECURSIVE' undeclared (first use in this function) + <pinotree> there are few patches in bugzilla to make it compile + <pochu> if they work, and you point me to them, I can upload a new gamin + with them included + <pinotree> #315644, #588337. #605246 + <pinotree> and iirc there's something else i have locally but not send yet + <pochu> please check and submit :) + <pinotree> ah no, those three contains all the build issues + <pinotree> .. plus relative proposed fixes + <pochu> ok, I'll try to get to it soonish + <pinotree> and you should know about two of them already ;D + <pochu> please remind me if I don't :) diff --git a/open_issues/locking.mdwn b/open_issues/locking.mdwn deleted file mode 100644 index 11a10524..00000000 --- a/open_issues/locking.mdwn +++ /dev/null @@ -1,40 +0,0 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_hurd]] - -Every now and then, new locking issues are discovered in -[[hurd/libdiskfs]] or [[hurd/translator/ext2fs]], for example. Nowadays -these in fact seem to be the most often encountered cause of Hurd crashes -/ lockups. - -One of these could be traced -recently, and turned out to be a lock inside [[hurd/libdiskfs]] that was taken -and not released in some cases. There is reason to believe that there are more -faulty paths causing these lockups. - -The task is systematically checking the [[hurd/libdiskfs]] code for this kind of locking -issues. To achieve this, some kind of test harness has to be implemented: For -example instrumenting the code to check locking correctness constantly at -runtime. Or implementing a [[unit testing]] framework that explicitly checks -locking in various code paths. (The latter could serve as a template for -implementing unit tests in other parts of the Hurd codebase...) - -(A [[systematic code review|security]] would probably suffice to find the -existing locking -issues; but it wouldn't document the work in terms of actual code produced, and -thus it's not suitable for a GSoC project...) - -This task requires experience with debugging locking issues in -[[multithreaded|multithreading]] applications. - -Tools have been written for automated [[code analysis]]; these can help to -locate and fix such errors. diff --git a/open_issues/multithreading.mdwn b/open_issues/multithreading.mdwn index 39203333..addc29c3 100644 --- a/open_issues/multithreading.mdwn +++ b/open_issues/multithreading.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,7 +10,21 @@ License|/fdl]]."]]"""]] [[!tag open_issue_hurd]] -Hurd servers / VFS libraries are multithreaded, roughly using one thread per +Hurd servers / VFS libraries are multithreaded. + + +# Implementation + + * well-known threading libraries + + * [[hurd/libthreads]] + + * [[hurd/libpthread]] + + +# Design + +Roughly using one thread per incoming request. This is not the best approach: it doesn't really make sense to scale the number of worker threads with the number of incoming requests, but instead they should be scaled according to the backends' characteristics. @@ -18,7 +32,9 @@ instead they should be scaled according to the backends' characteristics. The [[hurd/Critique]] should have some more on this. -Alternative approaches: +# Alternative approaches: + + * <http://www.concurrencykit.org/> * Continuation-passing style diff --git a/open_issues/nightly_builds.mdwn b/open_issues/nightly_builds.mdwn index 506697bb..5d1257fb 100644 --- a/open_issues/nightly_builds.mdwn +++ b/open_issues/nightly_builds.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -15,6 +15,8 @@ Resources: * [[toolchain/cross-gnu]] + * [[Debian_Cross_Toolchain]] + * As reported in the [[news/2010-05-31]] news, there's Hydra doing nightly builds / Nix packages. diff --git a/open_issues/nightly_builds_deb_packages.mdwn b/open_issues/nightly_builds_deb_packages.mdwn index 9f5e2373..11fc4c79 100644 --- a/open_issues/nightly_builds_deb_packages.mdwn +++ b/open_issues/nightly_builds_deb_packages.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -24,4 +24,8 @@ There is infrastructure available to test whole OS installations. --- +[[Debian_Cross_Toolchain]] for cross-building? + +--- + See also [[nightly_builds]]. diff --git a/open_issues/performance.mdwn b/open_issues/performance.mdwn index 9b3701b3..eb9f3f8a 100644 --- a/open_issues/performance.mdwn +++ b/open_issues/performance.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -18,7 +18,7 @@ In [[microkernel]]-based systems, there is generally a considerable [[RPC]] overhead. In a multi-server system, it is non-trivial to implement a high-performance -[[I/O System|io_system]]. +[[I/O System|community/gsoc/project_ideas/disk_io_performance]]. When providing [[faq/POSIX_compatibility]] (and similar interfaces) in an environemnt that doesn't natively implement these interfaces, there may be a diff --git a/open_issues/performance/fork.mdwn b/open_issues/performance/fork.mdwn index 2748be53..5ceb6455 100644 --- a/open_issues/performance/fork.mdwn +++ b/open_issues/performance/fork.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -14,3 +14,24 @@ Our [[`fork` implementation|glibc/fork]] is nontrivial. To do: hard numbers. [[Microbenchmarks]]? + + +# Windows / Cygwin + + * <http://www.google.com/search?q=cygwin+fork> + + * <http://www.redhat.com/support/wpapers/cygnus/cygnus_cygwin/architecture.html> + + In particular, *5.6. Process Creation*. + + * <http://archive.gamedev.net/community/forums/topic.asp?topic_id=360290> + + * <http://cygwin.com/cgi-bin/cvsweb.cgi/src/winsup/cygwin/how-cygheap-works.txt?cvsroot=src> + + > Cygwin has recently adopted something called the "cygwin heap". This is + > an internal heap that is inherited by forked/execed children. It + > consists of process specific information that should be inherited. So + > things like the file descriptor table, the current working directory, and + > the chroot value live there. + + * <http://www.perlmonks.org/?node_id=588994> diff --git a/open_issues/performance/io_system.mdwn b/open_issues/performance/io_system.mdwn deleted file mode 100644 index 0d41d3c7..00000000 --- a/open_issues/performance/io_system.mdwn +++ /dev/null @@ -1,49 +0,0 @@ -[[!meta copyright="Copyright © 2008, 2009, 2010 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] - -[[!meta title="I/O System"]] - -[[!tag open_issue_hurd]] - -The most obvious reason for the Hurd feeling slow compared to mainstream -systems like GNU/Linux, is a low I/O system performance, in particular very -slow hard disk access. - -The reason for this slowness is lack and/or bad implementation of common -optimization techniques, like scheduling reads and writes to minimize head -movement; effective block caching; effective reads/writes to partial blocks; -reading/writing multiple blocks at once; and read-ahead. The -[[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some -optimizations at a higher logical level. - -The goal of this project is to analyze the current situation, and implement/fix -various optimizations, to achieve significantly better disk performance. It -requires understanding the data flow through the various layers involved in -disk access on the Hurd ([[filesystem|hurd/virtual_file_system]], -[[pager|hurd/libpager]], driver), and general experience with -optimizing complex systems. That said, the killing feature we are definitely -missing is the read-ahead, and even a very simple implementation would bring -very big performance speedups. - -Here are some real testcases: - - * [[binutils_ld_64ksec]]; - - * running the Git testsuite which is mostly I/O bound; - - * use [[TopGit]] on a non-toy repository. - - -Possible mentors: Samuel Thibault (youpi) - -Exercise: Look through all the code involved in disk I/O, and try something -easy to improve. It's quite likely though that you will find nothing obvious -- -in this case, please contact us about a different exercise task. diff --git a/open_issues/performance/io_system/binutils_ld_64ksec.mdwn b/open_issues/performance/io_system/binutils_ld_64ksec.mdwn index b59a87a7..79c2300f 100644 --- a/open_issues/performance/io_system/binutils_ld_64ksec.mdwn +++ b/open_issues/performance/io_system/binutils_ld_64ksec.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,7 +10,8 @@ License|/fdl]]."]]"""]] [[!tag open_issue_hurd]] -This one may be considered as a testcase for I/O system optimization. +This one may be considered as a testcase for [[I/O system +optimization|community/gsoc/project_ideas/disk_io_performance]]. It is taken from the [[binutils testsuite|binutils]], `ld/ld-elf/sec64k.exp`, where this diff --git a/open_issues/performance/io_system/clustered_page_faults.mdwn b/open_issues/performance/io_system/clustered_page_faults.mdwn new file mode 100644 index 00000000..37433e06 --- /dev/null +++ b/open_issues/performance/io_system/clustered_page_faults.mdwn @@ -0,0 +1,105 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach open_issue_hurd]] + +[[community/gsoc/project_ideas/disk_io_performance]]. + +IRC, freenode, #hurd, 2011-02-16 + + <braunr> exceptfor the kernel, everything in an address space is + represented with a VM object + <braunr> those objects can represent anonymous memory (from malloc() or + because of a copy-on-write) + <braunr> or files + <braunr> on classic Unix systems, these are files + <braunr> on the Hurd, these are memory objects, backed by external pagers + (like ext2fs) + <braunr> so when you read a file + <braunr> the kernel maps it from ext2fs in your address space + <braunr> and when you access the memory, a fault occurs + <braunr> the kernel determines it's a region backed by ext2fs + <braunr> so it asks ext2fs to provide the data + <braunr> when the fault is resolved, your process goes on + <etenil> does the faul occur because Mach doesn't know how to access the + memory? + <braunr> it occurs because Mach intentionnaly didn't back the region with + physical memory + <braunr> the MMU is programmed not to know what is present in the memory + region + <braunr> or because it's read only + <braunr> (which is the case for COW faults) + <etenil> so that means this bit of memory is a buffer that ext2fs loads the + file into and then it is remapped to the application that asked for it + <braunr> more or less, yes + <braunr> ideally, it's directly written into the right pages + <braunr> there is no intermediate buffer + <etenil> I see + <etenil> and as you told me before, currently the page faults are handled + one at a time + <etenil> which wastes a lot of time + <braunr> a certain amount of time + <etenil> enough to bother the user :) + <etenil> I've seen pages have a fixed size + <braunr> yes + <braunr> use the PAGE_SIZE macro + <etenil> and when allocating memory, the size that's asked for is rounded + up to the page size + <etenil> so if I have this correctly, it means that a file ext2fs provides + could be split into a lot of pages + <braunr> yes + <braunr> once in memory, it is managed by the page cache + <braunr> so that pages more actively used are kept longer than others + <braunr> in order to minimize I/O + <etenil> ok + <braunr> so a better page cache code would also improve overall performance + <braunr> and more RAM would help a lot, since we are strongly limited by + the 768 MiB limit + <braunr> which reduces the page cache size a lot + <etenil> but the problem is that reading a whole file in means trigerring + many page faults just for one file + <braunr> if you want to stick to the page clustering thing, yes + <braunr> you want less page faults, so that there are less IPC between the + kernel and the pager + <etenil> so either I make pages bigger + <etenil> or I modify Mach so it can check up on a range of pages for faults + before actually processing + <braunr> you *don't* change the page size + <etenil> ah + <etenil> that's hardware isn't it? + <braunr> in Mach, yes + <etenil> ok + <braunr> and usually, you want the page size to be the CPU page size + <etenil> I see + <braunr> current CPU can support multiple page sizes, but it becomes quite + hard to correctly handle + <braunr> and bigger page sizes mean more fragmentation, so it only suits + machines with large amounts of RAM, which isn't the case for us + <etenil> ok + <etenil> so I'll try the second approach then + <braunr> that's what i'd recommand + <braunr> recommend* + <etenil> ok + +--- + +IRC, freenode, #hurd, 2011-02-16 + + <antrik> etenil: OSF Mach does have clustered paging BTW; so that's one + place to start looking... + <antrik> (KAM ported the OSF code to gnumach IIRC) + <antrik> there is also an existing patch for clustered paging in libpager, + which needs some adaptation + <antrik> the biggest part of the task is probably modifying the Hurd + servers to use the new interface + <antrik> but as I said, KAM's code should be available through google, and + can serve as a starting point + +<http://lists.gnu.org/archive/html/bug-hurd/2010-06/msg00023.html> diff --git a/open_issues/performance/io_system/read-ahead.mdwn b/open_issues/performance/io_system/read-ahead.mdwn new file mode 100644 index 00000000..b6851edd --- /dev/null +++ b/open_issues/performance/io_system/read-ahead.mdwn @@ -0,0 +1,301 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach open_issue_hurd]] + +[[community/gsoc/project_ideas/disk_io_performance]] + +IRC, #hurd, freenode, 2011-02-13: + + <etenil> youpi: Would libdiskfs/diskfs.h be in the right place to make + readahead functions? + <youpi> etenil: no, it'd rather be at the memory management layer, + i.e. mach, unfortunately + <youpi> because that's where you see the page faults + <etenil> youpi: Linux also provides a readahead() function for higher level + applications. I'll probably have to add the same thing in a place that's + higher level than mach + <youpi> well, that should just be hooked to the same common implementation + <etenil> the man page for readahead() also states that portable + applications should avoid it, but it could be benefic to have it for + portability + <youpi> it's not in posix indeed + +--- + +IRC, #hurd, freenode, 2011-02-14: + + <etenil> youpi: I've investigated prefetching (readahead) techniques. One + called DiskSeen seems really efficient. I can't tell yet if it's patented + etc. but I'll keep you informed + <youpi> don't bother with complicated techniques, even the most simple ones + will be plenty :) + <etenil> it's not complicated really + <youpi> the matter is more about how to plug it into mach + <etenil> ok + <youpi> then don't bother with potential pattents + <antrik> etenil: please take a look at the work KAM did for last year's + GSoC + <youpi> just use a trivial technique :) + <etenil> ok, i'll just go the easy way then + + <braunr> antrik: what was etenil referring to when talking about + prefetching ? + <braunr> oh, madvise() stuff + <braunr> i could help him with that + +--- + +[[Etenil]] is now working in this area. + +--- + +IRC, freenode, #hurd, 2011-02-15 + + <etenil> oh, I'm looking into prefetching/readahead to improve I/O + performance + <braunr> etenil: ok + <braunr> etenil: that's actually a VM improvement, like samuel told you + <etenil> yes + <braunr> a true I/O improvement would be I/O scheduling + <braunr> and how to implement it in a hurdish way + <braunr> (or if it makes sense to have it in the kernel) + <etenil> that's what I've been wondering too lately + <braunr> concerning the VM, you should look at madvise() + <etenil> my understanding is that Mach considers devices without really + knowing what they are + <braunr> that's roughly the interface used both at the syscall() and the + kernel levels in BSD, which made it in many other unix systems + <etenil> whereas I/O optimisations are often hard disk drives specific + <braunr> that's true for almost any kernel + <braunr> the device knowledge is at the driver level + <etenil> yes + <braunr> (here, I separate kernels from their drivers ofc) + <etenil> but Mach also contains some drivers, so I'm going through the code + to find the apropriate place for these improvements + <braunr> you shouldn't tough the drivers at all + <braunr> touch + <etenil> true, but I need to understand how it works before fiddling around + <braunr> hm + <braunr> not at all + <braunr> the VM improvement is about pagein clustering + <braunr> you don't need to know how pages are fetched + <braunr> well, not at the device level + <braunr> you need to know about the protocol between the kernel and + external pagers + <etenil> ok + <braunr> you could also implement pageout clustering + <etenil> if I understand you well, you say that what I'd need to do is a + queuing system for the paging in the VM? + <braunr> no + <braunr> i'm saying that, when a page fault occurs, the kernel should + (depending on what was configured through madvise()) transfer pages in + multiple blocks rather than one at a time + <braunr> communication with external pagers is already async, made through + regular ports + <braunr> which already implement message queuing + <braunr> you would just need to make the mapped regions larger + <braunr> and maybe change the interface so that this size is passed + <etenil> mmh + <braunr> (also don't forget that page clustering can include pages *before* + the page which caused the fault, so you may have to pass the start of + that region too) + <etenil> I'm not sure I understand the page fault thing + <etenil> is it like a segmentation error? + <etenil> I can't find a clear definition in Mach's manual + <braunr> ah + <braunr> it's a fundamental operating system concept + <braunr> http://en.wikipedia.org/wiki/Page_fault + <etenil> ah ok + <etenil> I understand now + <etenil> so what's currently happening is that when a page fault occurs, + Mach is transfering pages one at a time and wastes time + <braunr> sometimes, transferring just one page is what you want + <braunr> it depends on the application, which is why there is madvise() + <braunr> our rootfs, on the other hand, would benefit much from such an + improvement + <braunr> in UVM, this optimization is account for around 10% global + performance improvement + <braunr> accounted* + <etenil> not bad + <braunr> well, with an improved page cache, I'm sure I/O would matter less + on systems with more RAM + <braunr> (and another improvement would make mach support more RAM in the + first place !) + <braunr> an I/O scheduler outside the kernel would be a very good project + IMO + <braunr> in e.g. libstore/storeio + <etenil> yes + <braunr> but as i stated in my thesis, a resource scheduler should be as + close to its resource as it can + <braunr> and since mach can host several operating systems, I/O schedulers + should reside near device drivers + <braunr> and since current drivers are in the kernel, it makes sens to have + it in the kernel too + <braunr> so there must be some discussion about this + <etenil> doesn't this mean that we'll have to get some optimizations in + Mach and have the same outside of Mach for translators that access the + hardware directly? + <braunr> etenil: why ? + <etenil> well as you said Mach contains some drivers, but in principle, it + shouldn't, translators should do disk access etc, yes? + <braunr> etenil: ok + <braunr> etenil: so ? + <etenil> well, let's say if one were to introduce SATA support in Hurd, + nothing would stop him/her to do so with a translator rather than in Mach + <braunr> you should avoid the term translator here + <braunr> it's really hurd specific + <braunr> let's just say a user space task would be responsible for that + job, maybe multiple instances of it, yes + <etenil> ok, so in this case, let's say we have some I/O optimization + techniques like readahead and I/O scheduling within Mach, would these + also apply to the user-space task, or would they need to be + reimplemented? + <braunr> if you have user space drivers, there is no point having I/O + scheduling in the kernel + <etenil> but we also have drivers within the kernel + <braunr> what you call readahead, and I call pagein/out clustering, is + really tied to the VM, so it must be in Mach in any case + <braunr> well + <braunr> you either have one or the other + <braunr> currently we have them in the kernel + <braunr> if we switch to DDE, we should have all of them outside + <braunr> that's why such things must be discussed + <etenil> ok so if I follow you, then future I/O device drivers will need to + be implemented for Mach + <braunr> currently, yes + <braunr> but preferrably, someone should continue the work that has been + done on DDe so that drivers are outside the kernel + <etenil> so for the time being, I will try and improve I/O in Mach, and if + drivers ever get out, then some of the I/O optimizations will need to be + moved out of Mach + <braunr> let me remind you one of the things i said + <braunr> i said I/O scheduling should be close to their resource, because + we can host several operating systems + <braunr> now, the Hurd is the only system running on top of Mach + <braunr> so we could just have I/O scheduling outside too + <braunr> then you should consider neighbor hurds + <braunr> which can use different partitions, but on the same device + <braunr> currently, partitions are managed in the kernel, so file systems + (and storeio) can't make good scheduling decisions if it remains that way + <braunr> but that can change too + <braunr> a single storeio representing a whole disk could be shared by + several hurd instances, just as if it were a high level driver + <braunr> then you could implement I/O scheduling in storeio, which would be + an improvement for the current implementation, and reusable for future + work + <etenil> yes, that was my first instinct + <braunr> and you would be mostly free of the kernel internals that make it + a nightmare + <etenil> but youpi said that it would be better to modify Mach instead + <braunr> he mentioned the page clustering thing + <braunr> not I/O scheduling + <braunr> theseare really two different things + <etenil> ok + <braunr> you *can't* implement page clustering outside Mach because Mach + implements virtual memory + <braunr> both policies and mechanisms + <etenil> well, I'd rather think of one thing at a time if that's alright + <etenil> so what I'm busy with right now is setting up clustered page-in + <etenil> which need to be done within Mach + <braunr> keep clustered page-outs in mind too + <braunr> although there are more constraints on those + <etenil> yes + <etenil> I've looked up madvise(). There's a lot of documentation about it + in Linux but I couldn't find references to it in Mach (nor Hurd), does it + exist? + <braunr> well, if it did, you wouldn't be caring about clustered page + transfers, would you ? + <braunr> be careful about linux specific stuff + <etenil> I suppose not + <braunr> you should implement at least posix options, and if there are + more, consider the bsd variants + <braunr> (the Mach VM is the ancestor of all modern BSD VMs) + <etenil> madvise() seems to be posix + <braunr> there are system specific extensions + <braunr> be careful + <braunr> CONFORMING TO POSIX.1b. POSIX.1-2001 describes posix_madvise(3) + with constants POSIX_MADV_NORMAL, etc., with a behav‐ ior close to that + described here. There is a similar posix_fadvise(2) for file access. + <braunr> MADV_REMOVE, MADV_DONTFORK, MADV_DOFORK, MADV_HWPOISON, + MADV_MERGEABLE, and MADV_UNMERGEABLE are Linux- specific. + <etenil> I was about to post these + <etenil> ok, so basically madvise() allows tasks etc. to specify a usage + type for a chunk of memory, then I could apply the relevant I/O + optimization based on this + <braunr> that's it + <etenil> cool, then I don't need to worry about knowing what the I/O is + operating on, I just need to apply the optimizations as advised + <etenil> that's convenient + <etenil> ok I'll start working on this tonight + <etenil> making a basic readahead shouldn't be too hard + <braunr> readahead is a misleading name + <etenil> is pagein better? + <braunr> applies to too many things, doesn't include the case where + previous elements could be prefetched + <braunr> clustered page transfers is what i would use + <braunr> page prefetching maybe + <etenil> ok + <braunr> you should stick to something that's already used in the + literature since you're not inventing something new + <etenil> yes I've read a paper about prefetching + <etenil> ok + <etenil> thanks for your help braunr + <braunr> sure + <braunr> you're welcome + <antrik> braunr: madvise() is really the least important part of the + picture... + <antrik> very few applications actually use it. but pretty much all + applications will profit from clustered paging + <antrik> I would consider madvise() an optional goody, not an integral part + of the implementation + <antrik> etenil: you can find some stuff about KAM's work on + http://www.gnu.org/software/hurd/user/kam.html + <antrik> not much specific though + <etenil> thanks + <antrik> I don't remember exactly, but I guess there is also some + information on the mailing list. check the archives for last summer + <antrik> look for Karim Allah Ahmed + <etenil> antrik: I disagree, madvise gives me a good starting point, even + if eventually the optimisations should run even without it + <antrik> the code he wrote should be available from Google's summer of code + page somewhere... + <braunr> antrik: right, i was mentioning madvise() because the kernel (VM) + interface is pretty similar to the syscall + <braunr> but even a default policy would be nice + <antrik> etenil: I fear that many bits were discussed only on IRC... so + you'd better look through the IRC logs from last April onwards... + <etenil> ok + + <etenil> at the beginning I thought I could put that into libstore + <etenil> which would have been fine + + <antrik> BTW, I remembered now that KAM's GSoC application should have a + pretty good description of the necessary changes... unfortunately, these + are not publicly visible IIRC :-( + +--- + +IRC, freenode, #hurd, 2011-02-16 + + <etenil> braunr: I've looked in the kernel to see where prefetching would + fit best. We talked of the VM yesterday, but I'm not sure about it. It + seems to me that the device part of the kernel makes more sense since + it's logically what manages devices, am I wrong? + <braunr> etenil: you are + <braunr> etenil: well + <braunr> etenil: drivers should already support clustered sector + read/writes + <etenil> ah + <braunr> but yes, there must be support in the drivers too + <braunr> what would really benefit the Hurd mostly concerns page faults, so + the right place is the VM subsystem + +[[clustered_page_faults]] diff --git a/open_issues/perlmagick.mdwn b/open_issues/perlmagick.mdwn index 1daac62b..8a57a8fd 100644 --- a/open_issues/perlmagick.mdwn +++ b/open_issues/perlmagick.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -57,6 +57,49 @@ Etc. +/usr/lib/gcc/i486-gnu/4.4.2/include/omp.h: +# State as of 2011-03-06 + +freenode, #hurd channel, 2011-03-06: + + <pinotree> tschwinge: (speaking on working perl, how did it en with that + "(glibc) double free" crash with perl?) + <pinotree> *end + <tschwinge> I think I remember I suspected it's a libgomp (!) issue in the + end. I have not yet continued working on that. + <pinotree> libogmp? looks like you know more than me, then :) + <youpi> tschwinge: oh, I'm interested + <youpi> I know a bit about libgomp :) + <tschwinge> I bisected this down to where Imagemagick added -fgomp (or + whatever it is). And then the perl library (Imagemagick.pm?) which loads + the imagemagick.so segfaulted. + <tschwinge> ImageMagick did this change in the middle of a x.x.x.something + release.. + <tschwinge> My next step would have been to test whether libgomp works at + all for us. + <youpi> ./usr/sbin/debootstrap:DEBOOTSTRAP_CHECKSUM_FIELD="SHA$SHA_SIZE" + <youpi> erf + <youpi> so they switched to another checksum + <youpi> but we don't have that one on all of our packages :) + <youpi> tschwinge: + <youpi> buildd@bach:~$ OMP_NUM_THREADS=2 ./test + <youpi> I'm 0x1 + <youpi> I'm 0x3 + <youpi> libgomp works at least a bit + <tschwinge> OK. + <pinotree> i guess we should hope the working bits don't stop at that point + ;) + <tschwinge> If open_issues/perlmagick is to be believed a diff of 6.4.1-1 + and 6.4.1-2 should tell what exactly was changed. + <tschwinge> Oh! + <tschwinge> I even have it on the page already! ;-) + <tschwinge> -fopenmp + <youpi> I've tried the pragmas that imagemagick uses + <youpi> they work + <tschwinge> Might be the issue fixed itself? + <youpi> I don't know, it's the latest libc here + <youpi> (and latest hurd, to be uploaded) + + # Other [[!debbug 551017]] diff --git a/open_issues/pfinet.mdwn b/open_issues/pfinet.mdwn new file mode 100644 index 00000000..8782fe08 --- /dev/null +++ b/open_issues/pfinet.mdwn @@ -0,0 +1,18 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, #hurd + +pfinet explosion + + <antrik> I reproduce it with freeciv client connected to a remote X server + <antrik> it suffices to run freeciv-gtk2, and clicking "new game" diff --git a/open_issues/pfinet_vs_system_time_changes.mdwn b/open_issues/pfinet_vs_system_time_changes.mdwn new file mode 100644 index 00000000..714c8784 --- /dev/null +++ b/open_issues/pfinet_vs_system_time_changes.mdwn @@ -0,0 +1,42 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, unknown channel, unknown date. + + <grey_gandalf> I did a sudo date... + <grey_gandalf> and the machine hangs + +This was very likely a misdiagnosis: + +IRC, freenode, #hurd, 2011-03-25 + + <tschwinge> antrik: I suspect it'S some timing stuff in pfinet that perhaps + uses absolute time, and somehow wildely gets confused? + <antrik> tschwinge: BTW, pfinet doesn't actually die I think -- it just + drops open connections... + <antrik> perhaps it thinks they timed out + <tschwinge> antrik: Isn't the translator restarted instead? + <antrik> don't think so + <antrik> when pfinet actually dies, I also loose the NFS mounts, which + doesn't happen in this case + <antrik> hehe "... and the machine hangs" + <antrik> he didn't bother to check that the machine is perfectly fine, only + the SSH connection got dropped + <tschwinge> Ah, I see. So it'S perhaps indeed simply closes TCP + connections that have been without data for ``too long''? + <antrik> yeah, that's my guess + <antrik> my clock is speeding, so ntpdate sets it in the past + <antrik> perhaps there is some math that concludes the connection have been + inactive for -200 seconds, which (unsigned) is more than any timeout :-) + <tschwinge> (The other way round, you might likely get some integer + wrap-around, and thus the same result.) + <tschwinge> Yes. diff --git a/open_issues/pflocal_socket_credentials_for_local_sockets.mdwn b/open_issues/pflocal_socket_credentials_for_local_sockets.mdwn new file mode 100644 index 00000000..5a71412e --- /dev/null +++ b/open_issues/pflocal_socket_credentials_for_local_sockets.mdwn @@ -0,0 +1,42 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +IRC, freenode, #hurd, 2011-03-28 + +[[!tag open_issue_hurd]] + + <pinotree> basically, i'm trying to implement socket credentials for local + sockets, and i guessed doing it in pflocal would be the appropriate place + <pinotree> what i thought was filling the cmsg data for MSG_CRED at + S_socket_recv() call + <pinotree> in case i missed it, would there be a way to "identify" the + other side of the port associated to the sock_user of that call? + <pochu> pinotree: that's needed by dbus right? cool! (and I don't know) + <pinotree> (yes, and gamin) + <youpi> pinotree: you have them already, they're just not stored + <youpi> see S_io_reauthenticate + <youpi> Throw away the ids we went through all that trouble to get... + <youpi> (comment) + * pinotree looks + <pinotree> hm, and who calls that rpc? + <youpi> everybody + <youpi> since that's how ext2fs knows the permission to apply, for instance + <pinotree> ah, i was referring to the reauthenticate of pflocal, not + auth_server_authenticate() + <youpi> that's what I'm saying + <youpi> see __hurd_file_name_lookup_retry, which is the very internal part + of open() + <youpi> it calls io_reauthenticate() + <youpi> to authenticate itself to the underlying translator of the opened + node + <pinotree> youpi: so, hm, could be an option make the result of pflocal's + S_io_reauthenticate cached in the sock_user struct? + <youpi> yes + <pinotree> nice thanks, i will try that change first diff --git a/open_issues/profiling.mdwn b/open_issues/profiling.mdwn index f56ae974..7e3c7350 100644 --- a/open_issues/profiling.mdwn +++ b/open_issues/profiling.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -21,6 +21,8 @@ done for [[performance analysis|performance]] reasons. Have a look at this, integrate it into the main trees. - * <http://fosdem.org/2010/interview/mark-wielaard> + * [[LTTng]] - ... or some other Linux thing. + * [[SystemTap]] + + * ... or some other Linux thing. diff --git a/open_issues/rm_fr.mdwn b/open_issues/rm_fr.mdwn new file mode 100644 index 00000000..89a803ab --- /dev/null +++ b/open_issues/rm_fr.mdwn @@ -0,0 +1,27 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +From: Samuel Thibault <samuel.thibault@gnu.org> +Subject: rm -fr slowness + +I have always been surprised by the slowness of a mere rm -fr. Looking a +bit inside, I see that diskfs_dirremove_hard() calls diskfs_file_update +(dp, 1) (as does diskfs_truncate, diskfs_direnter_hard, and +diskfs_dirrewrite_hard). diskfs_file_update then calls pager_sync on +the pager, which thus writes back the whole ext2fs pager! + +This sounds a bit excessive to me, an unlink could just record it in +memory and actually sync later. Also, the wait flag is set, so we +really waits for all I/Os, which basically means strictly serializing +file removals: remove one file, wait for the disk to have done it +(~10ms), remove the next one, etc. I guess this is for safety reasons +against crashes, but isn't the sync option there for such kind of diff --git a/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn b/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn new file mode 100644 index 00000000..9db92250 --- /dev/null +++ b/open_issues/rpc_to_self_with_rendez-vous_leading_to_duplicate_port_destroy.mdwn @@ -0,0 +1,163 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +[RPC to self with rendez-vous leading to duplicate port +destroy](http://lists.gnu.org/archive/html/bug-hurd/2011-03/msg00045.html) + +IRC, freenode, #hurd, 2011-03-14 + + <antrik> youpi: I wonder, why does the root FS call diskfs_S_dir_lookup() + at all?... + <youpi> errr, because a client asked for it? + <youpi> (problem with RPCs is you can't easily know where they come from :) + ) + <youpi> (especially when it's the root fs...) + <antrik> ah, it's about a client request... didn't see that + <youpi> well, I just said "is called", yes + <antrik> I do not really understand though why it tries to reauthenticate + against itself... + <antrik> I fear my memory of the lookup mechanism grew a bit dim + <youpi> see the source + <youpi> it's about a translated entry + <antrik> (and I never fully understood some aspects anyways...) + <youpi> it needs to start the translated entry as another user, possibly + <antrik> yes, but a translated entry normally would be served by *another* + process?... + <youpi> sure, but ext2fs has to prepare it + <youpi> thus reauthenticate to prepare the correct set of rights + <antrik> prepare what? + <youpi> rights + <youpi> so the process is not root, doesn't have / opened as root, etc. + <antrik> rights for what? + <youpi> err, about everything + <antrik> IIRC the reauthentication is done by the parent FS on the port to + the *translated* node + <antrik> and the translated node should be a different process?... + <youpi> that's not what I read in the source + <youpi> fshelp_fetch_root + <youpi> ports[INIT_PORT_CRDIR] = reauth (getcrdir ()); + <youpi> here, getcrdir() returns ext2fs itself + <antrik> well, perhaps the issue is that I have no idea what + fshelp_fetch_root() does, nor why it is called here... + <youpi> it notably starts the translator that dir_lookup is looking at, if + needed + <youpi> possibly as a different user, thus reauthentication of CRDIR + <antrik> so this is about a port that is passed to the translator being + started? + <youpi> no + <youpi> well, depends on what you mean by "port" + <youpi> it's about reauthenticating a port to be passed to the translator + being started + <youpi> and for that a rendez-vous port is needed for the reauthentication + <youpi> and that's the one at stake + <antrik> yeah, I meant the port that is reauthenticated + <antrik> what is CRDIR? + <youpi> current root dir ... + <antrik> so the parent translator passes it's own root dir to the child + translator; and the issue is that for the root FS the root dir points to + the root FS itself... + <youpi> yes + <antrik> OK, that makes sense + <youpi> (but that's only one example, rgrep mach_port_destroy hurd/ show + other potential issues) + <antrik> well, that's actually what I wanted to mention next... why is the + rendez-vous port destroyed, instead of just deallocating the port right + and letting reference counting to it's thing?... + <antrik> do its thing + <youpi> "just to make sure" I guess + <antrik> it's pretty obvious that this will cause trouble for any RPC + referencing itself... + <youpi> well, follow-up with that on the list + <youpi> with roland/tb in CC + <youpi> only they would know any real reason for destroy + <youpi> btw, if you knew how we could make _hurd_select()'s raw __mach_msg + call be interruptible by signals, that'll permit to fix sudo + <youpi> (damn, I need sleep, my tenses are all wrong) + <antrik> BTW, does this cause any actual trouble?... + <antrik> I don't know much about interruption... cfhammer might have a + better idea, he look into that stuff quite a bit AIUI + <antrik> looked + <antrik> (hehe, it's not only your tenses... guess there's something in the + ether ;-) ) + <youpi> it makes sudo, mailq, etc. fail sometimes + <antrik> I mean the rendez-vous thing + <youpi> that's it, yes + <youpi> sudo etc. fail at least due to this + <antrik> so these are two different problems that both affect sudo? + <antrik> (rendez-vous and interruption I mean) + <youpi> yes + <youpi> with my patch the buildds have much fewer issues, but still some + <youpi> (my interrupt-related patch) + <youpi> I'm installing a s/destroy/deallocate/ version of ext2fs on the + buildds, we'll see how it behaves + <youpi> (it fixes my testcase at least) + <antrik> interrupt-related patch? + <antrik> only thing interrupt-related I remember was the reauthentication + race... + <youpi> that's what I mean + <antrik> well, cfhammer investigated this is quite some depth, explaining + quite well why the race is only mitigated but still exists... problem is + that we didn't know how to fix it properly + <antrik> because nobody seems to understand the cancellation code, except + perhaps for Roland and Thomas + <antrik> (and I'm not even entirely sure about them :-) ) + <antrik> I think his findings and our conclusions are documented on the + ML... + <youpi> by "much fewer issues", I mean that some of the symptoms have + disappeared, others haven't + <antrik> BTW, couldn't the rendez-vous thing be worked around by simply + ignoring the errors from the failing deallocate?... + <youpi> no, failing deallocate are actually dangerous + <antrik> why? + <youpi> since the name might have been reused for something else in the + meanwhile + <youpi> that's the whole point of the warning I had added in the kernel + itself + <antrik> I see + <youpi> such things really deserve tracking, since they can have any kind + of consequence + <antrik> does Mach try to reuse names quickly, rather than only after + wrapping around?... + <youpi> it seems to + <antrik> OK, then this is a serious problem indeed + <youpi> (note: I rarely divine issues when there aren't actual frequent + symptoms :) ) + <antrik> well, the problem with the warning is that it only shows in the + cases that do *not* cause a problem... so it's hard to associate them + with any specific issues + <youpi> well, most of the time the port is not reused quickly enough + <youpi> so in most case it shows up more often than causing problem + +IRC, freenode, #hurd, 2011-03-14 + + <youpi> ok, mach_port_deallocate actually can't be used + <youpi> since mach_reply_port() returns a receive right, not a send right + * youpi guesses he will really have to manage to understand all that port + stuff completely + <antrik> oh, right + <antrik> youpi: hm... now I'm confused though. if one client holds a + receive right, the other client (or in this case the same process) should + have a send or send-once right -- these should *not* share the same name + in my understanding + <antrik> destroying the receive right should turn the send right into a + dead name + <antrik> so unless I'm missing something, the destroy shouldn't be a + problem, and there must be something else going wrong + <antrik> hm... actually I'm probably wrong + <antrik> yeah, definitely wrong. receive rights and "ordinary" send rights + share the name. only send-once rights are special + <antrik> I wonder whether the problem could be worked around by using a + send-once right... + <antrik> mach_port_mod_refs(mach_task_self(), name, + MACH_PORT_RIGHT_RECEIVE, -1) can be used to deallocate only the receive + right + <antrik> oh, you already figured that out :-) diff --git a/open_issues/secure_file_descriptor_handling.mdwn b/open_issues/secure_file_descriptor_handling.mdwn index 1a514e69..45e983a7 100644 --- a/open_issues/secure_file_descriptor_handling.mdwn +++ b/open_issues/secure_file_descriptor_handling.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -12,7 +12,8 @@ License|/fdl]]."]]"""]] `O_CLOEXEC`, `dup3` et al.; see <http://udrepper.livejournal.com/20407.html>. [[tschwinge]] once worked -on this, posted patches to libc-alpha. This works needs to be resumed +on this, posted patches to [[mailing_lists/libc-alpha]]. This works needs to +be resumed and finished. --- diff --git a/open_issues/sync_but_still_unclean_filesystem.mdwn b/open_issues/sync_but_still_unclean_filesystem.mdwn index f1fbb4e0..83c7951e 100644 --- a/open_issues/sync_but_still_unclean_filesystem.mdwn +++ b/open_issues/sync_but_still_unclean_filesystem.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,9 +10,19 @@ License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] +Also filed as [[!GNU_Savannah_bug 29292]]. + \#hurd, 2010, end of May / beginning of June [runnign sync, but sill unclean filesystem at next boot] - <slpz> guillem: when libpager syncs an object, it sends an m_o_lock_request and waits (if the synchronous argument was specified) for a m_o_lock_completed. But m_o_lock_completed only means that dirty pages have been sent to the translator, and this one still needs to write them to the backing storage - <slpz> guillem: there's no problem if sync() returns before actually writting the changes to disk, but this also happens when shutting down the translator - <slpz> guillem: in theory, locking mechanisms in libpager should prevent this from happening by keeping track of write operations, but this seems to fail in some situations + <slpz> guillem: when libpager syncs an object, it sends an m_o_lock_request + and waits (if the synchronous argument was specified) for a + m_o_lock_completed. But m_o_lock_completed only means that dirty pages + have been sent to the translator, and this one still needs to write them + to the backing storage + <slpz> guillem: there's no problem if sync() returns before actually + writting the changes to disk, but this also happens when shutting down + the translator + <slpz> guillem: in theory, locking mechanisms in libpager should prevent + this from happening by keeping track of write operations, but this seems + to fail in some situations diff --git a/open_issues/system_initialization.mdwn b/open_issues/system_initialization.mdwn new file mode 100644 index 00000000..9048b615 --- /dev/null +++ b/open_issues/system_initialization.mdwn @@ -0,0 +1,24 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +IRC, freenode, #hurd, 2011-03-30 + + <kilobug> init=/bin/sh hack doesn't work for GNU/Hurd ? + <antrik> kilobug: I don't think you can override init on Hurd. the init + server is actually involved in bootstrapping part of the system core + <antrik> at some point we discussed the possibility to reduce the Hurd init + server to *only* do that, and then pass on to standard sysv init... with + that it could actually work + +--- + + * [[systemd]], etc. diff --git a/open_issues/unit_testing.mdwn b/open_issues/unit_testing.mdwn index 1cf7cfb8..a3dd9c18 100644 --- a/open_issues/unit_testing.mdwn +++ b/open_issues/unit_testing.mdwn @@ -8,6 +8,11 @@ Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] +This task may be suitable for [[community/GSoC]]: +[[community/gsoc/project_ideas/testing_framework]] + +--- + A collection of thoughts with respect to unit testing. We definitely want to add unit test suites to our code base. @@ -27,6 +32,14 @@ abandoned). * The [[glibc_testsuite]] has a home-grown system (Makefile-based), likewise does the [[Open_POSIX_Test_Suite]]. + * [Kyua](http://code.google.com/p/kyua/) (and its predecessor [ATF](http://www.NetBSD.org/~jmmv/atf/)). + + * Primarily used by NetBSD as its testing framework; FreeBSD is in the process of adopting it. + + * Provides bindings to write tests in C, C++ and POSIX shell. Lua is planned. + + * Builds and runs on many different Unix-based operating systems. + * [check](http://check.sourceforge.net/) * used by some GNU packages, for example GNU PDF (Jose E. Marchesi) @@ -54,6 +67,8 @@ abandoned). Developers*](http://lwn.net/Articles/412302/) by Steven Rostedt, 2010-10-28. [v2](http://lwn.net/Articles/414064/), 2010-11-08. + * <http://autotest.kernel.org/wiki/WhitePaper> + # Related @@ -66,3 +81,14 @@ abandoned). testing, too? * <http://ltp.sourceforge.net/> + + * [LaBrea](https://github.com/dustin/labrea/wiki), or similar tools can be + used for modelling certain aspects of system behavior (long response times, + for example). + + +# Discussion + +See the [[GSoC project idea|community/gsoc/project_ideas/testing_framework]]'s +[[discussion +subpage|community/gsoc/project_ideas/testing_framework/discussion]]. diff --git a/open_issues/valgrind.mdwn b/open_issues/valgrind.mdwn deleted file mode 100644 index 2b0624d7..00000000 --- a/open_issues/valgrind.mdwn +++ /dev/null @@ -1,80 +0,0 @@ -[[!meta copyright="Copyright © 2009, 2010 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!meta title="Porting Valgrind to the Hurd"]] - -[Valgrind](http://valgrind.org/) is an extremely useful debugging tool for memory errors. -(And some other kinds of hard-to-find errors too.) -Aside from being useful for program development in general, -a Hurd port will help finding out why certain programs segfault on the Hurd, -although they work on Linux. -Even more importantly, it will help finding bugs in the Hurd servers themselfs. - -To keep track of memory use, -Valgrind however needs to know how each [[system call]] affects the validity of memory regions. -This knowledge is highly kernel-specific, -and thus Valgrind needs to be explicitely ported for every system. - -Such a port involves two major steps: -making Valgrind understand how kernel traps work in general on the system in question; -and how all the individual kernel calls affect memory. -The latter step is where most of the work is, -as the behaviour of each single [[system call]] needs to be described. - -Compared to Linux, -[[microkernel/Mach]] (the microkernel used by the Hurd) has very few kernel traps. -Almost all [[system call]]s are implemented as [[RPC]]s instead -- -either handled by Mach itself, or by the various [[Hurd servers|hurd/translator]]. -All RPCs use a pair of `mach_msg` invocations: -one to send a request message, and one to receive a reply. -However, while all RPCs use the same `mach_msg` trap, -the actual effect of the call varies greatly depending on which RPC is invoked -- -similar to the `ioctl` call on Linux. -Each request thus must be handled individually. - -Unlike `ioctl`, -the RPC invocations have explicit type information for the parameters though, -which can be retrieved from the message header. -By analyzing the parameters of the RPC reply message, -Valgrind can know exactly which memory regions are affected by that call, -even without specific knowledge of the RPC in question. -Thus implementing a general parser for the reply messages -will already give Valgrind a fairly good approximation of memory validity -- -without having to specify the exact semantic of each RPC by hand. - -While this should make Valgrind quite usable on the Hurd already, it's not perfect: -some RPCs might return a buffer that is only partially filled with valid data; -or some reply parameters might be optional, -and only contain valid data under certain conditions. -Such specific semantics can't be deduced from the message headers alone. -Thus for a complete port, -it will still be necessary to go through the list of all known RPCs, -and implement special handling in Valgrind for those RPCs that need it. - -The goal of this task is at minimum to make Valgrind grok Mach traps, -and to implement the generic RPC handler. -Ideally, specific handling for RPCs needing it should also be implemented. - -Completing this project will require digging into Valgrind's handling of [[system call]]s, -and into Hurd RPCs. -It is not an easy task, but a fairly predictable one -- -there shouldn't be any unexpected difficulties, -and no major design work is necessary. -It doesn't require any specific previous knowledge: -only good programming skills in general. -On the other hand, -the student will obtain a good understanding of Hurd RPCs while working on this task, -and thus perfect qualifications for Hurd development in general :-) - -Possible mentors: Samuel Thibault (youpi) - -Exercise: As a starter, -students can try to teach valgrind a couple of Linux ioctls, -as this will make them learn how to use the read/write primitives of valgrind. |