diff options
| -rw-r--r-- | faq/how_many_developers/discussion.mdwn | 20 | ||||
| -rw-r--r-- | glibc/discussion.mdwn | 25 | ||||
| -rw-r--r-- | hurd/running/qemu.mdwn | 3 | ||||
| -rw-r--r-- | hurd/running/virtualbox.mdwn | 6 | ||||
| -rw-r--r-- | hurd/settrans/discussion.mdwn | 18 | ||||
| -rw-r--r-- | microkernel/mach/gnumach/debugging.mdwn | 15 | ||||
| -rw-r--r-- | microkernel/mach/gnumach/memory_management.mdwn | 30 | ||||
| -rw-r--r-- | microkernel/mach/rpc/discussion.mdwn | 117 | ||||
| -rw-r--r-- | open_issues/debugging.mdwn | 2 | ||||
| -rw-r--r-- | open_issues/debugging_gnumach_startup_qemu_gdb.mdwn | 116 | ||||
| -rw-r--r-- | open_issues/gnumach_constants.mdwn | 32 | ||||
| -rw-r--r-- | open_issues/gnumach_kernel_threads.mdwn | 23 | ||||
| -rw-r--r-- | open_issues/gnumach_memory_management.mdwn | 65 | ||||
| -rw-r--r-- | open_issues/placement_of_virtual_memory_regions.mdwn | 89 | ||||
| -rw-r--r-- | open_issues/rework_gnumach_ipc_spaces.mdwn | 197 | ||||
| -rw-r--r-- | open_issues/translate_fd_or_port_to_file_name.mdwn | 36 | ||||
| -rw-r--r-- | open_issues/xattr.mdwn | 27 |
17 files changed, 801 insertions, 20 deletions
diff --git a/faq/how_many_developers/discussion.mdwn b/faq/how_many_developers/discussion.mdwn new file mode 100644 index 00000000..9fc44f25 --- /dev/null +++ b/faq/how_many_developers/discussion.mdwn @@ -0,0 +1,20 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +IRC, freenode, #hurd, 2011-07-09 + + <antrik> gnu_srs1: regarding your question why people aren't interested in + workin on Hurd: Eric Raymond explains it pretty well in his famous + "Cathedral and Bazaar" paper + <antrik> people are more likely to work on something that *almost* works + for them, and where they only have to fill in a few missing bits + <antrik> the Hurd doesn't almost work for anyone + <antrik> actually, you should probably reread the whole paper. it's + essentially an analysis why the Hurd failed compared to Linux diff --git a/glibc/discussion.mdwn b/glibc/discussion.mdwn new file mode 100644 index 00000000..17b4fb32 --- /dev/null +++ b/glibc/discussion.mdwn @@ -0,0 +1,25 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + + +# TLS + + +## IRC, freenode, #hurd, 2011-06-11 + +[[!tag open_issue_documentation]] + + <civodul> youpi: local-tls-support.diff removes libc-tsd.h; what's the + rationale? + <youpi> it's completely replaced by __thread variables + <civodul> ok, but apparently there are still libc headers that #include it + <civodul> like malloc-machine.h + <youpi> they'll include bits/libc-tsd.h instead + <civodul> oh, ok diff --git a/hurd/running/qemu.mdwn b/hurd/running/qemu.mdwn index 4766b2be..9f085d24 100644 --- a/hurd/running/qemu.mdwn +++ b/hurd/running/qemu.mdwn @@ -60,6 +60,9 @@ Check if your CPU supports kvm: Do not enable kernel-kqemu, as that assumes some particular behavior from the guest kernel, which we are reluctant to artificially add to gnumach. +If QEMU with KVM is not available, [[Virtualbox]] reportedly has better +performance. + #### If you have hardware support (recommended): $ apt-get install qemu-kvm $ modprobe kvm diff --git a/hurd/running/virtualbox.mdwn b/hurd/running/virtualbox.mdwn index 02ab88e7..0731b8d6 100644 --- a/hurd/running/virtualbox.mdwn +++ b/hurd/running/virtualbox.mdwn @@ -20,3 +20,9 @@ supported. The network controller should be configured as PCnet-PCI II or PCNet-FAST III for instance. INTEL PRO or Paravirtualized Network do not work. + + +# Performance + +If [[QEMU with KVM|qemu]] is not available, VirtualBox reportedly has better +performance. diff --git a/hurd/settrans/discussion.mdwn b/hurd/settrans/discussion.mdwn new file mode 100644 index 00000000..c9ec4d34 --- /dev/null +++ b/hurd/settrans/discussion.mdwn @@ -0,0 +1,18 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation open_issue_hurd]] + + +# IRC, freenode, #hurd, 2011-06-01 + + <antrik> ugh... I just realized why settrans -a without -f doesn't + generally work on filesystem translators + <antrik> obviously, it needs -R too! diff --git a/microkernel/mach/gnumach/debugging.mdwn b/microkernel/mach/gnumach/debugging.mdwn index 3a93c6ad..2f52adf8 100644 --- a/microkernel/mach/gnumach/debugging.mdwn +++ b/microkernel/mach/gnumach/debugging.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2007, 2008, 2009 Free Software Foundation, +[[!meta copyright="Copyright © 2007, 2008, 2009, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -6,8 +6,8 @@ id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] Mach has a built-in kernel debugger. [Manual](http://www.gnu.org/software/hurd/gnumach-doc/Kernel-Debugger.html). @@ -67,3 +67,12 @@ The call of `halt_cpu` will -- as the name suggests -- halt the system afterwards. This might be what you want or it might not, but it is needed at some place when running the kernel inside QEMU, as QEMU somehow decides not to update its display buffer anymore under certain conditions. + + +IRC, freenode, #hurd, 2011-07-14: + + <braunr> one ugly trick i use when printf isn't available is to halt the + cpu + <braunr> then use info registers to know where the cpu is halted + <braunr> and you'll know if you reached that code or not + <braunr> (info registers is a qemu command) diff --git a/microkernel/mach/gnumach/memory_management.mdwn b/microkernel/mach/gnumach/memory_management.mdwn index 17c7ad79..43b99d83 100644 --- a/microkernel/mach/gnumach/memory_management.mdwn +++ b/microkernel/mach/gnumach/memory_management.mdwn @@ -50,3 +50,33 @@ IRC, freenode, #hurd, 2011-02-16 <braunr> antrik: but you're right, since mach uses a direct mapped kernel space, the true problem is the lack of linux-like highmem support <braunr> which isn't required if the kernel space is really virtual + + +--- + +IRC, freenode, #hurd, 2011-06-09 + + <braunr> btw, how can gnumach use 1 GiB of RAM ? did you lower the + user/kernel boundary address ? + <youpi> I did + <braunr> 2G ? + <youpi> yes + <braunr> ok + <youpi> it doesn't make so much sense to let processes have 3G addressing + space when there can't be more that 1G physical memory + <braunr> that's sad for an operating system which does most things by + mapping memory eh + <youpi> well, if a process wants to map crazy things, 3G may be tight + already + <youpi> e.g. ext2fs + <braunr> yes + <youpi> so there's little point in supporting them + <braunr> we need hurd/amd64 + <youpi> and there's quite some benefit in shrinking them to 2G + <youpi> yes + <youpi> actually even 2G may become a bit tight + <youpi> webkit linking needs about 1.5-2GiB + <youpi> things become really crazy + <braunr> wow + <braunr> i remember the linux support for 4G/4G split when there was enough + RAM to fill the kernel space with struct page entries diff --git a/microkernel/mach/rpc/discussion.mdwn b/microkernel/mach/rpc/discussion.mdwn new file mode 100644 index 00000000..00e4a012 --- /dev/null +++ b/microkernel/mach/rpc/discussion.mdwn @@ -0,0 +1,117 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + + +# IRC, freenode, #hurd, 2011-06-11 + + <antrik> I don't think we have a precendence case of Mach initiating RPCs + to userspace tasks + <braunr> well mach regularly sends RPCs to external pagers + <antrik> hm, right + <antrik> anyways, the ds_ in device.defs is for use *inside* Mach, not for + the userspace interface + <braunr> what makes you think so ? + <antrik> several things + <antrik> not least the fact that without zhengda's modifications, the + device handling never calls out to userspace for all I know + <braunr> hm, it does + <braunr> for async I/O + <braunr> when the kernel has finished its I/O, it calls + ds_device_read_reply/ds_device_write_reply + <antrik> I see + <antrik> I never quite understood the _reply stuff + <braunr> although i wonder how mig is supposed to forge those names + <antrik> braunr: it isn't + <antrik> braunr: there is a separate device_reply.defs + <antrik> braunr: and it sets a *userprefix* of ds_ + <antrik> rather than a serverprefix + <braunr> i saw, yes + <braunr> ah right + <antrik> so ds still refers to the in-Mach device server, not anything + userspace + <braunr> so this is where the patch is supposed to introduce the + device_intr_notify RPC + <antrik> or at least that's my understanding... + <braunr> no, it doesn't refer to in-mach servers + <braunr> it really forges the right rpcs to be called by mach + <antrik> the definition of "RPC" is rather unclear here + <braunr> why ? + <braunr> mach has its own mach_msg() call for kernel-to-user messaging + <antrik> yes, but this is used only to send the reply message for the RPC + earlier initiated by userspace AIUI + <antrik> it doesn't look like there is any special RPC for async I/O + <braunr> yes, because this is the only use case they had + <braunr> hence the name "reply" + <braunr> intr_notify isn't a reply, but it uses the same mechanism + <braunr> these are declared as simpleroutine + <antrik> sure. but the fact that it isn't a reply message, but rather + initiates a new RPC, changes things from MiG point of view I believe + <antrik> right, as there is no reply to the reply :-) + <braunr> :) + <braunr> a simpleroutine is how to turn an rpc into a simple ipc + <antrik> I know + <antrik> so in _reply, we pretend that the reply is actually a new RPC, + with server and client roles reversed, and no reply + <antrik> (this is actually rather kludgy... apparently MIG has no real + notion of async replies) + <braunr> i don't understand what you mean + <braunr> simpleroutine is the explicit solution for async replies + <braunr> as stated in + http://www.cs.cmu.edu/afs/cs/project/mach/public/doc/unpublished/mig.ps + <braunr> it's not a new rpc with roles reversed + <braunr> it's not a reply either + <antrik> it might be an explicit solution for that, but it still seems + kludgy :-) + <braunr> i don't see why :/ + <braunr> would you have expected something like an option to create both + sync and async versions ? + <antrik> because it requires an extra .defs file + <antrik> yes + <braunr> ok + <braunr> well this seems cumbersome to me :) + <braunr> i prefer the simpleroutine approach + <braunr> but i agree this seems odd since mach has a high level ipc api + <antrik> anyways, my point is that the ds_ in device_reply.defs still + refers to the Mach side of things + <braunr> npnth: which package fails to build ? + <antrik> though a userspace process that actually handles the replies in an + async fashion will of course need some kind of device server too, just + like the DDE stuff... + <antrik> though naming it ds_ is confusing IMHO, because of the name clash + with the device server in Mach + <braunr> hm again, i fail to see why + <braunr> ds_ just means device_server + <braunr> and as most things in mach, it can be in kernel or not + <braunr> i mean, this is an interface prefix, i don't refer to an actual + single instance of a "device server" out there + <antrik> oh, right... DDE implements the Mach device protocol, so it *does* + do the ds_ part... but that makes the interrupt notification stuff even + more confusing + <braunr> hm + <braunr> because it provides a ds_device_intr_notify() which will never be + used, just to completely implement the interface ? + <antrik> yeah, that's what I suspect... + <braunr> sounds likely + <antrik> the device interface actually has two parts: one for "generic" + RPCs on the master device port, and one for device-specific RPCs. DDE + implements the latter, and uses the former... + <antrik> they live in separate places though I think: the individual device + RPCs are implemented in libmachdev, while the intr_ stuff is used in + libddekit probably + <braunr> it would be hairy to build otherwise + <antrik> so we *really* need to know what component npnth gets the error + with + <antrik> braunr: nah, not really. that's why we always have a separate + prefix for the server routines in Hurd RPCs + <braunr> right, i really need to read about mig again + <antrik> it's pretty normal for a translator to both implement and use an + interface diff --git a/open_issues/debugging.mdwn b/open_issues/debugging.mdwn index e5fbf7a0..b2d49b26 100644 --- a/open_issues/debugging.mdwn +++ b/open_issues/debugging.mdwn @@ -49,3 +49,5 @@ We have debugging infrastructure. For example: * <http://lwn.net/Articles/415728/>, or <http://lwn.net/Articles/415471/> -- just two examples; there's a lot of such stuff for Linux. + + * [[debugging_gnumach_startup_QEMU_GDB]] diff --git a/open_issues/debugging_gnumach_startup_qemu_gdb.mdwn b/open_issues/debugging_gnumach_startup_qemu_gdb.mdwn new file mode 100644 index 00000000..e3a6b648 --- /dev/null +++ b/open_issues/debugging_gnumach_startup_qemu_gdb.mdwn @@ -0,0 +1,116 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Debugging GNU Mach's startup in QEMU with GDB"]] + +[[!tag open_issue_gdb open_issue_gnumach]] + + +# IRC, freenode, #hurd, 2011-07-14 + + <mcsim> Hello. I have problem with debugging gnumach. I set 2 brakepoints + in file i386/i386at/model_dep.c on functions gdt_init and idt_init. Then + I start qemu with patched gnumach kernel and stop at gdt_init. When I + enter command "continue" in gdb, qemu hangs. But when I go step by step, + using command "next", I freely reach idt_init. What can cause this + problem? + <braunr> hm + <braunr> not sure + <braunr> let me try + <braunr> mcsim: works for me :/ + <mcsim> it works without my patch, but with it qemu hangs + <braunr> oh, i thought it worked when not using continue + <mcsim> with my patch I can reach idt_init only using next + <mcsim> and without all works fine + <braunr> mcsim: are you sure you correctly built it with debugging symbols + ? + <mcsim> I've written in /etc/dpkg/buildflags.conf SET CFLAGS -g3 -O0 + <braunr> hm + <braunr> i have internal kvm errors actually + <braunr> mcsim: do you use kvm ? + <braunr> mcsim: and why break on those functions ? + <braunr> i'm not sure the address space is already fine at this point + <mcsim> no. I don't have hardware virtualisation support. + <braunr> hm actually, you won't be able to use gdb + <braunr> i just remembered how gnumach is linked and mapped :/ + <braunr> the addresses in the elf image are low addresses, matching the + image as it is loaded by the boot loader + <mcsim> I was wondering why qemu hangs. + <braunr> then the kernel uses segmentation to map itself at 2 (or 3 + previously) GiB + <braunr> well, if the addresses are wrong, your breakpoints are wrong + <braunr> i even wonder how it can work when stepping + <braunr> i don't have the issue with x15 because of its linker script + <mcsim> Are there any ways of such debugging without qemu? + <braunr> i don't think so + <braunr> as antrik told you, the in kernel debugger needs many services + running before being usable + <braunr> you'll have to use printf, and there may be steps during bootstrap + when even that isn't available + <mcsim> So I need computer with hardware virtualisation? + <braunr> well, of course stepping works, since the breakpoints are relative + <braunr> no + <braunr> kvm has nothing to do with the problem + <braunr> it's just that the problem appears differently with kvm enabled + <mcsim> ok. thank you. + <braunr> good luck + <antrik> braunr: would it be hard to "fix" gnumach to avoid the + segmentation magic?... + <braunr> antrik: because of the linux drivers, it may + <antrik> or alternatively, implement something in GDB to deal with that?... + <braunr> antrik: i didn't study that part enough to know for sure + <antrik> uhm... why would the Linux drivers depend on that? does Linux also + do such magic?... + <braunr> well it should simply be a matter of shifting the address by a + fixed offset + <braunr> antrik: linux drivers rely on physical memory being allocated + through kmalloc + <braunr> so there must be a direct mapping between virtual kernel memory + and physical memory + <braunr> they don't specifically need that segmentation settings + <braunr> so if you remove the offset implemented through segmentation, you + have to replace it with page mapping + <braunr> and i don't know how much needs to be done for that + <braunr> you also need to link the kernel differently + <antrik> hm, OK + <antrik> so adding GDB support for the offset would probably be easier... + <braunr> yes + <braunr> but using the offset must only be done once segmentation is set up + <braunr> so you must break after gdt_init + <braunr> not on it + <braunr> mcsim: why do you break on these functions btw ? + <mcsim> I just wanted to find out why qemu hangs + <braunr> yes but why those ? + <mcsim> I found out that before gdt_init all workes fine, but after qemu + hangs. So idt_init is just the next function + <braunr> ok + <braunr> and does your patch change something to how segmentation is + initialized ? + <mcsim> now + <mcsim> no + <braunr> try to build it with the regular cflags + <braunr> i don't know if gnumach can work with -O0 + <mcsim> I've tried. But all the same + <mcsim> Regular are -g -O2 + <braunr> can you make your patch available ? + <mcsim> yes + <mcsim> it is available in gnumach repository at savannah + <mcsim> tree mplaneta/libbraunr/master + <antrik> well, if the segmentation stuff is the thing GDB has problems + with, I don't see how it can work without your patch... + <braunr> without ? + <antrik> well, the patch shouldn't affect the segmentation... so I don't + see how it can make a difference + <braunr> he said qemu hanged + <braunr> so let's not introduce gdb yet + <braunr> qemu can hang for other reasons + <antrik> oh, right, without GDB... + <antrik> though if that's what he meant, his statement was very misleading + at least diff --git a/open_issues/gnumach_constants.mdwn b/open_issues/gnumach_constants.mdwn new file mode 100644 index 00000000..16c8cf41 --- /dev/null +++ b/open_issues/gnumach_constants.mdwn @@ -0,0 +1,32 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach]] + +At compile-time, GNU Mach is parameterized with several constants. These might +need some tuning. Debian has some patches. + + +# IRC, freenode, #hurd, 2011-06-09 + + <braunr> youpi: in ipc/ipc_hash.c, there is code which computes the size of + the global (space, port)->entry hash table + <braunr> youpi: you may be interested in tuning this one too + <youpi> I know + <braunr> ok + <youpi> the current value is not so bad + <youpi> it's big enough for buildds to run fine + <braunr> 256 if i'm right + <braunr> well + <braunr> it won't fail + <youpi> we're limited by the 4000 object limitation anyway + <braunr> since it's a chained hash table + <braunr> but increasing it may help performances a bit + <braunr> and it certainly can't hurt much diff --git a/open_issues/gnumach_kernel_threads.mdwn b/open_issues/gnumach_kernel_threads.mdwn new file mode 100644 index 00000000..9591986b --- /dev/null +++ b/open_issues/gnumach_kernel_threads.mdwn @@ -0,0 +1,23 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach]] + +IRC, freenode, #hurd, 2011-07-13 + + <braunr> jkoenig: why does gnumach appear as "root=device:hd0s1" in ps ? + <jkoenig> braunr, it's the closest we can do to its command line + <braunr> doesn't it deserve something special like kernel threads in linux + ? + <braunr> so that it's actually clear that it's a special task/process + <jkoenig> you mean something like [mach root=device:hd0s1] ? + <braunr> something like that yes + <braunr> also, it would be nice if gnumach threads could actually be seen, + i don't remember if the mach interface allows it though diff --git a/open_issues/gnumach_memory_management.mdwn b/open_issues/gnumach_memory_management.mdwn index a5dd6955..f15c4d25 100644 --- a/open_issues/gnumach_memory_management.mdwn +++ b/open_issues/gnumach_memory_management.mdwn @@ -12,7 +12,10 @@ License|/fdl]]."]]"""]] There is a [[!FF_project 266]][[!tag bounty]] on this task. -IRC, freenode, #hurd, 2011-04-12: +[[!toc]] + + +# IRC, freenode, #hurd, 2011-04-12 <antrik> braunr: do you think the allocator you wrote for x15 could be used for gnumach? and would you be willing to mentor this? :-) @@ -597,7 +600,8 @@ IRC, freenode, #hurd, 2011-04-12: <mcsim> It would be better to rewrite it using debugfs, but when I was writing this test I didn't know about trace_* macros -2011-04-15 + +# IRC, freenode, #hurd, 2011-04-15 <mcsim> There is a hack in zone_gc when it allocates and frees two vm_map_kentry_zone elements to make sure the gc will be able to allocate @@ -632,7 +636,8 @@ IRC, freenode, #hurd, 2011-04-12: <braunr> i think that's why i have "sources" in my slab allocator, the default source (vm_kern) and a custom one for kernel map entries -2011-04-18 + +# IRC, freenode, #hurd, 2011-04-18 <mcsim> braunr: you've said that once page is completely free, it is returned to the vm. @@ -640,7 +645,8 @@ IRC, freenode, #hurd, 2011-04-12: <braunr> mcsim: i also said i was wrong about that <braunr> zone_gc is the only one -2011-04-19 + +# IRC, freenode, #hurd, 2011-04-19 <braunr> antrik: mcsim: i added back a new per-cpu layer as planned <braunr> @@ -649,7 +655,8 @@ IRC, freenode, #hurd, 2011-04-12: loops, just as in zone_gc, to reduce contention and avoid deadlocks <braunr> this is really common in memory allocators -2011-04-23 + +# IRC, freenode, #hurd, 2011-04-23 <mcsim> I've looked through some allocators and all of them use different per cpu cache policy. AFAIK gnuhurd doesn't support multiprocessing, but @@ -661,7 +668,8 @@ IRC, freenode, #hurd, 2011-04-12: <antrik> I'm not sure I suggested that explicitly to you; but probably it makes most sense to use that in gnumach -2011-04-24 + +# IRC, freenode, #hurd, 2011-04-24 <mcsim> antrik: Yes, I have. He uses both global and per cpu caches. But he also suggested to look through slqb, where there are only per cpu @@ -773,7 +781,8 @@ IRC, freenode, #hurd, 2011-04-12: <braunr> the fact that the kernel allocator uses virtual memory doesn't mean the kernel has no mean to allocate contiguous physical memory ... -2011-05-02 + +# IRC, freenode, #hurd, 2011-05-02 <braunr> hm nice, my allocator uses less memory than glibc (squeeze version) on both 32 and 64 bits systems @@ -822,7 +831,8 @@ IRC, freenode, #hurd, 2011-04-12: <braunr> it could also be used to drop the overloaded (and probably over imbalanced) page cache hash table -2011-05-03 + +# IRC, freenode, #hurd, 2011-05-03 <mcsim> antrik: How should I start porting? Have I just include rbraun's allocator to gnumach and make it compile? @@ -845,3 +855,42 @@ IRC, freenode, #hurd, 2011-04-12: <braunr> there is work to be done ;) <braunr> (not to mention the obvious about replacing all the calls to the zone allocator, and testing/debugging afterwards) + + +# IRC, freenode, #hurd, 2011-07-14 + + <braunr> can you make your patch available ? + <mcsim> it is available in gnumach repository at savannah + <mcsim> tree mplaneta/libbraunr/master + <braunr> mcsim: i'll test your branch + <mcsim> ok. I'll give you a link in a minute + <braunr> hm why balloc ? + <mcsim> Braun's allocator + <braunr> err + <braunr> + http://git.sceen.net/rbraun/x15mach.git/?a=blob;f=kern/kmem.c;h=37173fa0b48fc9d7e177bf93de531819210159ab;hb=HEAD + <braunr> mcsim: this is the interface i had in mind for a kernel version :) + <braunr> very similar to the original slab allocator interface actually + <braunr> well, you've been working + <mcsim> But I have a problem with this patch. When I apply it to gnumach + code from debian repository. I have to make a change in file ramdisk.c + with sed -i 's/kernel_map/\&kernel_map/' device/ramdisk.c + <mcsim> because in git repository there is no such file + <braunr> mcsim: how do you configure the kernel before building ? + <braunr> mcsim: you should keep in touch more often i think, so that you + get feedback from us and don't spend too much time "off course" + <mcsim> I didn't configure it. I just run dpkg-buildsource -b. + <braunr> oh you build the debian package + <braunr> well my version was by configure --enable-kdb --enable-rtl8139 + <braunr> and it seems stuck in an infinite loop during bootstrap + <mcsim> and printf doesn't work. The first function called by c_boot_entry + is printf(version). + <braunr> mcsim: also, you're invited to get the x15mach version of my + files, which are gplv2+ licensed + <braunr> be careful of my macros.h file, it can conflict with the + macros_help.h file from gnumach iirc + <mcsim> There were conflicts with MACRO_BEGIN and MACRO_END. But I solved + it + <braunr> ok + <braunr> it's tricky + <braunr> mcsim: try to find where the first use of the allocator is made diff --git a/open_issues/placement_of_virtual_memory_regions.mdwn b/open_issues/placement_of_virtual_memory_regions.mdwn new file mode 100644 index 00000000..95b9e545 --- /dev/null +++ b/open_issues/placement_of_virtual_memory_regions.mdwn @@ -0,0 +1,89 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach]] + +IRC, freenode, #hurd, 2011-07-13 + + <braunr> does anyone know if posix (or mach) has requirements or a policy + about the placement of allocations of virtual space ? + <braunr> a policy such as bottom-up ? + <braunr> or "find lowest vailable space" ? + <jkoenig> braunr, you mean for vm_allocate ? You may want to check mmap() + but I can't remember ever coming across such a thing (except maybe + wrt. alignment) + <braunr> i was wondering how e.g. libraries are linked near the stack + (possibly at slightly random addresses) + <braunr> does the linker walk the address space entries top-down ? + <braunr> jkoenig: i didn't see anything either in the mach interface, but i + may have missed something + <braunr> jkoenig: most systems i've been studying mark the vm regions for + the heap and the stack + <braunr> but for mach, the stack is just allocated virtual memory at the + top of the space + <braunr> so the "placement policy" is either completely outside the kernel, + or relies on its interface + <jkoenig> braunr, actually I'm surprised Mach would even dictate where the + (one?) stack should be, I would have expected it to be the job of + whatever creates a thread to make this kind of choice + <braunr> jkoenig: threads have their own stacks, under the responsibility + of the user trhead implementation + <braunr> but a program usually needs a stack even before it runs + <braunr> i had to set one to bootstrap modules in v0.1 + <braunr> but i wonder if it's just for bootstrapping (and then propagated + by fork()) or part of the interface + <braunr> but this doesn't matter much actually, the allocation mechanism i + have in mind can actually support multiple policies + <jkoenig> I would guess the former (just for bootstrapping), since a new + task has no thread, and a new thread has no state. (but I'm no expert) + <braunr> i think so + <braunr> i'll have a look at the exec server + <braunr> jkoenig: did the previous implementation of procfs show task maps + ? + <jkoenig> braunr, I don't think so, I would probably have felt compelled to + include them in the new one if it did :-) + <braunr> hmmm + <braunr> we definitely need that + <jkoenig> is there a compelling use case you think about in particular? + <braunr> yes + <braunr> i failed to understand how gnumach behaved wrt ipc right spaces + +[[rework_gnumach_ipc_spaces]] + + <braunr> and when i did, i found out my work was impossible to integrate + <jkoenig> "ipc right spaces" ? + <braunr> each task have an ipc space, which contains righs + <braunr> rights + <braunr> the ipc translation layer converts space/name and space/port + tuples to rights + <braunr> i wanted to replace the splay tree with a radix tree but didn't + get how the ipc table made the splay tree almost unused + <braunr> i don't want to make this kind of mistake again, so i'd like a + clear and detailed view of the vm spaces + <braunr> (it's only compelling for myself, all right) + <braunr> but + <braunr> we have vminfo + <braunr> rbraun@nordrassil:~$ vminfo $$ | wc -l + <braunr> 1046 + <braunr> oh my + <braunr> in comparison, a firefox instance has less than 500 on linux + <jkoenig> you mean there's some kind of port name table (or functional + equivalent) which actually resides in the task's memory? (and that's what + shows up at the beginning of the address space with prot=0?) + <braunr> jkoenig: sorry for being confusing, it's not that at all + <jkoenig> (btw feel free to tell me to just go read the source or whatever) + <braunr> jkoenig: don't worry + <jkoenig> braunr, no problem + <braunr> jkoenig: i just compared a previous attempt to improve gnumach + which failed because i didn't have enough insight of the inner workings + of the kernel + <braunr> jkoenig: i really want to miss as little as possible on the vm + part, so having detailed information about what actually happens on + running hurd systems is something i need diff --git a/open_issues/rework_gnumach_ipc_spaces.mdwn b/open_issues/rework_gnumach_ipc_spaces.mdwn index f5d40abd..b3d1b4a4 100644 --- a/open_issues/rework_gnumach_ipc_spaces.mdwn +++ b/open_issues/rework_gnumach_ipc_spaces.mdwn @@ -10,7 +10,10 @@ License|/fdl]]."]]"""]] [[!tag open_issue_gnumach]] -IRC, freenode, #hurd, 2011-05-07 +[[!toc] + + +# IRC, freenode, #hurd, 2011-05-07 <braunr> things that are referred to as "system calls" in glibc are actually RPCs to the kernel or other tasks, those RPCs have too lookup @@ -20,7 +23,8 @@ IRC, freenode, #hurd, 2011-05-07 There is a [[!FF_project 268]][[!tag bounty]] on this task. -IRC, freenode, #hurd, 2011-04-23 + +# IRC, freenode, #hurd, 2011-04-23 <braunr> youpi: is there any use of the port renaming facility ? <youpi> I don't know @@ -250,7 +254,8 @@ IRC, freenode, #hurd, 2011-04-23 <antrik> well, if some processes really feel they must use random numbers for port names, they *ought* to be penalized ;-) -2011-04-27 + +# IRC, freenode, #hurd, 2011-04-27 <braunr> antrik: remember when you asked why high numbers would be a problem with radix trees ? @@ -319,7 +324,182 @@ IRC, freenode, #hurd, 2011-04-23 <braunr> antrik: a data structure used to map integers to pointers <braunr> http://fxr.watson.org/fxr/source/lib/idr.c?v=linux-2.6 -2011-06-18 + +# IRC, freenode, #hurd, 2011-06-08 + + <braunr> hm, reverse space/port to name lookups also suck + <braunr> having separate types for simple ipc entries and splay tree + entries really makes many parts of the ipc code complicated + <braunr> and a global hash table for these operations is scary + + +# IRC, freenode, #hurd, 2011-06-09 + + <braunr> hm nice, my radix tree code runs inside gnumach, along with the + original splay tree code and assertions making sure results are the same + <braunr> there is this "collision" thing i'm not sure to understand but + once this is solved, replacing the splay trees should be easy + + <braunr> youpi: is there a way to easily know the number of send rights + associated to a port ? + <youpi> portinfo ? + <braunr> portinfo gives information in a space + <braunr> but this is specific to a port + <braunr> is there an option for that ? + <youpi> -v + <braunr> hm ok + <youpi> 25: send (refs: 550) + <braunr> nice + <braunr> youpi: if you have time, could you give me the min/max/avg numbers + of send rights referring to the same port on buildds ? + <braunr> i'm trying to estimate if it's better to have space->list_of_ports + or port->list_of_spaces to replace the global ipc hash table + <braunr> the latter seems better but there could be unexpected cases on + machines using large amounts of resources like the buildds + <youpi> max is 64k + <youpi> min is 1 of course :) + <braunr> 64k + <braunr> then it's not what i'm looking for + <youpi> avg is 55 + <braunr> isn't this the number of urefs ? + <youpi> I don't know + <braunr> hmm + <braunr> what i'm looking for is the number of *pure send rights* for the + same port + <braunr> i don't think portinfo can give it + <braunr> there can only be one such send right per task for the same port + <braunr> 64k would mean there are 64k tasks + <youpi> ok, so it's more difficult + <youpi> it means using -t + <braunr> ahh + <youpi> and run n^2 portinfo over the n processes + <braunr> i see + <youpi> Mmm, that will however still show any duplicate send right + <youpi> but then by min/max/avg, you mean, over time ? + <braunr> i'll change the source code, simpler + <youpi> e.g. min would be right after boot? + <braunr> min is 1 + <youpi> 1 what ? + <braunr> 1 send right to a port + <youpi> ah, 1 for a given port + <braunr> yes + <youpi> ok, it becomes really hairy to compute, I don't hav ethe time :) + <braunr> avg and max are more interesting :) + <braunr> no worries + <youpi> braunr: I wouldn't be surprised that max is the number of tasks + <youpi> e.g. for a send port to the proc server for instance + <braunr> youpi: it is, but i'm not looking for potential numbers + <youpi> I'm not talking about a potential number, but an actual number + that's almost always true + <braunr> for one port, yes + <braunr> but yes, ok for max + <braunr> this makes choosing an appropriate data structure difficult + + <antrik> braunr: actually, min number of send rights to a port is 0... but + I'm sure you know that already :-) + + <antrik> youpi: normally each client gets a separate port. I'm not sure + there are any ports with send rights distributed over many tasks... + + <jkoenig> antrik, what about / ? + + <youpi> antrik: not necessarily + + <antrik> jkoenig: not sure... isn't the "/" port authenticated to the + specific user? + + <jkoenig> antrik, I guess so, but a single user could still have many + tasks. + <jkoenig> (wrt /) + <antrik> jkoenig: well, in theory the tasks having exactly the same UIDs + and GITs could probably share an auth token... but that's not how things + are handled in general + <antrik> at least I don't think so + <antrik> tasks are authenticated, not users + <antrik> err... GIDs :-) + <jkoenig> antrik, still, my quick glance to the fork() code seemed to + indicate the port is inherited as-is, maybe authentication happens only + when something is actually looked up? + <jkoenig> hmm "rpctrace ls -d /" does not show any authentication calls, + only a lookup("") on the root which returns a different port + <jkoenig> so I guess the root port is "deauthenticated" or something when + the uid of a process is changed. + <antrik> too bad cfhammer isn't around, he digged into all this stuff... + <antrik> I know that there is a mechanism which reauths all FDs when the + IDs of a process change + <antrik> but I'm not sure the "/" port uses this mechanism + + <braunr> antrik: but the radix tree codee is really used as is, which means + no locking, no preloading before locking, all of this because simple + locks don't exist on UP, and because the kernel isn't preemptible + + <braunr> antrik: and yes, min number is 0, but in that case you don't need + (space, port) -> right lookups :) + <braunr> antrik: or put in another way, whichever reasonable structure you + use, when it's empty, you don't care much + <braunr> which also means that the min number has actually no value here + <braunr> because the same applies to 1 + + <braunr> then what seems to take most time is forks + <braunr> and i hope my upcoming kernel changes will help the situation a + bit + <pinotree> what are your incoming gnumach changes about? + <braunr> the ipc translation layer speed + <braunr> which basically means operating on port names (looking them up, + inserting, removing, renaming, looking up send rights to a specific + ports) will be faster + <braunr> and i believe forks are (one of) the most demanding use cases wrt + ipc space manipulation + <braunr> i'm really surprised how badly the splay trees are used + <braunr> the worst case for this data structure is traversal + <braunr> and this is done in many situations + <braunr> leaving the tree in its worst case shape + <braunr> and i didn't mentioned the bunch of memory writes occurring, event + for things like lookups or traversals + <braunr> this is slow and can disrupt many cpu cache lines + <braunr> and when there are 10k to 100+k (e.g. in some ext2fs instances on + buildds), just imagine the number of operations involved in those + operations + <braunr> a simple traversal_next involves a rotation *gasp* + <braunr> this means potentially writing on 3 different cache lines, for + *one* next operation + <pinotree> what are you replacing that splay tree with? + <braunr> radix trees + <braunr> much shorter paths + <braunr> extremely few memory writes + <braunr> locality of reference when traversing + <braunr> good cache usage (as many of the top nodes are reused) + <braunr> the two drawbacks are 1/ memory allocation for external nodes, + which means the tree must be preloaded before locking + <braunr> and 2/ high memory overhead if the keys are sparse + <braunr> but this isn't the case with port names, unless someone messes it + up by assigning random names to many rights + + <antrik> braunr: so, when will we see the first performance comparision? + :-) + <braunr> antrik: that's a topic of itself, how to compare ? + <braunr> antrik: the thing is, my current gnumach patches only makes + assertions + <braunr> this is the best way i found to test my tree in real life + conditions + <braunr> much cleanup is needed + <braunr> and what i'd like to do is to completely replace all teh + translation layer structures with it + <braunr> which means removing much code, making sure it still works as + expected + <braunr> this is tedious + <braunr> so one month at least + <antrik> braunr: comparing shouldn't be too hard... the average configure + script does a lot of forking, which should be a good benchmark according + to your observations + <braunr> rough estimates are easy, yes + <braunr> but my observations my be wrong :p + <antrik> braunr: well, we don't really need precise numbers... + <antrik> unless you need to do some kind of fine-tuning? + <braunr> i don't know yet + + +# IRC, freenode, #hurd, 2011-06-18 < braunr> hmm, i'm having a problem with integrating my radix tree code in gnumach @@ -373,7 +553,8 @@ IRC, freenode, #hurd, 2011-04-23 < braunr> not much < braunr> testing is what requires time really -2011-06-27 + +# IRC, freenode, #hurd, 2011-06-27 < braunr> ok, here is the radix tree code: http://git.sceen.net/rbraun/libbraunr.git/ @@ -399,7 +580,8 @@ IRC, freenode, #hurd, 2011-04-23 < youpi> k < braunr> before locking the ipc spaces -2011-06-28 + +# IRC, freenode, #hurd, 2011-06-28 < braunr> antrik: i think i won't write the code for the preloading stuff actually @@ -456,7 +638,8 @@ IRC, freenode, #hurd, 2011-04-23 only < braunr> but released in both the entry and the splay tree code ... -2011-06-29 + +# IRC, freenode, #hurd, 2011-06-29 < braunr> hmm i missed an important thing :/ < braunr> and it's scary diff --git a/open_issues/translate_fd_or_port_to_file_name.mdwn b/open_issues/translate_fd_or_port_to_file_name.mdwn index 25a74456..485fb985 100644 --- a/open_issues/translate_fd_or_port_to_file_name.mdwn +++ b/open_issues/translate_fd_or_port_to_file_name.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -10,7 +10,10 @@ License|/fdl]]."]]"""]] [[!tag open_issue_glibc open_issue_hurd]] -\#hurd, freenode, June (?) 2010 +[[!toc]] + + +# IRC, freenode, #hurd, June (?) 2010 <pochu> is there a way (POSIX or Hurdish) to get the corresponding file name for a fd or a hurd port? <marcusb> there is a way @@ -52,3 +55,32 @@ License|/fdl]]."]]"""]] <marcusb> which combines file and directory operations <marcusb> of course, files and directories implement those functions differently <antrik> marcusb: do you know why this behavior (cat on directories) was changed? + + +# IRC, freenode, #hurd, 2011-07-13 + +A related issue: + + <braunr> rbraun@nordrassil:~$ vminfo $$ | wc -l + <braunr> 1039 + <braunr> any idea why a shell would consume more than 1039 map entries ? + <braunr> (well, not more actually) + <braunr> even the kernel and ext2fs have around 100 + <braunr> (the kernel has actually only 23, which is very good and expected) + <tschwinge> braunr: I agree that having some sort of debugging information + for memory objects et al. would be quite hand. To see where they're + coming from, etc. + <braunr> tschwinge: this would require naming objects at the mach level + <braunr> e.g. when creating an object + <braunr> giving it the path of a file for example + <tschwinge> braunr: I have recently seen something (due to youpi fixing a + bug) that bash is doing its own memory management. Perhaps all these are + such regions? + <tschwinge> braunr: For example, yes. + <braunr> what ? + <braunr> ?! + <tschwinge> braunr: + http://lists.gnu.org/archive/html/bug-bash/2011-04/msg00097.html + <braunr> i see + +Also see email thread starting at `id:"20110714082216.GA8335@sceen.net"`. diff --git a/open_issues/xattr.mdwn b/open_issues/xattr.mdwn new file mode 100644 index 00000000..8b08ef1e --- /dev/null +++ b/open_issues/xattr.mdwn @@ -0,0 +1,27 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="xattr: extended attributes"]] + +[[!tag open_issue_glibc open_issue_hurd]] + +IRC, freenode, #hurd, 2011-06-01 + + <gnu_srs> Another thing: the lsattr and chattr programs seems to be bogus + on Hurd. Are they present since they are part of e2fsprogs? + <youpi> I don't think the Hurd has an interface for it + <tschwinge> gnu_srs, youpi: lsattr/chattr are extended attributes, right? + We do have some patches from years ago for adding some support in glibc, + but they're not yet integrated. And also we do have a plan for using + these instead of our Hurd-specific passive translator information in + inodes. + +If interested in working on this, see [[!GNU_Savannah_task +#5503]]/[[!GNU_Savannah_patch #5126]], and talk to [[tschwinge]]. |