From 4d93ba7548629fff82aa03351132c85d478a8734 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Mon, 14 Feb 2011 17:08:47 +0100 Subject: open_issues/performance/io_system/read-ahead: New. --- microkernel/mach/gnumach/projects.mdwn | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) (limited to 'microkernel/mach/gnumach') diff --git a/microkernel/mach/gnumach/projects.mdwn b/microkernel/mach/gnumach/projects.mdwn index 47a2756c..f4ef192a 100644 --- a/microkernel/mach/gnumach/projects.mdwn +++ b/microkernel/mach/gnumach/projects.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2005, 2006, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2005, 2006, 2007, 2008, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] This page is a place to keep track of ideas about things that may be improved in GNU Mach, so that it'll evolve to a reliable microkernel for The Hurd, both @@ -58,7 +58,8 @@ so that no duplicate efforts end up. * Improve the external pagers interface - * Implement read-ahead (huge I/O improvements expected). + * Implement [[open_issues/performance/io_system/read-ahead]] (huge I/O + improvements expected). * Making this interface synchronous should improve I/O performance significantly, without (almost) any drawbacks (we also get some -- cgit v1.3.1 From 2a086501eda8117bbf73445854251a8361de360d Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Mon, 14 Feb 2011 17:10:23 +0100 Subject: microkernel/mach/gnumach/building/example: Remove obsolete page. --- microkernel/mach/gnumach/building.mdwn | 3 -- microkernel/mach/gnumach/building/example.mdwn | 54 -------------------------- 2 files changed, 57 deletions(-) delete mode 100644 microkernel/mach/gnumach/building/example.mdwn (limited to 'microkernel/mach/gnumach') diff --git a/microkernel/mach/gnumach/building.mdwn b/microkernel/mach/gnumach/building.mdwn index 9c075600..99e566bb 100644 --- a/microkernel/mach/gnumach/building.mdwn +++ b/microkernel/mach/gnumach/building.mdwn @@ -1,6 +1,3 @@ -Additional to the following text, a further [[example]] has be posted. - - # Building [[GNU_Mach|gnumach]] from Source If you want to build the [[GNU_Mach|gnumach]] kernel yourself instead of just using a diff --git a/microkernel/mach/gnumach/building/example.mdwn b/microkernel/mach/gnumach/building/example.mdwn deleted file mode 100644 index 7db98547..00000000 --- a/microkernel/mach/gnumach/building/example.mdwn +++ /dev/null @@ -1,54 +0,0 @@ -[[!meta copyright="Copyright © 2007, 2008 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] - -## Compiling GNU Mach microkernel - -Host development system is IBM T41 running Debian Sarge 3.1r0a GNU/Linux. - -* gcc version: 3.3.5 -* GNU sed version: 4.1.2 -* GNU make version: 3.8 -* mig version: 1.3-4 - -Obtained gnumach-1-branch sources from cvs: - - export CVS_RSH="ssh" - cvs -z3 -d:ext:anoncvs@ savannah.gnu.org:/cvsroot/hurd co -r gnumach-1-branch gnumach - -Obtained mig_1.3-4_i386.deb from -http://www.hadrons.org/~guillem/debian/pool/main/mig/. Installed it using dpkg: - - dpkg -i mig_1.3-4_i386.deb - -Entered into the gnumach sources and did the following for compilation: - - mkdir build - cd build - ../configure --host=i386-unknown-gnu0.2 --build=i586-pc-linux-gnu \ - --enable-kdb --enable-ide - make - -The kernel file is created in the build directory. Move it to /boot on the -testing x86 system Hurd partition. Rename it as gnumach1 and compress it: - - mv kernel gnumach1 - gzip gnumach1 - -Add a new entry on the testing machine /boot/grub/menu.lst to boot the new -kernel. - - title GNU Hurd K10 Compiled gnumach - kernel (hd0,3)/boot/gnumach1.gz root=device:hd2s4 -s - module (hd0,3)/hurd/ext2fs.static--multiboot-command-line=${kernel-command-line} \\ - --host-priv-port=${host-port} --device-master-port=${device-port} \\ - --exec-server-task=${exec-task} -T typed ${root} $(task-create)$(task-resume) - module (hd0,3)/lib/ld.so.1 /hurd/exec $(exec-task=task-create) - -Reboot into the new compiled mygnumach1.gz kernel! -- cgit v1.3.1 From b91f1d19f234c6da2ad4795db625030be855c992 Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Wed, 16 Feb 2011 10:11:16 +0100 Subject: microkernel/mach/gnumach/memory_management: New. --- microkernel/mach/gnumach.mdwn | 9 ++++--- microkernel/mach/gnumach/memory_management.mdwn | 36 +++++++++++++++++++++++++ 2 files changed, 41 insertions(+), 4 deletions(-) create mode 100644 microkernel/mach/gnumach/memory_management.mdwn (limited to 'microkernel/mach/gnumach') diff --git a/microkernel/mach/gnumach.mdwn b/microkernel/mach/gnumach.mdwn index f3d6d5f9..b385ca09 100644 --- a/microkernel/mach/gnumach.mdwn +++ b/microkernel/mach/gnumach.mdwn @@ -1,13 +1,13 @@ -[[!meta copyright="Copyright © 2001, 2002, 2007, 2008 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2001, 2002, 2007, 2008, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled -[[GNU Free Documentation License|/fdl]]."]]"""]] +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] GNU Mach is the microkernel that the [[GNU_Hurd|hurd]] system is based on. @@ -75,6 +75,7 @@ GNU/Hurd. * [[Building]] * [[Debugging]] * [[Boot_Trace]] + * [[Memory_Management]] * [[Projects]] * [[Rules]] * [[Open Issues|tag/open_issue_gnumach]] diff --git a/microkernel/mach/gnumach/memory_management.mdwn b/microkernel/mach/gnumach/memory_management.mdwn new file mode 100644 index 00000000..49a082e9 --- /dev/null +++ b/microkernel/mach/gnumach/memory_management.mdwn @@ -0,0 +1,36 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +IRC, freenode, #hurd, 2011-02-15 + + etenil: originally, mach had its own virtual space (the kernel + space) + etenil: in order to use linux 2.0 drivers, it now directly maps + physical memory, as linux does + etenil: but there is nothing similar to kmap() or vmalloc() in + mach, so the kernel is limited to its 1 GiB + (3 GiB userspace / 1 GiB kernelspace) + that's the short version, there is a vmalloc() in mach, but this + trick made it behave almost like a kmalloc() + braunr: the direct mapping is *only* for the benefit of Linux + drivers?... + also, the configuration of segments limits the kernel space + antrik: i'm not sure, as i said, this is the short version + antrik: but there is a paper which describes the integration of + those drivers in mach + you mean the linux 2.0 drivers? + braunr: I read it once, but I don't remember anything about the + physical mapping in there... + etenil: well, originally it was 1.3, but essentially that's the + same... + i don't see any other reason why there would be a direct mapping + except for performance (because you can use larger - even very + lage - pages without resetting the mmu often thanks to global pages, but + that didn't exist at the time) -- cgit v1.3.1 From bdd896e0b81cfb40c8d24a78f9022f6cd1ae5e8c Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Thu, 17 Feb 2011 14:15:11 +0100 Subject: open_issues/performance/io_system/clustered_page_faults: New. And some more IRC discussions. --- microkernel/mach/external_pager_mechanism.mdwn | 16 +++- microkernel/mach/gnumach/memory_management.mdwn | 14 +++ microkernel/mach/memory_object.mdwn | 4 +- open_issues/performance/io_system.mdwn | 3 +- .../io_system/clustered_page_faults.mdwn | 103 +++++++++++++++++++++ open_issues/performance/io_system/read-ahead.mdwn | 19 ++++ 6 files changed, 155 insertions(+), 4 deletions(-) create mode 100644 open_issues/performance/io_system/clustered_page_faults.mdwn (limited to 'microkernel/mach/gnumach') diff --git a/microkernel/mach/external_pager_mechanism.mdwn b/microkernel/mach/external_pager_mechanism.mdwn index d9b6c2c8..05a6cc56 100644 --- a/microkernel/mach/external_pager_mechanism.mdwn +++ b/microkernel/mach/external_pager_mechanism.mdwn @@ -1,5 +1,5 @@ -[[!meta copyright="Copyright © 2002, 2007, 2008, 2010 Free Software Foundation, -Inc."]] +[[!meta copyright="Copyright © 2002, 2007, 2008, 2010, 2011 Free Software +Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this @@ -181,3 +181,15 @@ fashion. The server is not required to send a response to the kernel. (D) The manager then transfers the data to the storeio server which eventually sends it to disk. The device driver consumes the memory doing the equivalent of a `vm_deallocate`. + + +# Issues + + * [[open_issues/performance/io_system/read-ahead]] + + * [[open_issues/performance/io_system/clustered_page_faults]] + + +# GNU Hurd Usage + +Read about the [[Hurd's I/O path|hurd/io_path]]. diff --git a/microkernel/mach/gnumach/memory_management.mdwn b/microkernel/mach/gnumach/memory_management.mdwn index 49a082e9..17dbe46f 100644 --- a/microkernel/mach/gnumach/memory_management.mdwn +++ b/microkernel/mach/gnumach/memory_management.mdwn @@ -34,3 +34,17 @@ IRC, freenode, #hurd, 2011-02-15 except for performance (because you can use larger - even very lage - pages without resetting the mmu often thanks to global pages, but that didn't exist at the time) + +IRC, freenode, #hurd, 2011-02-15 + + however, the kernel won't work in 64 bit mode without some changes + to physical memory management + and mmu management + (but maybe that's what you meant by physical memory) + +IRC, freenode, #hurd, 2011-02-16 + + antrik: youpi added it for xen, yes + antrik: but you're right, since mach uses a direct mapped kernel + space, the true problem is the lack of linux-like highmem support + which isn't required if the kernel space is really virtual diff --git a/microkernel/mach/memory_object.mdwn b/microkernel/mach/memory_object.mdwn index 2342145c..f32fe778 100644 --- a/microkernel/mach/memory_object.mdwn +++ b/microkernel/mach/memory_object.mdwn @@ -1,4 +1,4 @@ -[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation, +[[!meta copyright="Copyright © 2002, 2003, 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable @@ -29,3 +29,5 @@ last one tried is the *default memory manager* that resides in the microkernel, in contrast to most of the others. The default memory manager is needed because the microkernel can't wait infinitely for someone else to free the memory cache: it just calls the next memory manager hoping it to succeed. + +Read about [[GNU Mach's memory management|gnumach/memory_management]]. diff --git a/open_issues/performance/io_system.mdwn b/open_issues/performance/io_system.mdwn index dbf7012a..4af093ba 100644 --- a/open_issues/performance/io_system.mdwn +++ b/open_issues/performance/io_system.mdwn @@ -20,7 +20,8 @@ slow hard disk access. The reason for this slowness is lack and/or bad implementation of common optimization techniques, like scheduling reads and writes to minimize head movement; effective block caching; effective reads/writes to partial blocks; -reading/writing multiple blocks at once; and [[read-ahead]]. The +[[reading/writing multiple blocks at once|clustered_page_faults]]; and +[[read-ahead]]. The [[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some optimizations at a higher logical level. diff --git a/open_issues/performance/io_system/clustered_page_faults.mdwn b/open_issues/performance/io_system/clustered_page_faults.mdwn new file mode 100644 index 00000000..3a187523 --- /dev/null +++ b/open_issues/performance/io_system/clustered_page_faults.mdwn @@ -0,0 +1,103 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach open_issue_hurd]] + +IRC, freenode, #hurd, 2011-02-16 + + exceptfor the kernel, everything in an address space is + represented with a VM object + those objects can represent anonymous memory (from malloc() or + because of a copy-on-write) + or files + on classic Unix systems, these are files + on the Hurd, these are memory objects, backed by external pagers + (like ext2fs) + so when you read a file + the kernel maps it from ext2fs in your address space + and when you access the memory, a fault occurs + the kernel determines it's a region backed by ext2fs + so it asks ext2fs to provide the data + when the fault is resolved, your process goes on + does the faul occur because Mach doesn't know how to access the + memory? + it occurs because Mach intentionnaly didn't back the region with + physical memory + the MMU is programmed not to know what is present in the memory + region + or because it's read only + (which is the case for COW faults) + so that means this bit of memory is a buffer that ext2fs loads the + file into and then it is remapped to the application that asked for it + more or less, yes + ideally, it's directly written into the right pages + there is no intermediate buffer + I see + and as you told me before, currently the page faults are handled + one at a time + which wastes a lot of time + a certain amount of time + enough to bother the user :) + I've seen pages have a fixed size + yes + use the PAGE_SIZE macro + and when allocating memory, the size that's asked for is rounded + up to the page size + so if I have this correctly, it means that a file ext2fs provides + could be split into a lot of pages + yes + once in memory, it is managed by the page cache + so that pages more actively used are kept longer than others + in order to minimize I/O + ok + so a better page cache code would also improve overall performance + and more RAM would help a lot, since we are strongly limited by + the 768 MiB limit + which reduces the page cache size a lot + but the problem is that reading a whole file in means trigerring + many page faults just for one file + if you want to stick to the page clustering thing, yes + you want less page faults, so that there are less IPC between the + kernel and the pager + so either I make pages bigger + or I modify Mach so it can check up on a range of pages for faults + before actually processing + you *don't* change the page size + ah + that's hardware isn't it? + in Mach, yes + ok + and usually, you want the page size to be the CPU page size + I see + current CPU can support multiple page sizes, but it becomes quite + hard to correctly handle + and bigger page sizes mean more fragmentation, so it only suits + machines with large amounts of RAM, which isn't the case for us + ok + so I'll try the second approach then + that's what i'd recommand + recommend* + ok + +--- + +IRC, freenode, #hurd, 2011-02-16 + + etenil: OSF Mach does have clustered paging BTW; so that's one + place to start looking... + (KAM ported the OSF code to gnumach IIRC) + there is also an existing patch for clustered paging in libpager, + which needs some adaptation + the biggest part of the task is probably modifying the Hurd + servers to use the new interface + but as I said, KAM's code should be available through google, and + can serve as a starting point + + diff --git a/open_issues/performance/io_system/read-ahead.mdwn b/open_issues/performance/io_system/read-ahead.mdwn index 241cda41..3ee30b5d 100644 --- a/open_issues/performance/io_system/read-ahead.mdwn +++ b/open_issues/performance/io_system/read-ahead.mdwn @@ -278,3 +278,22 @@ IRC, freenode, #hurd, 2011-02-15 BTW, I remembered now that KAM's GSoC application should have a pretty good description of the necessary changes... unfortunately, these are not publicly visible IIRC :-( + +--- + +IRC, freenode, #hurd, 2011-02-16 + + braunr: I've looked in the kernel to see where prefetching would + fit best. We talked of the VM yesterday, but I'm not sure about it. It + seems to me that the device part of the kernel makes more sense since + it's logically what manages devices, am I wrong? + etenil: you are + etenil: well + etenil: drivers should already support clustered sector + read/writes + ah + but yes, there must be support in the drivers too + what would really benefit the Hurd mostly concerns page faults, so + the right place is the VM subsystem + +[[clustered_page_faults]] -- cgit v1.3.1