open_issues/performance/io_system/clustered_page_faults: New. And some more IRC discussions.

6 files changed, 155 insertions, 4 deletions

diff --git a/microkernel/mach/external_pager_mechanism.mdwn b/microkernel/mach/external_pager_mechanism.mdwn
index d9b6c2c8..05a6cc56 100644
--- a/microkernel/mach/external_pager_mechanism.mdwn
+++ b/microkernel/mach/external_pager_mechanism.mdwn
@@ -1,5 +1,5 @@
-[[!meta copyright="Copyright © 2002, 2007, 2008, 2010 Free Software Foundation,
-Inc."]]
+[[!meta copyright="Copyright © 2002, 2007, 2008, 2010, 2011 Free Software
+Foundation, Inc."]]
 
 [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
 id="license" text="Permission is granted to copy, distribute and/or modify this
@@ -181,3 +181,15 @@ fashion.  The server is not required to send a response to the kernel.
 (D) The manager then transfers the data to the storeio server which
 eventually sends it to disk.  The device driver consumes the memory
 doing the equivalent of a `vm_deallocate`.
+
+
+# Issues
+
+  * [[open_issues/performance/io_system/read-ahead]]
+
+  * [[open_issues/performance/io_system/clustered_page_faults]]
+
+
+# GNU Hurd Usage
+
+Read about the [[Hurd's I/O path|hurd/io_path]].
diff --git a/microkernel/mach/gnumach/memory_management.mdwn b/microkernel/mach/gnumach/memory_management.mdwn
index 49a082e9..17dbe46f 100644
--- a/microkernel/mach/gnumach/memory_management.mdwn
+++ b/microkernel/mach/gnumach/memory_management.mdwn
@@ -34,3 +34,17 @@ IRC, freenode, #hurd, 2011-02-15
     <braunr> except for performance (because you can use larger - even very
       lage - pages without resetting the mmu often thanks to global pages, but
       that didn't exist at the time)
+
+IRC, freenode, #hurd, 2011-02-15
+
+    <antrik> however, the kernel won't work in 64 bit mode without some changes
+      to physical memory management
+    <braunr> and mmu management
+    <braunr> (but maybe that's what you meant by physical memory)
+
+IRC, freenode, #hurd, 2011-02-16
+
+    <braunr> antrik: youpi added it for xen, yes
+    <braunr> antrik: but you're right, since mach uses a direct mapped kernel
+      space, the true problem is the lack of linux-like highmem support
+    <braunr> which isn't required if the kernel space is really virtual
diff --git a/microkernel/mach/memory_object.mdwn b/microkernel/mach/memory_object.mdwn
index 2342145c..f32fe778 100644
--- a/microkernel/mach/memory_object.mdwn
+++ b/microkernel/mach/memory_object.mdwn
@@ -1,4 +1,4 @@
-[[!meta copyright="Copyright © 2002, 2003, 2010 Free Software Foundation,
+[[!meta copyright="Copyright © 2002, 2003, 2010, 2011 Free Software Foundation,
 Inc."]]
 
 [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
@@ -29,3 +29,5 @@ last one tried is the *default memory manager* that resides in the microkernel,
 in contrast to most of the others.  The default memory manager is needed
 because the microkernel can't wait infinitely for someone else to free the
 memory cache: it just calls the next memory manager hoping it to succeed.
+
+Read about [[GNU Mach's memory management|gnumach/memory_management]].
diff --git a/open_issues/performance/io_system.mdwn b/open_issues/performance/io_system.mdwn
index dbf7012a..4af093ba 100644
--- a/open_issues/performance/io_system.mdwn
+++ b/open_issues/performance/io_system.mdwn
@@ -20,7 +20,8 @@ slow hard disk access.
 The reason for this slowness is lack and/or bad implementation of common
 optimization techniques, like scheduling reads and writes to minimize head
 movement; effective block caching; effective reads/writes to partial blocks;
-reading/writing multiple blocks at once; and [[read-ahead]].  The
+[[reading/writing multiple blocks at once|clustered_page_faults]]; and
+[[read-ahead]].  The
 [[ext2_filesystem_server|hurd/translator/ext2fs]] might also need some
 optimizations at a higher logical level.
 
diff --git a/open_issues/performance/io_system/clustered_page_faults.mdwn b/open_issues/performance/io_system/clustered_page_faults.mdwn
new file mode 100644
index 00000000..3a187523
--- /dev/null
+++ b/open_issues/performance/io_system/clustered_page_faults.mdwn
@@ -0,0 +1,103 @@
+[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled [[GNU Free Documentation
+License|/fdl]]."]]"""]]
+
+[[!tag open_issue_gnumach open_issue_hurd]]
+
+IRC, freenode, #hurd, 2011-02-16
+
+    <braunr> exceptfor the kernel, everything in an address space is
+      represented with a VM object
+    <braunr> those objects can represent anonymous memory (from malloc() or
+      because of a copy-on-write)
+    <braunr> or files
+    <braunr> on classic Unix systems, these are files
+    <braunr> on the Hurd, these are memory objects, backed by external pagers
+      (like ext2fs)
+    <braunr> so when you read a file
+    <braunr> the kernel maps it from ext2fs in your address space
+    <braunr> and when you access the memory, a fault occurs
+    <braunr> the kernel determines it's a region backed by ext2fs
+    <braunr> so it asks ext2fs to provide the data
+    <braunr> when the fault is resolved, your process goes on
+    <etenil> does the faul occur because Mach doesn't know how to access the
+      memory?
+    <braunr> it occurs because Mach intentionnaly didn't back the region with
+      physical memory
+    <braunr> the MMU is programmed not to know what is present in the memory
+      region
+    <braunr> or because it's read only
+    <braunr> (which is the case for COW faults)
+    <etenil> so that means this bit of memory is a buffer that ext2fs loads the
+      file into and then it is remapped to the application that asked for it
+    <braunr> more or less, yes
+    <braunr> ideally, it's directly written into the right pages
+    <braunr> there is no intermediate buffer
+    <etenil> I see
+    <etenil> and as you told me before, currently the page faults are handled
+      one at a time
+    <etenil> which wastes a lot of time
+    <braunr> a certain amount of time
+    <etenil> enough to bother the user :)
+    <etenil> I've seen pages have a fixed size
+    <braunr> yes
+    <braunr> use the PAGE_SIZE macro
+    <etenil> and when allocating memory, the size that's asked for is rounded
+      up to the page size
+    <etenil> so if I have this correctly, it means that a file ext2fs provides
+      could be split into a lot of pages
+    <braunr> yes
+    <braunr> once in memory, it is managed by the page cache
+    <braunr> so that pages more actively used are kept longer than others
+    <braunr> in order to minimize I/O
+    <etenil> ok
+    <braunr> so a better page cache code would also improve overall performance
+    <braunr> and more RAM would help a lot, since we are strongly limited by
+      the 768 MiB limit
+    <braunr> which reduces the page cache size a lot
+    <etenil> but the problem is that reading a whole file in means trigerring
+      many page faults just for one file
+    <braunr> if you want to stick to the page clustering thing, yes
+    <braunr> you want less page faults, so that there are less IPC between the
+      kernel and the pager
+    <etenil> so either I make pages bigger
+    <etenil> or I modify Mach so it can check up on a range of pages for faults
+      before actually processing
+    <braunr> you *don't* change the page size
+    <etenil> ah
+    <etenil> that's hardware isn't it?
+    <braunr> in Mach, yes
+    <etenil> ok
+    <braunr> and usually, you want the page size to be the CPU page size
+    <etenil> I see
+    <braunr> current CPU can support multiple page sizes, but it becomes quite
+      hard to correctly handle
+    <braunr> and bigger page sizes mean more fragmentation, so it only suits
+      machines with large amounts of RAM, which isn't the case for us
+    <etenil> ok
+    <etenil> so I'll try the second approach then
+    <braunr> that's what i'd recommand
+    <braunr> recommend*
+    <etenil> ok
+
+---
+
+IRC, freenode, #hurd, 2011-02-16
+
+    <antrik> etenil: OSF Mach does have clustered paging BTW; so that's one
+      place to start looking...
+    <antrik> (KAM ported the OSF code to gnumach IIRC)
+    <antrik> there is also an existing patch for clustered paging in libpager,
+      which needs some adaptation
+    <antrik> the biggest part of the task is probably modifying the Hurd
+      servers to use the new interface
+    <antrik> but as I said, KAM's code should be available through google, and
+      can serve as a starting point
+
+<http://lists.gnu.org/archive/html/bug-hurd/2010-06/msg00023.html>
diff --git a/open_issues/performance/io_system/read-ahead.mdwn b/open_issues/performance/io_system/read-ahead.mdwn
index 241cda41..3ee30b5d 100644
--- a/open_issues/performance/io_system/read-ahead.mdwn
+++ b/open_issues/performance/io_system/read-ahead.mdwn
@@ -278,3 +278,22 @@ IRC, freenode, #hurd, 2011-02-15
     <antrik> BTW, I remembered now that KAM's GSoC application should have a
       pretty good description of the necessary changes... unfortunately, these
       are not publicly visible IIRC :-(
+
+---
+
+IRC, freenode, #hurd, 2011-02-16
+
+    <etenil> braunr: I've looked in the kernel to see where prefetching would
+      fit best. We talked of the VM yesterday, but I'm not sure about it. It
+      seems to me that the device part of the kernel makes more sense since
+      it's logically what manages devices, am I wrong?
+    <braunr> etenil: you are
+    <braunr> etenil: well
+    <braunr> etenil: drivers should already support clustered sector
+      read/writes
+    <etenil> ah
+    <braunr> but yes, there must be support in the drivers too
+    <braunr> what would really benefit the Hurd mostly concerns page faults, so
+      the right place is the VM subsystem
+
+[[clustered_page_faults]]