From ad27ca55949852c57acb7d1088085f7ae2ada2dd Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Thu, 21 Jul 2011 10:43:13 +0200 Subject: open_issues/gnumach_vm_map_entry_forward_merging: IRC. --- .../gnumach_vm_map_entry_forward_merging.mdwn | 120 +++++++++++++++++++++ 1 file changed, 120 insertions(+) (limited to 'open_issues') diff --git a/open_issues/gnumach_vm_map_entry_forward_merging.mdwn b/open_issues/gnumach_vm_map_entry_forward_merging.mdwn index 58013021..b2b20c5b 100644 --- a/open_issues/gnumach_vm_map_entry_forward_merging.mdwn +++ b/open_issues/gnumach_vm_map_entry_forward_merging.mdwn @@ -49,6 +49,126 @@ License|/fdl]]."]]"""]] braunr, as you pointed out, "vminfo $$" seems to indicate that merging _is_ incomplete. this could actually have a noticeable impact on package builds + hm + the number of entries for instances of bash running scripts don't + exceed 50-55 :/ + the issue seems to affect only certain instances (login shells, + and su -) + jkoenig: i guess dash is just much lighter than bash in many ways + :) + braunr, the number seems to increase with usage (100 here for a + newly started interactive shell, vs. 150 in an old one) + yes, merging is far from complete in the vm_map code + it only handles null objects (private zeroed memory), and only + tries to extend a previous entry (this isn't even a true merge) + this works well for the kernel however, which is why there are so + few as 25 entries + but any request, be it e.g. mmap(), or mprotect(), can easily + split entries + making their number larger + my ext2fs has ~6500 entries, but I guess this is related to + mapping blocks from the filesystem, right? + i think so + hm not sure actually + i'd say it's fragmentation due to copy on writes when client have + mapped memory from it + there aren't that many file mappings though :( + jkoenig: this might just be the same problem as in bash + 0x1308000[0x3000] (prot=RW, max_prot=RWX, mem_obj=584) + 0x130b000[0x6000] (prot=RW, max_prot=RWX, mem_obj=585) + 0x1311000[0x3000] (prot=RX, max_prot=RWX, mem_obj=586) + 0x1314000[0x1000] (prot=RW, max_prot=RWX, mem_obj=586) + 0x1315000[0x2000] (prot=RX, max_prot=RWX, mem_obj=587) + the first two could be merged but not the others + theoritically, those may correspond to memory objects backed by + different portions of the disk, right? + jkoenig: this looks very much like the same issue (many private + mappings not merged) + jkoenig: i'm not sure + jkoenig: normally there is an offset when the object is valid + but vminfo may simply not display it if 0 + * jkoenig goes read about memory object + ok, vminfo can't actually tell if the object is anonymous or + file-backed memory + (I'm perplexed about how the kernel can merge two memory objects + if disctinct port names exist in the tasks' name space -- that's what + mem_obj is, right?) + i don't see why + jkoenig: can you be more specific ? + braunr, if, say, 584 and 585 above are port names which the task + expects to be able to access and do stuff with, what will happen to them + when the memory objects are merged? + good question + but hm + no it's not really a problem + memory objects aren't directly handled by the vm system + vm_object and memory_object are different things + vm_objects can be split and merged + and shadow objects form chains ending on a final vm_object + which references a memory object + hm + jkoenig: ok no solution, they can't be merged :) + braunr, I'm confused :-) + jkoenig: but at least, if two vm_objects are created but reference + the same externel memory object, the vm should be able to merge them back + external* + are created as a result of a split + say, you map a memory object, mprotect part of it (=split), then + mprotect the reste of it (=merge), it should work + jkoenig: does that clarify things a bit ? + ok so if I get it right, the entries shown by vmstat are the + vm_object, and the mem_obj listed is a send right to the memory object + they're referencing ? + yes + i'm not sure about the type of the integer showed (port name or + simply an index) + jkoenig: another possibility explaining the high number of entries + is how anonymous memory is implemented + if every vm_allocate request implies the creation of a memory + object from the default pager + the vm has no way to merge them + and a vm_object is not a capability, but just an internal kernel + structure used to record the composition of the address space + jkoenig: not exactly the address space, but close enough + jkoenig: it's a container used to know what's in physical memory + and what isn't + braunr, ok I think I'm starting to get it, thanks. + glad i could help + i wonder when vm_map_enter() gets null objects though :/ + "If this port is MEMORY_OBJECT_NULL, then zero-filled memory is + allocated instead" + which means vm_allocate() + braunr, when the task uses vm_allocate(), or maybe vm_map_enter() + with MEMORY_OBJECT_NULL, there's an opportunity to extend an existing + object though, is that what you referred to earlier ? + jkoenig: yes, and that's what is done + but how does that play out with the default pager? (I'm thinking + aloud, as always feel free to ignore ;-) + the default pager backs vm_objects providing zero filled memory + hm, guess it wasn't your question + well, swap isn't like a file, pages can be placed dynamically, + which is why the offset is always 0 for this type of memory + hmm I see, apparently a memory object does not have a size + are you sure ? + from what I can gather from + http://www.gnu.org/software/hurd/gnumach-doc/External-Memory-Management.html, + but I looked very quickly + vm_objects have a size + and each map entry recors the offset within the object where the + mapping begins + offset and sizes are used by the kernel when querying the memory + object pager + see memory_object_data_request for example + right. + but the default pager has another interface + jkoenig: after some simple tests, i haven't seen a simple case + where forward merging could be applied :( + which means it's a lot harder than it first looked + hm + actually, there seems to be cases where this can be done + all of them occurring after a first merge was done + (which means a mapping request perfectly fits between two map + entries) # Procedure -- cgit v1.2.3