[[!meta copyright="Copyright © 2011, 2014 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] [[!tag open_issue_gnumach]] # IRC, freenode, #hurd, 2011-10-16 braunr: I realize that kmem_alloc_wired maps the allocated pages in the kernel map it's a bit of a waste when my allocation is exactly a page size is there a proper page allocation which would simply return its physical address? pages returned by vm_page_grab may get swapped out, right? so could it be a matter of calling vm_page_alloc then vm_page_wire (with lock_queues held) ? s/alloc/grab/ vm_page_grab() is only used at boot iirc youpi: mapping allocated memory in the kernel map is normal, even if it's only a page the allocated area usually gets merged with an existing vm_map_entry youpi: also, i'm not sure about what you're trying to do here, so my answers may be out of scope :p saving addressing space with that scheme we're using twice as much addressing space for kernel buffers kernel or user task ? kernl hm are there so many wired areas ? several MiBs, yes there are also the zalloc areas that's pretty normal which I've recently incrased hm forget what i've just said about vm_page_grab() youpi: is there a particular problem caused by kernel memory exhaustion ? I currently can't pass the dh_strip stage of iceweasel due to this it can not allocate a stack a kernel thread stack ? yes that's surprising it'd be good to have a kernel memory profile vminfo is able to return the kernel map well, it's not suprising if the kernel map is full but it doesn't tell what allocates which p ieces that's what i mean, it's surprising to have a full kernel map (that's what profiling is about) right well, it's not really surprising, considering that the krenel map size is arbitrarily chosen youpi: 2 GiB is really high enough it's not 2GiB, precisely there is much of the 2GiB addr space which is spent on physical memory mapping then there is the virtual mapping are you sure we're talking about the kernel map, or e.g. the kmem map which is currently only 192MiB the kmem_map part of kernel_map ok, the kmem_map submap then netbsd has used 128 MiB for yeas with almost no issue mach uses more kernel objects so it's reasonable to have a bigger map but that big .. I've made the zalloc areas quite big err, not only zalloc area kernel stacks are allocated directly from the kernel map kalloc to 64MiB, zalloc to 64MiB ipc map size to 8MiB youpi: it could be the lack of kernel map entries and the device io map to 16MiB do you have the exact error message ? no more room for vm_map_find_entry in 71403294 no more rooom for kmem_alloc_aligned in 71403294 ah, aligned for a stack which is 4 pages only memory returned by kmem functions always return pages hum kmem functions always return memory in page units and my xen driver is allocating 1MiB memory for the network buffer 4 pages for kernel stacks ? through kmem_alloc_wire that seems a lot that's needed for xen page updates without having to split the update in several parts ok but are there any alignment requirements ? I guess mach uses the alignment trick to find "self" anyway, an alignment on 4pages shouldn't be a problem i think kmem_alloc_aligned() is the generic function used both for requests with and without alignment constraints so I was thinking about at least moving my xen net driver to vm_grab_page instead of kmem_alloc and along this, maybe others but you only get a vm_page, you can't access the memory it describes non, a lloc_aligned always aligns why? because it's not mapped there's even vm_grab_page_physical_addr it is, in the physical memory map ah, you mean using the direct mapped area yes then yes i don't know that part much what I'm afraid of is the wiring why ? because I don't want to see my page swapped out :) or whatever might happen if I don't wire it oh i'm almost sure you won't why? why some people need to wire it, and I won't? because in most mach vm derived code i've seen, you have to explicitely tell the vm your area is pageable ah, mach does such thing indeed wiring can be annoying when you're passing kernel data to external tasks you have to make sure the memory isn't wired once passed but that's rather a security/resource management problem in the net driver case, it's not passed to anything else I'm seeing 19MiB kmem_alloc_wired atm looks ok to me be aware that the vm_resident code was meant for single page allocations what does this mean? there is no buddy algorithm or anything else decent enough wrt performance vm_page_grab_contiguous_pages() can be quite slow err, ok, but what is the relation with the question at stake ? you need 4 pages of direct mapped memory for stacks those pages need to be physically contiguous if you want to avoid the mapping allocating physically contiguous pages in mach is slow :) I didn't mean I wanted to avoid the mapping for stacks for anything more than a page, kmem mapping should be fine I'm concerned with code which allocates only page per page which thus really doesn't need any mapping i don't know the mach details but in derived implementations, there is almost no overhead when allocating single pages except for the tlb programming well, there is: twice as much addressing space well netbsd doesn't directly map physical memory and for others, like freebsd the area is just one vm_map_entry and on modern mmus, 4 MiBs physical mappings are used in pmap again, I don't care about tlb & performance just about the addressing space hm you say "twice" which is short when you're trying to link crazy stuff like iceweasel & co yes ok, the virtual space is doubled yes but the resources consume to describe them aren't even on mach since you have both the physical mapping and the kmem mapping I don't care much about the resources but about addressing space well there are a limited numbers of solutions the space it takes and has to be taken from something else, that is, here physical memory available to Mach reduce the physical mapping increase the kmem mapping or reduce kernel memory consumption and instead of taking the space from physical mapping, we can as well avoid doubling the space consumption when it's trivial lto yes, the hird +t that's what I'm asking from the beginning :) 18:21 < youpi> I don't care much about the resources actually, you care :) yes and no i understand what you mean not in the sense "it takes a page_t to allocate a page" you want more virtual space, and aren't that much concerned about the number of objects used yes then it makes sense but in this case, it could be a good idea to generalize it have our own kmalloc/vmalloc interfaces maybe a gsoc project :) err, don't we have them already? I mean, what exactly do you want to generalize? mach only ever had vmalloc we already have a hell lot of allocators :) and it's a pain to distribute the available space to them yes but what you basically want is to introduce something similar to kmalloc for single pages or just patch the few cases that need it into just grabbing a page there are probably not so many ok i've just read vm_page_grab() it only removes a page from the free list other functions such as vm_page_alloc insert them afterwards if a page is in no list, it can't be paged out so i think it's probably safe to assume it's naturally wired you don't even need a call to vm_page_wire or a flag of some sort ok although considering the low amount of work done by vm_page_wire(), you could, for clarity braunr: I was also wondering about the VM_PAGE_FREE_RESERVED & such constants they're like 50 pages is this still reasonable nowadays? that's a question i'm still asking myself quite often :) also, the BURST_{MAX,MIN} & such in vm_pageout.c are probably out of date? i didn't study the pageout code much k but correct handling of low memory thresholds is a good point to keep in mind youpi: i often wonder how linux can sometimes have so few free pages left and still be able to work without any visible failure well, as long as you have enough pages to be able to make progress, you're fine that' the point of the RESERVED pages in mach I guess youpi: yes but, obviously, hard values are *bad* linux must adjust it, depending on the number of processors, the number of pdflush threads, probably other things i don't have in mind i don't know what should make us adjust that value in mach which value? the size of the reserved pool I don't think it's adjusted that's what i'm saying i guess there is an #ifndef line for arch specific definitions err, you just said linux must adjust it : )) there is none linux adjusts it dynamically well ok that's another way to say it we don't have code to get rid of this macro but i don't even know how we, as maintainers, are supposed to guess it # `k0ro/advisory_pageout/master` [[!GNU_Savannah_Git_hurd_gnumach 666299d037be6ffa83345d6d281fa955431f55fe]]. [[user/Sergio_Lopez]], [[libpager_deadlock]]. # Increase the pageout thread priority * [[!message-id "1341845097-24763-1-git-send-email-rbraun@sceen.net"]]. * [[!GNU_Savannah_Git_hurd_gnumach c7cdf5ff96e7c3bb008877893aa194908dca2185]]. # Tune VM Parameters * [[!message-id "h2k97f2a0d81004181028ycc10c46codc45d6ea33b2b0d5@mail.gmail.com"]]. * [[!message-id "1341845097-24763-1-git-send-email-rbraun@sceen.net"]]. * [[!GNU_Savannah_Git_hurd_gnumach 91f0887ca2345c2bd02747e4b437076641d77cd9]].