From 49a086299e047b18280457b654790ef4a2e5abfa Mon Sep 17 00:00:00 2001 From: Samuel Thibault Date: Wed, 18 Feb 2015 00:58:35 +0100 Subject: Revert "rename open_issues.mdwn to service_solahart_jakarta_selatan__082122541663.mdwn" This reverts commit 95878586ec7611791f4001a4ee17abf943fae3c1. --- open_issues/performance/degradation.mdwn | 52 + open_issues/performance/fork.mdwn | 37 + .../performance/io_system/binutils_ld_64ksec.mdwn | 39 + .../io_system/clustered_page_faults.mdwn | 165 ++ open_issues/performance/io_system/read-ahead.mdwn | 3076 ++++++++++++++++++++ open_issues/performance/ipc_virtual_copy.mdwn | 395 +++ open_issues/performance/microbenchmarks.mdwn | 13 + .../performance/microkernel_multi-server.mdwn | 226 ++ 8 files changed, 4003 insertions(+) create mode 100644 open_issues/performance/degradation.mdwn create mode 100644 open_issues/performance/fork.mdwn create mode 100644 open_issues/performance/io_system/binutils_ld_64ksec.mdwn create mode 100644 open_issues/performance/io_system/clustered_page_faults.mdwn create mode 100644 open_issues/performance/io_system/read-ahead.mdwn create mode 100644 open_issues/performance/ipc_virtual_copy.mdwn create mode 100644 open_issues/performance/microbenchmarks.mdwn create mode 100644 open_issues/performance/microkernel_multi-server.mdwn (limited to 'open_issues/performance') diff --git a/open_issues/performance/degradation.mdwn b/open_issues/performance/degradation.mdwn new file mode 100644 index 00000000..1aaae4d2 --- /dev/null +++ b/open_issues/performance/degradation.mdwn @@ -0,0 +1,52 @@ +[[!meta copyright="Copyright © 2011, 2012 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!meta title="Degradation of GNU/Hurd ``system performance''"]] + +[[!tag open_issue_gnumach open_issue_hurd]] + +[[!toc]] + + +# Email, [[!message-id "87mxg2ahh8.fsf@kepler.schwinge.homeip.net"]] (bug-hurd, 2011-07-25, Thomas Schwinge) + +> Building a certain GCC configuration on a freshly booted system: 11 h. +> Remove build tree, build it again (2nd): 12 h 50 min. Huh. Remove build +> tree, reboot, build it again (1st): back to 11 h. Remove build tree, build +> it again (2nd): 12 h 40 min. Remove build tree, build it again (3rd): 15 h. + +IRC, freenode, #hurd, 2011-07-23: + + < antrik> tschwinge: yes, the system definitely gets slower with + time. after running for a couple of weeks, it needs at least twice as + long to open a new shell for example + < antrik> I don't know whether this is only related to swap usage, or there + are some serious fragmentation issues + < braunr> antrik: both could be induced by fragmentation + + +# During [[IPC_virtual_copy]] testing + +IRC, freenode, #hurd, 2011-09-02: + + interestingly, running it several times has made the performance + drop quite much (i'm getting 400-500MB/s with 1M now, compared to nearly + 800 fifteen minutes ago) + manuel: i observed the same behaviour + [...] + + +# IRC, freenode, #hurd, 2011-09-22 + +See [[/open_issues/resource_management_problems/pagers]], IRC, freenode, #hurd, +2011-09-22. + + +# [[ext2fs_page_cache_swapping_leak]] diff --git a/open_issues/performance/fork.mdwn b/open_issues/performance/fork.mdwn new file mode 100644 index 00000000..5ceb6455 --- /dev/null +++ b/open_issues/performance/fork.mdwn @@ -0,0 +1,37 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_glibc open_issue_hurd]] + +Our [[`fork` implementation|glibc/fork]] is nontrivial. + +To do: hard numbers. +[[Microbenchmarks]]? + + +# Windows / Cygwin + + * + + * + + In particular, *5.6. Process Creation*. + + * + + * + + > Cygwin has recently adopted something called the "cygwin heap". This is + > an internal heap that is inherited by forked/execed children. It + > consists of process specific information that should be inherited. So + > things like the file descriptor table, the current working directory, and + > the chroot value live there. + + * diff --git a/open_issues/performance/io_system/binutils_ld_64ksec.mdwn b/open_issues/performance/io_system/binutils_ld_64ksec.mdwn new file mode 100644 index 00000000..931fd0ee --- /dev/null +++ b/open_issues/performance/io_system/binutils_ld_64ksec.mdwn @@ -0,0 +1,39 @@ +[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_hurd]] + +This one may be considered as a testcase for [[I/O system +optimization|community/gsoc/project_ideas/disk_io_performance]]. + +It is taken from the [[binutils testsuite|binutils]], +`ld/ld-elf/sec64k.exp`, where this +test may occasionally [[trigger a timeout|binutils#64ksec]]. It is +extracted from cdf7c161ebd4a934c9e705d33f5247fd52975612 sources, 2010-10-24. + + $ wget -O - http://www.gnu.org/software/hurd/open_issues/performance/io_system/binutils_ld_64ksec/test.tar.xz | xz -d | tar -x + $ cd test/ + $ \time ./ld-new.stripped -o dump dump?.o dump??.o + 0.00user 0.00system 2:46.11elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k + 0inputs+0outputs (0major+0minor)pagefaults 0swaps + +On the idle grubber, this one repeatedly takes a few minutes wall time to +complete successfully, contrary to a few seconds on a GNU/Linux system. + +While processing the object files, there is heavy interaction with the relevant +[[hurd/translator/ext2fs]] process. Running [[hurd/debugging/rpctrace]] on +the testee shows that (primarily) an ever-repeating series of `io_seek` and +`io_read` is being processed. Running the testee on GNU/Linux with strace +shows the equivalent thing (`_llseek`, `read`) -- but Linux' I/O system isn't +as slow as the Hurd's. + +As Samuel figured out later, this slowness may in fact be due to a Xen-specific +issue, see [[Xen_lseek]]. After the latter has been addressed, we can +re-evaluate this issue here. diff --git a/open_issues/performance/io_system/clustered_page_faults.mdwn b/open_issues/performance/io_system/clustered_page_faults.mdwn new file mode 100644 index 00000000..8bd6ba72 --- /dev/null +++ b/open_issues/performance/io_system/clustered_page_faults.mdwn @@ -0,0 +1,165 @@ +[[!meta copyright="Copyright © 2011, 2014 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach open_issue_hurd]] + +[[community/gsoc/project_ideas/disk_io_performance]]. + +[[!toc]] + + +# IRC, freenode, #hurd, 2011-02-16 + + exceptfor the kernel, everything in an address space is + represented with a VM object + those objects can represent anonymous memory (from malloc() or + because of a copy-on-write) + or files + on classic Unix systems, these are files + on the Hurd, these are memory objects, backed by external pagers + (like ext2fs) + so when you read a file + the kernel maps it from ext2fs in your address space + and when you access the memory, a fault occurs + the kernel determines it's a region backed by ext2fs + so it asks ext2fs to provide the data + when the fault is resolved, your process goes on + does the faul occur because Mach doesn't know how to access the + memory? + it occurs because Mach intentionnaly didn't back the region with + physical memory + the MMU is programmed not to know what is present in the memory + region + or because it's read only + (which is the case for COW faults) + so that means this bit of memory is a buffer that ext2fs loads the + file into and then it is remapped to the application that asked for it + more or less, yes + ideally, it's directly written into the right pages + there is no intermediate buffer + I see + and as you told me before, currently the page faults are handled + one at a time + which wastes a lot of time + a certain amount of time + enough to bother the user :) + I've seen pages have a fixed size + yes + use the PAGE_SIZE macro + and when allocating memory, the size that's asked for is rounded + up to the page size + so if I have this correctly, it means that a file ext2fs provides + could be split into a lot of pages + yes + once in memory, it is managed by the page cache + so that pages more actively used are kept longer than others + in order to minimize I/O + ok + so a better page cache code would also improve overall performance + and more RAM would help a lot, since we are strongly limited by + the 768 MiB limit + which reduces the page cache size a lot + but the problem is that reading a whole file in means trigerring + many page faults just for one file + if you want to stick to the page clustering thing, yes + you want less page faults, so that there are less IPC between the + kernel and the pager + so either I make pages bigger + or I modify Mach so it can check up on a range of pages for faults + before actually processing + you *don't* change the page size + ah + that's hardware isn't it? + in Mach, yes + ok + and usually, you want the page size to be the CPU page size + I see + current CPU can support multiple page sizes, but it becomes quite + hard to correctly handle + and bigger page sizes mean more fragmentation, so it only suits + machines with large amounts of RAM, which isn't the case for us + ok + so I'll try the second approach then + that's what i'd recommand + recommend* + ok + + +# IRC, freenode, #hurd, 2011-02-16 + + etenil: OSF Mach does have clustered paging BTW; so that's one + place to start looking... + (KAM ported the OSF code to gnumach IIRC) + there is also an existing patch for clustered paging in libpager, + which needs some adaptation + the biggest part of the task is probably modifying the Hurd + servers to use the new interface + but as I said, KAM's code should be available through google, and + can serve as a starting point + + + + +# IRC, freenode, #hurd, 2011-07-22 + + but concerning clustered pagins/outs, i'm not sure it's a mach + interface limitation + the external memory pager interface does allow multiple pages to + be transfered + isn't it an internal Mach VM problem ? + isn't it simply the page fault handler ? + braunr: are you sure? I was under the impression that changing the + pager interface was among the requirements... + hm... I wonder whether for pageins, it could actually be handled + in the pages instead of Mach... though this wouldn't work for pageouts, + so probably not very helpful + err... in the pagers + antrik: i'm almost sure + but i've be proven wrong many times, so .. + there are two main facts that lead me to think this + 1/ + http://www.gnu.org/software/hurd/gnumach-doc/Memory-Objects-and-Data.html#Memory-Objects-and-Data + says lengths are provided and doesn't mention the limitation + 2/ when reading about UVM, one of the major improvements (between + 10 and 30% of global performance depending on the benchmarks) was + implementing the madvise semantics + and this didn't involve a new pager interface, but rather a new + page fault handler + braunr: hm... the interface indeed looks like it can handle + multiple pages in both directions... perhaps it was at the Hurd level + where the pager interface needs to be modified, not the Mach one?... + antrik: would be nice wouldn't it ? :) + antrik: more probably the page fault handler + + +# IRC, freenode, #hurd, 2011-09-28 + + antrik: I've just recovered part of my old multipage I/O work + antrik: I intend to clean and submit it after finishing the changes + to the pageout system. + slpz: oh, great! + didn't know you worked on multipage I/O + slpz: BTW, have you checked whether any of the work done for GSoC + last year is any good?... + (apart from missing copyright assignments, which would be a + serious problem for the Hurd parts...) + antrik: It was seven years ago, but I did: + http://www.mail-archive.com/bug-hurd@gnu.org/msg10285.html :-) + antrik: Sincerely, I don't think the quality of that code is good + enough to be considered... but I think it was my fault as his mentor for + not correcting him soon enough... + slpz: I see + TBH, I feel guilty myself, for not asking about the situation + immediately when he stopped attending meetings... + slpz: oh, you even already looked into vm_pageout_scan() back then + :-) + + +# [[Read-Ahead]] diff --git a/open_issues/performance/io_system/read-ahead.mdwn b/open_issues/performance/io_system/read-ahead.mdwn new file mode 100644 index 00000000..59f22187 --- /dev/null +++ b/open_issues/performance/io_system/read-ahead.mdwn @@ -0,0 +1,3076 @@ +[[!meta copyright="Copyright © 2011, 2012, 2013, 2014 Free Software Foundation, +Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_gnumach open_issue_hurd]] + +[[!toc]] + + +# [[community/gsoc/project_ideas/disk_io_performance]] + + +# [[gnumach_page_cache_policy]] + + +# 2011-02 + +[[Etenil]] has been working in this area. + + +## IRC, freenode, #hurd, 2011-02-13 + + youpi: Would libdiskfs/diskfs.h be in the right place to make + readahead functions? + etenil: no, it'd rather be at the memory management layer, + i.e. mach, unfortunately + because that's where you see the page faults + youpi: Linux also provides a readahead() function for higher level + applications. I'll probably have to add the same thing in a place that's + higher level than mach + well, that should just be hooked to the same common implementation + the man page for readahead() also states that portable + applications should avoid it, but it could be benefic to have it for + portability + it's not in posix indeed + + +## IRC, freenode, #hurd, 2011-02-14 + + youpi: I've investigated prefetching (readahead) techniques. One + called DiskSeen seems really efficient. I can't tell yet if it's patented + etc. but I'll keep you informed + don't bother with complicated techniques, even the most simple ones + will be plenty :) + it's not complicated really + the matter is more about how to plug it into mach + ok + then don't bother with potential pattents + etenil: please take a look at the work KAM did for last year's + GSoC + just use a trivial technique :) + ok, i'll just go the easy way then + + antrik: what was etenil referring to when talking about + prefetching ? + oh, madvise() stuff + i could help him with that + + +## IRC, freenode, #hurd, 2011-02-15 + + oh, I'm looking into prefetching/readahead to improve I/O + performance + etenil: ok + etenil: that's actually a VM improvement, like samuel told you + yes + a true I/O improvement would be I/O scheduling + and how to implement it in a hurdish way + (or if it makes sense to have it in the kernel) + that's what I've been wondering too lately + concerning the VM, you should look at madvise() + my understanding is that Mach considers devices without really + knowing what they are + that's roughly the interface used both at the syscall() and the + kernel levels in BSD, which made it in many other unix systems + whereas I/O optimisations are often hard disk drives specific + that's true for almost any kernel + the device knowledge is at the driver level + yes + (here, I separate kernels from their drivers ofc) + but Mach also contains some drivers, so I'm going through the code + to find the apropriate place for these improvements + you shouldn't tough the drivers at all + touch + true, but I need to understand how it works before fiddling around + hm + not at all + the VM improvement is about pagein clustering + you don't need to know how pages are fetched + well, not at the device level + you need to know about the protocol between the kernel and + external pagers + ok + you could also implement pageout clustering + if I understand you well, you say that what I'd need to do is a + queuing system for the paging in the VM? + no + i'm saying that, when a page fault occurs, the kernel should + (depending on what was configured through madvise()) transfer pages in + multiple blocks rather than one at a time + communication with external pagers is already async, made through + regular ports + which already implement message queuing + you would just need to make the mapped regions larger + and maybe change the interface so that this size is passed + mmh + (also don't forget that page clustering can include pages *before* + the page which caused the fault, so you may have to pass the start of + that region too) + I'm not sure I understand the page fault thing + is it like a segmentation error? + I can't find a clear definition in Mach's manual + ah + it's a fundamental operating system concept + http://en.wikipedia.org/wiki/Page_fault + ah ok + I understand now + so what's currently happening is that when a page fault occurs, + Mach is transfering pages one at a time and wastes time + sometimes, transferring just one page is what you want + it depends on the application, which is why there is madvise() + our rootfs, on the other hand, would benefit much from such an + improvement + in UVM, this optimization is account for around 10% global + performance improvement + accounted* + not bad + well, with an improved page cache, I'm sure I/O would matter less + on systems with more RAM + (and another improvement would make mach support more RAM in the + first place !) + an I/O scheduler outside the kernel would be a very good project + IMO + in e.g. libstore/storeio + yes + but as i stated in my thesis, a resource scheduler should be as + close to its resource as it can + and since mach can host several operating systems, I/O schedulers + should reside near device drivers + and since current drivers are in the kernel, it makes sens to have + it in the kernel too + so there must be some discussion about this + doesn't this mean that we'll have to get some optimizations in + Mach and have the same outside of Mach for translators that access the + hardware directly? + etenil: why ? + well as you said Mach contains some drivers, but in principle, it + shouldn't, translators should do disk access etc, yes? + etenil: ok + etenil: so ? + well, let's say if one were to introduce SATA support in Hurd, + nothing would stop him/her to do so with a translator rather than in Mach + you should avoid the term translator here + it's really hurd specific + let's just say a user space task would be responsible for that + job, maybe multiple instances of it, yes + ok, so in this case, let's say we have some I/O optimization + techniques like readahead and I/O scheduling within Mach, would these + also apply to the user-space task, or would they need to be + reimplemented? + if you have user space drivers, there is no point having I/O + scheduling in the kernel + but we also have drivers within the kernel + what you call readahead, and I call pagein/out clustering, is + really tied to the VM, so it must be in Mach in any case + well + you either have one or the other + currently we have them in the kernel + if we switch to DDE, we should have all of them outside + that's why such things must be discussed + ok so if I follow you, then future I/O device drivers will need to + be implemented for Mach + currently, yes + but preferrably, someone should continue the work that has been + done on DDe so that drivers are outside the kernel + so for the time being, I will try and improve I/O in Mach, and if + drivers ever get out, then some of the I/O optimizations will need to be + moved out of Mach + let me remind you one of the things i said + i said I/O scheduling should be close to their resource, because + we can host several operating systems + now, the Hurd is the only system running on top of Mach + so we could just have I/O scheduling outside too + then you should consider neighbor hurds + which can use different partitions, but on the same device + currently, partitions are managed in the kernel, so file systems + (and storeio) can't make good scheduling decisions if it remains that way + but that can change too + a single storeio representing a whole disk could be shared by + several hurd instances, just as if it were a high level driver + then you could implement I/O scheduling in storeio, which would be + an improvement for the current implementation, and reusable for future + work + yes, that was my first instinct + and you would be mostly free of the kernel internals that make it + a nightmare + but youpi said that it would be better to modify Mach instead + he mentioned the page clustering thing + not I/O scheduling + theseare really two different things + ok + you *can't* implement page clustering outside Mach because Mach + implements virtual memory + both policies and mechanisms + well, I'd rather think of one thing at a time if that's alright + so what I'm busy with right now is setting up clustered page-in + which need to be done within Mach + keep clustered page-outs in mind too + although there are more constraints on those + yes + I've looked up madvise(). There's a lot of documentation about it + in Linux but I couldn't find references to it in Mach (nor Hurd), does it + exist? + well, if it did, you wouldn't be caring about clustered page + transfers, would you ? + be careful about linux specific stuff + I suppose not + you should implement at least posix options, and if there are + more, consider the bsd variants + (the Mach VM is the ancestor of all modern BSD VMs) + madvise() seems to be posix + there are system specific extensions + be careful + CONFORMING TO POSIX.1b. POSIX.1-2001 describes posix_madvise(3) + with constants POSIX_MADV_NORMAL, etc., with a behav‐ ior close to that + described here. There is a similar posix_fadvise(2) for file access. + MADV_REMOVE, MADV_DONTFORK, MADV_DOFORK, MADV_HWPOISON, + MADV_MERGEABLE, and MADV_UNMERGEABLE are Linux- specific. + I was about to post these + ok, so basically madvise() allows tasks etc. to specify a usage + type for a chunk of memory, then I could apply the relevant I/O + optimization based on this + that's it + cool, then I don't need to worry about knowing what the I/O is + operating on, I just need to apply the optimizations as advised + that's convenient + ok I'll start working on this tonight + making a basic readahead shouldn't be too hard + readahead is a misleading name + is pagein better? + applies to too many things, doesn't include the case where + previous elements could be prefetched + clustered page transfers is what i would use + page prefetching maybe + ok + you should stick to something that's already used in the + literature since you're not inventing something new + yes I've read a paper about prefetching + ok + thanks for your help braunr + sure + you're welcome + braunr: madvise() is really the least important part of the + picture... + very few applications actually use it. but pretty much all + applications will profit from clustered paging + I would consider madvise() an optional goody, not an integral part + of the implementation + etenil: you can find some stuff about KAM's work on + http://www.gnu.org/software/hurd/user/kam.html + not much specific though + thanks + I don't remember exactly, but I guess there is also some + information on the mailing list. check the archives for last summer + look for Karim Allah Ahmed + antrik: I disagree, madvise gives me a good starting point, even + if eventually the optimisations should run even without it + the code he wrote should be available from Google's summer of code + page somewhere... + antrik: right, i was mentioning madvise() because the kernel (VM) + interface is pretty similar to the syscall + but even a default policy would be nice + etenil: I fear that many bits were discussed only on IRC... so + you'd better look through the IRC logs from last April onwards... + ok + + at the beginning I thought I could put that into libstore + which would have been fine + + BTW, I remembered now that KAM's GSoC application should have a + pretty good description of the necessary changes... unfortunately, these + are not publicly visible IIRC :-( + + +## IRC, freenode, #hurd, 2011-02-16 + + braunr: I've looked in the kernel to see where prefetching would + fit best. We talked of the VM yesterday, but I'm not sure about it. It + seems to me that the device part of the kernel makes more sense since + it's logically what manages devices, am I wrong? + etenil: you are + etenil: well + etenil: drivers should already support clustered sector + read/writes + ah + but yes, there must be support in the drivers too + what would really benefit the Hurd mostly concerns page faults, so + the right place is the VM subsystem + +[[clustered_page_faults]] + + +# 2012-03 + + +## IRC, freenode, #hurd, 2012-03-21 + + I thought that readahead should have some heuristics, like + accounting size of object and last access time, but i didn't find any in + kam's patch. Are heuristics needed or it will be overhead for + microkernel? + size of object and last access time are not necessarily useful to + take into account + what would usually typically be kept is the amount of contiguous + data that has been read lately + to know whether it's random or sequential, and how much is read + (the whole size of the object does not necessarily give any + indication of how much of it will be read) + if big object is accessed often, performance could be increased if + frame that will be read ahead will be increased too. + yes, but the size of the object really does not matter + you can just observe how much data is read and realize that it's + read a lot + all the more so with userland fs translators + it's not because you mount a CD image that you need to read it all + youpi: indeed. this will be better. But on other hand there is + principle about policy and mechanism. And kernel should implement + mechanism, but heuristics seems to be policy. Or in this case moving + readahead policy to user level would be overhead? + mcsim: paging policy is all in kernel anyways; so it makes perfect + sense to put the readahead policy there as well + (of course it can be argued -- probably rightly -- that all of + this should go into userspace instead...) + antrik: probably defpager partly could do that. AFAIR, it is + possible for defpager to return more memory than was asked. + antrik: I want to outline what should be done during gsoc. First, + kernel should support simple readahead for specified number of pages + (regarding direction of access) + simple heuristic for changing frame + size. Also default pager could make some analysis, for instance if it has + many data located consequentially it could return more data then was + asked. For other pagers I won't do anything. Is it suitable? + mcsim: I think we actually had the same discussion already with + KAM ;-) + for clustered pageout, the kernel *has* to make the decision. I'm + really not convinced it makes sense to leave the decision for clustered + pagein to the individual pagers + especially as this will actually complicate matters because a) it + will require work in *every* pager, and b) it will probably make handling + of MADVISE & friends more complex + implementing readahead only for the default pager would actually + be rather unrewarding. I'm pretty sure it's the one giving the *least* + benefit + it's much, much more important for ext2 + mcsim: maybe try to dig in the irc logs, we discussed about it with + neal. the current natural place would be the kernel, because it's the + piece that gets the traps and thus knows what happens with each + projection, while the backend just provides the pages without knowing + which projection wants it. Moving to userland would not only be overhead, + but quite difficult + antrik: OK, but I'm not sure that I could do it for ext2. + OK, I'll dig. + + +## IRC, freenode, #hurd, 2012-04-01 + + as part of implementing of readahead project I have to add + interface for setting appropriate behaviour for memory range. This + interface than should be compatible with madvise call, that has a lot of + possible advises, but most part of them are specific for Linux (according + to man page). Should mach also support these Linux-specific values? + p.s. these Linux-specific values shouldn't affect readahead + algorithm. + the interface shouldn't prevent from adding them some day + so that we don't have to add them yet + ok. And what behaviour with value MADV_NORMAL should be look like? + Seems that it should be synonym to MADV_SEQUENTIAL, isn't it? + no, it just means "no idea what it is" + in the linux implementation, that means some given readahead value + while SEQUENTIAL means twice as much + and RANDOM means zero + youpi: thank you. + youpi: Than, it seems to be better that kernel interface for + setting behaviour will accept readahead value, without hiding it behind + such constants, like VM_BEHAVIOR_DEFAULT (like it was in kam's + patch). And than implementation of madvise will call vm_behaviour_set + with appropriate frame size. Is that right? + question of taste, better ask on the list + ok + + +## IRC, freenode, #hurd, 2012-06-09 + + hello. What fictitious pages in gnumach are needed for? + I mean why real page couldn't be grabbed straight, but in sometimes + fictitious page is grabbed first and than converted to real? + mcsim: iirc, fictitious pages are needed by device pagers which + must comply with the vm pager interface + mcsim: specifically, they must return a vm_page structure, but + this vm_page describes device memory + mcsim: and then, it must not be treated like normal vm_page, which + can be added to page queues (e.g. page cache) + + +## IRC, freenode, #hurd, 2012-06-22 + + braunr: Ah. Patch for large storages introduced new callback + pager_notify_evict. User had to define this callback on his own as + pager_dropweak, for instance. But neal's patch change this. Now all + callbacks could have any name, but user defines structure with pager ops + and supplies it in pager_create. + So, I just changed notify_evict to confirm it to new style. + braunr: I want to changed interface of mo_change_attributes and + test my changes with real partitions. For both these I have to update + ext2fs translator, but both partitions I have are bigger than 2Gb, that's + why I need apply this patch.z + But what to do with mo_change_attributes? I need somehow inform + kernel about page fault policy. + When I change mo_ interface in kernel I have to update all programs + that use this interface and ext2fs is one of them. + + braunr: Who do you think better to inform kernel about fault + policy? At the moment I've added fault_strategy parameter that accepts + following strategies: randow, sequential with single page cluster, + sequential with double page cluster and sequential with quad page + cluster. OSF/mach has completely another interface of + mo_change_attributes. In OSF/mach mo_change_attributes accepts structure + of parameter. This structure could have different formats depending o + This rpc could be useful because it is not very handy to update + mo_change_attributes for kernel, for hurd libs and for glibc. Instead of + this kernel will accept just one more structure format. + well, like i wrote on the mailing list several weeks ago, i don't + think the policy selection is of concern currently + you should focus on the implementation of page clustering and + readahead + concerning the interface, i don't think it's very important + also, i really don't like the fact that the policy is per object + it should be per map entry + i think it mentioned that in my mail too + i really think you're wasting time on this + http://lists.gnu.org/archive/html/bug-hurd/2012-04/msg00064.html + http://lists.gnu.org/archive/html/bug-hurd/2012-04/msg00029.html + mcsim: any reason you completely ignored those ? + braunr: Ok. I'll do clustering for map entries. + no it's not about that either :/ + clustering is grouping several pages in the same transfer between + kernel and pager + the *policy* is held in map entries + mcsim: I'm not sure I properly understand your question about the + policy interface... but if I do, it's IMHO usually better to expose + individual parameters as RPC arguments explicitly, rather than hiding + them in an opaque structure... + (there was quite some discussion about that with libburn guy) + antrik: Following will be ok? kern_return_t vm_advice(map, address, + length, advice, cluster_size) + Where advice will be either random or sequential + looks fine to me... but then, I'm not an expert on this stuff :-) + perhaps "policy" would be clearer than "advice"? + madvise has following prototype: int madvise(void *addr, size_t + len, int advice); + hmm... looks like I made a typo. Or advi_c_e is ok too? + advise is a verb; advice a noun... there is a reason why both + forms show up in the madvise prototype :-) + so final variant should be kern_return_t vm_advise(map, address, + length, policy, cluster_size)? + mcsim: nah, you are probably right that its better to keep + consistency with madvise, even if the name of the "advice" parameter + there might not be ideal... + BTW, where does cluster_size come from? from the filesystem? + I see merits both to naming the parameter "policy" (clearer) or + "advice" (more consistent) -- you decide :-) + antrik: also there is variant strategy, like with inheritance :) + I'll choose advice for now. + What do you mean under "where does cluster_size come from"? + well, madvise doesn't have this parameter; so the value must come + from a different source? + in madvise implementation it could fixed value or somehow + calculated basing on size of memory range. In OSF/mach cluster size is + supplied too (via mo_change_attributes). + ah, so you don't really know either :-) + well, my guess is that it is derived from the cluster size used by + the filesystem in question + so for us it would always be 4k for now + (and thus you can probably leave it out alltogether...) + well, fatfs can use larger clusters + I would say, implement it only if it's very easy to do... if it's + extra effort, it's probably not worth it + There is sense to make cluster size bigger for ext2 too, since most + likely consecutive clusters will be within same group. + But anyway I'll handle this later. + well, I don't know what cluster_size does exactly; but by the + sound of it, I'd guess it makes an assumption that it's *always* better + to read in this cluster size, even for random access -- which would be + simply wrong for 4k filesystem clusters... + BTW, I agree with braunr that madvice() is optional -- it is way + way more important to get readahead working as a default policy first + + +## IRC, freenode, #hurd, 2012-07-01 + + youpi: Do you think you could review my code? + sure, just post it to the list + make sure to break it down into logical pieces + youpi: I pushed it my branch at gnumach repository + youpi: or it is still better to post changes to list? + posting to the list would permit feedback from other people too + mcsim: posix distinguishes normal, sequential and random + we should probably too + the system call should probably be named "vm_advise", to be a verb + like allocate etc. + youpi: ok. A have a talk with antrik regarding naming, I'll change + this later because compiling of glibc take a lot of time. + mcsim: I find it odd that vm_for_every_page allocates non-existing + pages + there should probably be at least a flag to request it or not + youpi: normal policy is synonym to default. And this could be + treated as either random or sequential, isn't it? + mcsim: normally, no + yes, the normal policy would be the default + it doesn't mean random or sequential + it's just to be a compromise between both + random is meant to make no read-ahead, since that'd be spurious + anyway + while by default we should make readahead + and sequential makes even more aggressive readahead, which usually + implies a greater number of pages to fetch + that's all + yes + well, that part is handled by the cluster_size parameter actually + what about reading pages preceding the faulted paged ? + Shouldn't sequential clean some pages (if they, for example, are + not precious) that are placed before fault page? + ? + that could make sense, yes + you lost me + and something that you wouldn't to with the normal policy + braunr: clear what has been read previously + ? + since the access is supposed to be sequential + oh + the application will proabably not re-read what was already read + you mean to avoid caching it ? + yes + inactive memory is there for that + while with the normal policy you'd assume that the application + might want to go back etc. + yes, but you can help it + yes + instead of making other pages compete with it + but then, it's for precious pages + I have to say I don't know what a precious page it + s + does it mean dirty pages? + no + precious means cached pages + "If precious is FALSE, the kernel treats the data as a temporary + and may throw it away if it hasn't been changed. If the precious value is + TRUE, the kernel treats its copy as a data repository and promises to + return it to the manager; the manager may tell the kernel to throw it + away instead by flushing and not cleaning the data" + hm no + precious means the kernel must keep it + youpi: According to vm_for_every_page. What kind of flag do you + suppose? If object is internal, I suppose not to cross the bound of + object, setting in_end appropriately in vm_calculate_clusters. + If object is external we don't know its actual size, so we should + make mo request first. And for this we should create fictitious pages. + mcsim: but how would you implement this "cleaning" with sequential + ? + mcsim: ah, ok, I thought you were allocating memory, but it's just + fictitious pages + comment "Allocate a new page" should be fixed :) + braunr: I don't now how I will implement this specifically (haven't + tried yet), but I don't think that this is impossible + braunr: anyway it's useful as an example where normal and + sequential would be different + if it can be done simply + because i can see more trouble than gains in there :) + braunr: ok :) + mcsim: hm also, why fictitious pages ? + fictitious pages should normally be used only when dealing with + memory mapped physically which is not real physical memory, e.g. device + memory + but vm_fault could occur when object represent some device memory. + that's exactly why there are fictitious pages + at the moment of allocating of fictitious page it is not know what + backing store of object is. + really ? + damn, i've got used to UVM too much :/ + braunr: I said something wrong? + no no + it's just that sometimes, i'm confusing details about the various + BSD implementations i've studied + out-of-gsoc-topic question: besides network drivers, do you think + we'll have other drivers that will run in userspace and have to implement + memory mapping ? like framebuffers ? + or will there be a translation layer such as storeio that will + handle mapping ? + framebuffers typically will, yes + that'd be antrik's work on drm + hmm + ok + mcsim: so does the implementation work, and do you see performance + improvement? + youpi: I haven't tested it yet with large ext2 :/ + youpi: I'm going to finish now moving of ext2 to new interface, + than other translators in hurd repository and than finish memory policies + in gnumach. Is it ok? + which new interface? + Written by neal. I wrote some temporary code to make ext2 work with + it, but I'm going to change this now. + you mean the old unapplied patch? + yes + did you have a look at Karim's work? + (I have to say I never found the time to check how it related with + neal's patch) + I found only his work in kernel. I didn't see his work in applying + of neal's patch. + ok + how do they relate with each other? + (I have never actually looked at either of them :/) + his work in kernel and neal's patch? + yes + They do not correlate with each other. + ah, I must be misremembering what each of them do + in kam's patch was changes to support sequential reading in reverse + order (as in OSF/Mach), but posix does not support such behavior, so I + didn't implement this either. + I can't find the pointer to neal's patch, do you have it off-hand? + http://comments.gmane.org/gmane.os.hurd.bugs/351 + thx + I think we are not talking about the same patch from Karim + I mean lists.gnu.org/archive/html/bug-hurd/2010-06/msg00023.html + I mean this patch: + http://lists.gnu.org/archive/html/bug-hurd/2010-06/msg00024.html + Oh. + ok + seems, this is just the same + yes + from a non-expert view, I would have thought these patches play + hand in hand, do they really? + this patch is completely for kernel and neal's one is completely + for libpager. + i.e. neal's fixes libpager, and karim's fixes the kernel + yes + ending up with fixing the whole path? + AIUI, karim's patch will be needed so that your increased readahead + will end up with clustered page request? + I will not use kam's patch + is it not needed to actually get pages in together? + how do you tell libpager to fetch pages together? + about the cluster size, I'd say it shouldn't be specified at + vm_advise() level + in other OSes, it is usually automatically tuned + by ramping it up to a maximum readahead size (which, however, could + be specified) + that's important for the normal policy, where there are typically + successive periods of sequential reads, but you don't know in advance for + how long + braunr said that there are legal issues with his code, so I cannot + use it. + did i ? + mcsim: can you give me a link to the code again please ? + see above :) + which one ? + both + they only differ by a typo + mcsim: i don't remember saying that, do you have any link ? + or log ? + sorry, can you rephrase "ending up with fixing the whole path"? + cluster_size in vm_advise also could be considered as advise + no + it must be the third time we're talking about this + mcsim: I mean both parts would be needed to actually achieve + clustered i/o + again, why make cluster_size a per object attribute ? :( + wouldn't some objects benefit from bigger cluster sizes, while + others wouldn't? + but again, I believe it should rather be autotuned + (for each object) + if we merely want posix compatibility (and for a first attempt, + it's quite enough), vm_advise is good, and the kernel selects the + implementation (and thus the cluster sizes) + if we want finer grained control, perhaps a per pager cluster_size + would be good, although its efficiency depends on several parameters + (e.g. where the page is in this cluster) + but a per object cluster size is a large waste of memory + considering very few applications (if not none) would use the "feature" + .. + (if any*) + there must be a misunderstanding + why would it be a waste of memory? + "per object" + so? + there can be many memory objects in the kernel + so? + so such an overhead must be useful to accept it + in my understanding, a cluster size per object is just a mere + integer for each object + what overhead? + yes + don't we have just thousands of objects? + for now + remember we're trying to remove the page cache limit :) + that still won't be more than tens of thousands of objects + times an integer + that's completely neglectible + braunr: Strange, Can't find in logs. Weird things are happening in + my memory :/ Sorry. + mcsim: i'm almost sure i never said that :/ + but i don't trust my memory too much either + youpi: depends + mcsim: I mean both parts would be needed to actually achieve + clustered i/o + braunr: I made I call vm_advise that applies policy to memory range + (vm_map_entry to be specific) + mcsim: good + actually the cluster size should even be per memory range + youpi: In this sense, yes + k + sorry, Internet connection lags + when changing a structure used to create many objects, keep in + mind one thing + if its size gets larger than a threshold (currently, powers of + two), the cache used by the slab allocator will allocate twice the + necessary amount + sure + this is the case with most object caching allocators, although + some can have specific caches for common sizes such as 96k which aren't + powers of two + anyway, an integer is negligible, but the final structure size + must be checked + (for both 32 and 64 bits) + braunr: ok. + But I didn't understand what should be done with cluster size in + vm_advise? Should I delete it? + to me, the cluster size is a pager property + to me, the cluster size is a map property + whereas vm_advise indicates what applications want + you could have several process accessing the same file in different + ways + youpi: that's why there is a policy + isn't cluster_size part of the policy? + but if the pager abilities are limited, it won't change much + i'm not sure + cluster_size is the amount of readahead, isn't it? + no, it's the amount of data in a single transfer + Yes, it is. + ok, i'll have to check your code + shouldn't transfers permit unbound amounts of data? + braunr: than I misunderstand what readahead is + well then cluster size is per policy :) + e.g. random => 0, normal => 3, sequential => 15 + why make it per map entry ? + because it depends on what the application doezs + let me check the code + if it's accessing randomly, no need for big transfers + just page transfers will be fine + if accessing sequentially, rather use whole MiB of transfers + and these behavior can be for the same file + mcsim: the call is vm_advi*s*e + mcsim: the call is vm_advi_s_e + not advice + yes, he agreed earlier + ok + cluster_size is the amount of data that I try to read at one time. + at singe mo_data_request + *single + which, to me, will depend on the actual map + ok so it is the transfer size + and should be autotuned, especially for normal behavior + youpi: it makes no sense to have both the advice and the actual + size per map entry + to get big readahead with all apps + braunr: the size is not only dependent on the advice, but also on + the application behavior + youpi: how does this application tell this ? + even for sequential, you shouldn't necessarily use very big amounts + of transfers + there is no need for the advice if there is a cluster size + there can be, in the case of sequential, as we said, to clear + previous pages + but otherwise, indeed + but for me it's the converse + the cluster size should be tuned anyway + and i'm against giving the cluster size in the advise call, as we + may want to prefetch previous data as well + I don't see how that collides + well, if you consider it's the transfer size, it doesn't + to me cluster size is just the size of a window + if you consider it's the amount of pages following a faulted page, + it will + also, if your policy says e.g. "3 pages before, 10 after", and + your cluster size is 2, what happens ? + i would find it much simpler to do what other VM variants do: + compute the I/O sizes directly from the policy + don't they autotune, and use the policy as a maximum ? + depends on the implementations + ok, but yes I agree + although casting the size into stone in the policy looks bogus to + me + but making cluster_size part of the kernel interface looks way too + messy + it is + that's why i would have thought it as part of the pager properties + the pager is the true component besides the kernel that is + actually involved in paging ... + well, for me the flexibility should still be per application + by pager you mean the whole pager, not each file, right? + if a pager can page more because e.g. it's a file system with big + block sizes, why not fetch more ? + yes + it could be each file + but only if we have use for it + and i don't see that currently + well, posix currently doesn't provide a way to set it + so it would be useless atm + i was thinking about our hurd pagers + could we perhaps say that the policy maximum could be a fraction of + available memory? + why would we want that ? + (total memory, I mean) + to make it not completely cast into stone + as have been in the past in gnumach + i fail to understand :/ + there must be a misunderstanding then + (pun not intended) + why do you want to limit the policy maximum ? + how to decide it? + the pager sets it + actually I don't see how a pager could decide it + on what ground does it make the decision? + readahead should ideally be as much as 1MiB + 02:02 < braunr> if a pager can page more because e.g. it's a file + system with big block sizes, why not fetch more ? + is the example i have in mind + otherwise some default values + that's way smaller than 1MiB, isn't it? + yes + and 1 MiB seems a lot to me :) + for readahead, not really + maybe for sequential + that's what we care about! + ah, i thought we cared about normal + "as much as 1MiB", I said + I don't mean normal :) + right + but again, why limit ? + we could have 2 or more ? + at some point you don't get more efficiency + but eat more memory + having the pager set the amount allows us to easily adjust it over + time + braunr: Do you think that readahead should be implemented in + libpager? + than needed + mcsim: no + mcsim: err + mcsim: can't answer + mcsim: do you read the log of what you have missed during + disconnection? + i'm not sure about what libpager does actually + yes + for me it's just mutualisation of code used by pagers + i don't know the details + youpi: yes + youpi: that's why we want these values not hardcoded in the kernel + youpi: so that they can be adjusted by our shiny user space OS + (btw apparently linux uses minimum 16k, maximum 128 or 256k) + that's more reasonable + that's just 4 times less :) + braunr: You say that pager should decide how much data should be + read ahead, but each pager can't implement it on it's own as there will + be too much overhead. So the only way is to implement this in libpager. + mcsim: gni ? + why couldn't they ? + mcsim: he means the size, not the actual implementation + the maximum size, actually + actually, i would imagine it as the pager giving per policy + parameters + right + like how many before and after + I agree, then + the kernel could limit, sure, to avoid letting pagers use + completely insane values + (and that's just a max, the kernel autotunes below that) + why not + that kernel limit could be a fraction of memory, then? + it could, yes + i see what you mean now + mcsim: did you understand our discussion? + don't hesitate to ask for clarification + I supposed cluster_size to be such parameter. And advice will help + to interpret this parameter (whether data should be read after fault page + or some data should be cleaned before) + mcsim: we however believe that it's rather the pager than the + application that would tell that + at least for the default values + posix doesn't have a way to specify it, and I don't think it will + in the future + and i don't think our own hurd-specific programs will need more + than that + if they do, we can slightly change the interface to make it a per + object property + i've checked the slab properties, and it seems we can safely add + it per object + cf http://www.sceen.net/~rbraun/slabinfo.out + so it would still be set by the pager, but if depending on the + object, the pager could set different values + youpi: do you think the pager should just provide one maximum size + ? or per policy sizes ? + I'd say per policy size + so people can increase sequential size like crazy when they know + their sequential applications need it, without disturbing the normal + behavior + right + so the last decision is per pager or per object + mcsim: i'd say whatever makes your implementation simpler :) + braunr: how kernel knows that object are created by specific pager? + that's the kind of things i'm referring to with "whatever makes + your implementation simpler" + but usually, vm_objects have an ipc port and some properties + relatedto their pagers + -usually + the problem i had in mind was the locking protocol but our spin + locks are noops, so it will be difficult to detect deadlocks + braunr: and for every policy there should be variable in vm_object + structure with appropriate cluster_size? + if you want it per object, yes + although i really don't think we want it + better keep it per pager for now + let's imagine youpi finishes his 64-bits support, and i can + successfully remove the page cache limit + we'd jump from 1.8 GiB at most to potentially dozens of GiB of RAM + and 1.8, mostly unused + to dozens almost completely used, almost all the times for the + most interesting use cases + we may have lots and lots of objects to keep around + so if noone really uses the feature ... there is no point + but also lots and lots of memory to spend on it :) + a lot of objects are just one page, but a lof of them are not + sure + we wouldn't be doing that otherwise :) + i'm just saying there is no reason to add the overhead of several + integers for each object if they're simply not used at all + hmm, 64-bits, better page cache, clustered paging I/O :> + (and readahead included in the last ofc) + good night ! + than, probably, make system-global max-cluster_size? This will save + some memory. Also there is usually no sense in reading really huge chunks + at once. + but that'd be tedious to set + there are only a few pagers, that's no wasted memory + the user being able to set it for his own pager is however a very + nice feature, which can be very useful for databases, image processing, + etc. + In conclusion I have to implement following: 3 memory policies per + object and per vm_map_entry. Max cluster size for every policy should be + set per pager. + So, there should be 2 system calls for setting memory policy and + one for setting cluster sizes. + Also amount of data to transfer should be tuned automatically by + every page fault. + youpi: Correct me, please, if I'm wrong. + I believe that's what we ended up to decide, yes + + +## IRC, freenode, #hurd, 2012-07-02 + + is it safe to say that all memory objects implemented by external + pagers have "file" semantics ? + i wonder if the current memory manager interface is suitable for + device pagers + braunr: What does "file" semantics mean? + mcsim: anonymous memory doesn't have the same semantics as a file + for example + anonymous memory that is discontiguous in physical memory can be + contiguous in swap + and its location can change with time + whereas with a memory object, the data exchanged with pagers is + identified with its offset + in (probably) all other systems, this way of specifying data is + common to all files, whatever the file system + linux uses the struct vm_file name, while in BSD/Solaris they are + called vnodes (the link between a file system inode and virtual memory) + my question is : can we implement external device pagers with the + current interface, or is this interface really meant for files ? + also + mcsim: something about what you said yesterday + 02:39 < mcsim> In conclusion I have to implement following: 3 + memory policies per object and per vm_map_entry. Max cluster size for + every policy should be set per pager. + not per object + one policy per map entry + transfer parameters (pages before and after the faulted page) per + policy, defined by pagers + 02:39 < mcsim> So, there should be 2 system calls for setting + memory policy and one for setting cluster sizes. + adding one call for vm_advise is good because it mirrors the posix + call + but for the parameters, i'd suggest changing an already existing + call + not sure which one though + braunr: do you know how mo_change_attributes implemented in + OSF/Mach? + after a quick reading of the reference manual, i think i + understand why they made it per object + mcsim: no + did they change the call to include those paging parameters ? + it accept two parameters: flavor and pointer to structure with + parameters. + flavor determines semantics of structure with parameters. + + http://www.darwin-development.org/cgi-bin/cvsweb/osfmk/src/mach_kernel/vm/memory_object.c?rev=1.1 + structure can have 3 different views and what exect view will be is + determined by value of flavor + So, I thought about implementing similar call that could be used + for various purposes. + like ioctl + "pointer to structure with parameters" <= which one ? + mcsim: don't model anything anywhere like ioctl please + memory_object_info_t attributes + ioctl is the very thing we want NOT to have on the hurd + ok attributes + and what are the possible values of flavour, and what kinds of + attributes ? + and then appears something like this on each case: behave = + (old_memory_object_behave_info_t) attributes; + ok i see + flavor could be OLD_MEMORY_OBJECT_BEHAVIOR_INFO, + MEMORY_OBJECT_BEHAVIOR_INFO, MEMORY_OBJECT_PERFORMANCE_INFO etc + i don't really see the point of flavour here, other than + compatibility + having attributes is nice, but you should probably add it as a + call parameter, not inside a structure + as a general rule, we don't like passing structures too much + to/from the kernel, because handling them with mig isn't very clean + ok + What policy parameters should be defined by pager? + i'd say number of pages to page-in before and after the faulted + page + Only pages before and after the faulted page? + for me yes + youpi might have different things in mind + the page cleaning in sequential mode is something i wouldn't do + 1/ applications might want data read sequentially to remain in the + cache, for other sequential accesses + 2/ applications that really don't want to cache anything should + use O_DIRECT + 3/ it's complicated, and we're in july + i'd rather have a correct and stable result than too many unused + features + braunr: MADV_SEQUENTIAL Expect page references in sequential order. + (Hence, pages in the given range can be aggressively read ahead, and may + be freed soon after they are accessed.) + this is from linux man + braunr: Can I at least make keeping in mind that it could be + implemented? + I mean future rpc interface + braunr: From behalf of kernel pager is just a port. + That's why it is not clear for me how I can make in kernel + per-pager policy + mcsim: you can't + 15:19 < braunr> after a quick reading of the reference manual, i + think i understand why they made it per object + + http://pubs.opengroup.org/onlinepubs/009695399/functions/posix_madvise.html + POSIX_MADV_SEQUENTIAL + Specifies that the application expects to access the specified + range sequentially from lower addresses to higher addresses. + linux might free pages after their access, why not, but this is + entirely up to the implementation + I know, when but applications might want data read sequentially to + remain in the cache, for other sequential accesses this kind of access + could be treated rather normal or random + we can do differently + mcsim: no + sequential means the access will be sequential + so aggressive readahead (e.g. 0 pages before, many after), should + be used + for better performance + from my pov, it has nothing to do with caching + i actually sometimes expect data to remain in cache + e.g. before playing a movie from sshfs, i sometimes prefetch it + using dd + then i use mplayer + i'd be very disappointed if my data didn't remain in the cache :) + At least these pages could be placed into inactive list to be first + candidates for pageout. + that's what will happen by default + mcsim: if we need more properties for memory objects, we'll adjust + the call later, when we actually implement them + so, first call is vm_advise and second is changed + mo_change_attributes? + yes + there will appear 3 new parameters in mo_c_a: policy, pages before + and pages after? + braunr: With vm_advise I didn't understand one thing. This call is + defined in defs file, so that should mean that vm_advise is ordinal rpc + call. But on the same time it is defined as syscall in mach internals (in + mach_trap_table). + mcsim: what ? + were is it "defined" ? (it doesn't exit in gnumach currently) + Ok, let consider vm_map + I define it both in mach_trap_table and in defs file. + But why? + uh ? + let me see + Why defining in defs file is not enough? + and previous question: there will appear 3 new parameters in + mo_c_a: policy, pages before and pages after? + mcsim: give me the exact file paths please + mcsim: we'll discuss the new parameters after + kern/syscall_sw.c + right i see + here mach_trap_table in defined + i think they're not used + they were probably introduced for performance + and ./include/mach/mach.defs + don't bother adding vm_advise as a syscall + about the parameters, it's a bit more complicated + you should add 6 parameters + before and after, for the 3 policies + but + as seen in the posix page, there could be more policies .. + ok forget what i said, it's stupid + yes, the 3 parameters you had in mind are correct + don't forget a "don't change" value for the policy though, so the + kernel ignores the before/after values if we don't want to change that + ok + mcsim: another reason i asked about "file semantics" is the way we + handle the cache + mcsim: file semantics imply data is cached, whereas anonymous and + device memory usually isn't + (although having the cache at the vm layer instead of the pager + layer allows nice things like the swap cache) + But this shouldn't affect possibility of implementing of device + pager. + yes it may + consider how a fault is actually handled by a device + mach must use weird fictitious pages for that + whereas it would be better to simply let the pager handle the + fault as it sees fit + setting may_cache to false should resolve the issue + for the caching problem, yes + which is why i still think it's better to handle the cache at the + vm layer, unlike UVM which lets the vnode pager handle its own cache, and + removes the vm cache completely + The only issue with pager interface I see is implementing of + scatter-gather DMA (as current interface does not support non-consecutive + access) + right + but that's a performance issue + my problem with device pagers is correctness + currently, i think the kernel just asks pagers for "data" + whereas a device pager should really map its device memory where + the fault happen + braunr: You mean that every access to memory should cause page + fault? + I mean mapping of device memory + no + i mean a fault on device mapped memory should directly access a + shared region + whereas file pagers only implement backing store + let me explain a bit more + here is what happens with file mapped memory + you map it, access it (some I/O is done to get the page content in + physical memory), then later it's flushed back + whereas with device memory, there shouldn't be any I/O, the device + memory should directly be mapped (well, some devices need the same + caching behaviour, while others provide direct access) + one of the obvious consequences is that, when you map device + memory (e.g. a framebuffer), you expect changes in your mapped memory to + be effective right away + while with file mapped memory, you need to msync() it + (some framebuffers also need to be synced, which suggests greater + control is needed for external pagers) + Seems that I understand you. But how it is implemented in other + OS'es? Do they set something in mmu? + mcsim: in netbsd, pagers have a fault operatin in addition to get + and put + the device pager sets get and put to null and implements fault + only + the fault callback then calls the d_mmap callback of the specific + driver + which usually results in the mmu being programmed directly + (e.g. pmap_enter or similar) + in linux, i think raw device drivers, being implemented as + character device files, must provide raw read/write/mmap/etc.. functions + so it looks pretty much similar + i'd say our current external pager interface is insufficient for + device pagers + but antrik may know more since he worked on ggi + antrik: ^ + braunr: Seems he used io_map + mcsim: where ar eyou looking at ? the incubator ? + his master's thesis + ah the thesis + but where ? :) + I'll give you a link + http://dl.dropbox.com/u/36519904/kgi_on_hurd.pdf + thanks + see p 158 + arg, more than 200 pages, and he says he's lazy :/ + mcsim: btw, have a look at m_o_ready + braunr: This is old form of mo_change attributes + I'm not going to change it + mcsim: these are actually the default object parameters right ? + mcsim: if you don't change it, it means the kernel must set + default values until the pager changes them, if it does + yes. + mcsim: madvise() on Linux has a separate flag to indicate that + pages won't be reused. thus I think it would *not* be a good idea to + imply it in SEQUENTIAL + braunr: yes, my KMS code relies on mapping memory objects for the + framebuffer + (it should be noted though that on "modern" hardware, mapping + graphics memory directly usually gives very poor performance, and drivers + tend to avoid it...) + mcsim: BTW, it was most likely me who warned about legal issues + with KAM's work. AFAIK he never managed to get the copyright assignment + done :-( + (that's not really mandatory for the gnumach work though... only + for the Hurd userspace parts) + also I'd like to point out again that the cluster_size argument + from OSF Mach was probably *not* meant for advice from application + programs, but rather was supposed to reflect the cluster size of the + filesystem in question. at least that sounds much more plausible to me... + braunr: I have no idea whay you mean by "device pager". device + memory is mapped once when the VM mapping is established; there is no + need for any fault handling... + mcsim: to be clear, I think the cluster_size parameter is mostly + orthogonal to policy... and probably not very useful at all, as ext2 + almost always uses page-sized clusters. I'm strongly advise against + bothering with it in the initial implementation + mcsim: to avoid confusion, better use a completely different name + for the policy-decided readahead size + antrik: ok + braunr: well, yes, the thesis report turned out HUGE; but the + actual work I did on the KGI port is fairly tiny (not more than a few + weeks of actual hacking... everything else was just brooding) + braunr: more importantly, it's pretty much the last (and only + non-trivial) work I did on the Hurd :-( + (also, I don't think I used the word "lazy"... my problem is not + laziness per se; but rather inability to motivate myself to do anything + not providing near-instant gratification...) + antrik: right + antrik: i shouldn't consider myself lazy either + mcsim: i agree with antrik, as i told you weeks ago + about + 21:45 < antrik> mcsim: to be clear, I think the cluster_size + parameter is mostly orthogonal to policy... and probably not very useful + at all, as ext2 almost always uses page-sized clusters. I'm strongly + advise against bothering with it + in the initial implementation + antrik: but how do you actually map device memory ? + also, strangely enough, here is the comment in dragonflys + madvise(2) + 21:45 < antrik> mcsim: to be clear, I think the cluster_size + parameter is mostly orthogonal to policy... and probably not very useful + at all, as ext2 almost always uses page-sized clusters. I'm strongly + advise against bothering with it + in the initial implementation + arg + MADV_SEQUENTIAL Causes the VM system to depress the priority of + pages immediately preceding a given page when it is faulted in. + braunr: interesting... + (about SEQUENTIAL on dragonfly) + as for mapping device memory, I just use to device_map() on the + mem device to map the physical address space into a memory object, and + then through vm_map into the driver (and sometimes application) address + space + formally, there *is* a pager involved of course (implemented + in-kernel by the mem device), but it doesn't really do anything + interesting + thinking about it, there *might* actually be page faults involved + when the address ranges are first accessed... but even then, the handling + is really trivial and not terribly interesting + antrik: it does the most interesting part, create the physical + mapping + and as trivial as it is, it requires a special interface + i'll read about device_map again + but yes, the fact that it's in-kernel is what solves the problem + here + what i'm interested in is to do it outside the kernel :) + why would you want to do that? + there is no policy involved in doing an MMIO mapping + you ask for the pysical memory region you are interested in, and + that's it + whether the kernel adds the page table entries immediately or on + faults is really an implementation detail + braunr: ^ + yes it's a detail + but do we currently have the interface to make such mappings from + userspace ? + and i want to do that because i'd like as many drivers as possible + outside the kernel of course + again, the userspace driver asks the kernel to establish the + mapping (through device_map() and then vm_map() on the resulting memory + object) + hm i'm missing something + + http://www.gnu.org/software/hurd/gnumach-doc/Device-Map.html#Device-Map + <= this one ? + yes, this one + but this implies the device is implemented by the kernel + the mem device is, yes + but that's not a driver + ah + it's just the interface for doing MMIO + (well, any physical mapping... but MMIO is probably the only real + use case for that) + ok + i was thinking about completely removing the device interface from + the kernel actually + but it makes sense to have such devices there + well, in theory, specific kernel drivers can expose their own + device_map() -- but IIRC the only one that does (besides mem of course) + is maptime -- which is not a real driver either... + +[[Mapped-time_interface|microkernel/mach/gnumach/interface/device/time]]. + + oh btw, i didn't know you had a blog :) + well, it would be possible to replace the device interface by + specific interfaces for the generic pseudo devices... I'm not sure how + useful that would be + there are lots of interesting stuff there + hehe... another failure ;-) + failure ? + well, when I realized that I'm speding a lot of time pondering + things, and never can get myself to actually impelemnt any of them, I had + the idea that if I write them down, there might at least be *some* good + from it... + unfortunately it turned out that I need so much effort to write + things down, that most of the time I can't get myself to do that either + :-( + i see + well it's still nice to have it + (notice that the latest entry is two years old... and I haven't + even started describing most of my central ideas :-( ) + antrik: i tried to create a blog once, and found what i wrote so + stupid i immediately removed it + hehe + actually some of my entries seem silly in retrospect as well... + but I guess that's just the way it is ;-) + :) + i'm almost sure other people would be interested in what i had to + say + BTW, I'm actually not sure whether the Mach interfaces are + sufficient to implement GEM/TTM... we would certainly need kernel support + for GART (as for any other kind IOMMU in fact); but beyond that it's not + clear to me + GEM ? TTM ? GART ? + GEM = Graphics Execution Manager. part of the "new" DRM interface, + closely tied with KMS + TTM = Translation Table Manager. does part of the background work + for most of the GEM drivers + "The Graphics Execution Manager (GEM) is a computer software + system developed by Intel to do memory management for device drivers for + graphics chipsets." hmm + (in fact it was originally meant to provide the actual interface; + but the Inter folks decided that it's not useful for their UMA graphics) + GART = Graphics Aperture + kind of an IOMMU for graphics cards + allowing the graphics card to work with virtual mappings of main + memory + (i.e. allowing safe DMA) + ok + all this graphics stuff looks so complex :/ + it is + I have a whole big chapter on that in my thesis... and I'm not + even sure I got everything right + what is nvidia using/doing (except for getting the finger) ? + flushing out all the details for KMS, GEM etc. took the developers + like two years (even longer if counting the history of TTM) + Nvidia's proprietary stuff uses a completely own kernel interface, + which is of course not exposed or docuemented in any way... but I guess + it's actually similar in what it does) + ok + (you could ask the nouveau guys if you are truly + interested... they are doing most of their reverse engineering at the + kernel interface level) + it seems graphics have very special needs, and a lot of them + and the interfaces are changing often + so it's not that much interesting currently + it just means we'll probably have to change the mach interface too + like you said + so the answer to my question, which was something like "do mach + external pagers only implement files ?", is likely yes + well, KMS/GEM had reached some stability; but now there are + further changes ahead with the embedded folks coming in with all their + dedicated hardware, calling for unified buffer management across the + whole pipeline (from capture to output) + and yes: graphics hardware tends to be much more complex regarding + the interface than any other hardware. that's because it's a combination + of actual I/O (like most other devices) with a very powerful coprocessor + and the coprocessor part is pretty much unique amongst peripherial + devices + (actually, the I/O part is also much more complex than most other + hardware... but that alone would only require a more complex driver, not + special interfaces) + embedded hardware makes it more interesting in that the I/O + part(s) are separate from the coprocessor ones; and that there are often + several separate specialised ones of each... the DRM/KMS stuff is not + prepared to deal with this + v4l over time has evolved to cover such things; but it's not + really the right place to implement graphics drivers... which is why + there are not efforts to unify these frameworks. funny times... + + +## IRC, freenode, #hurd, 2012-07-03 + + mcsim: vm_for_every_page should be static + braunr: ok + mcsim: see http://gcc.gnu.org/onlinedocs/gcc/Inline.html + and it looks big enough that you shouldn't make it inline + let the compiler decide for you (which is possible only if the + function is static) + (otherwise a global symbol needs to exist) + mcsim: i don't know where you copied that comment from, but you + should review the description of the vm_advice call in mach.Defs + braunr: I see + braunr: It was vm_inherit :) + mcsim: why isn't NORMAL defined in vm_advise.h ? + mcsim: i figured actually ;) + braunr: I was going to do it later when. + mcsim: for more info on inline, see + http://www.kernel.org/doc/Documentation/CodingStyle + arg that's an old one + braunr: I know that I do not follow coding style + mcsim: this one is about linux :p + mcsim: http://lxr.linux.no/linux/Documentation/CodingStyle should + have it + mcsim: "Chapter 15: The inline disease" + I was going to fix it later during refactoring when I'll merge + mplaneta/gsoc12/working to mplaneta/gsoc12/master + be sure not to forget :p + and the best not to forget is to do it asap + +way + As to inline. I thought that even if I specify function as inline + gcc makes final decision about it. + There was a specifier that made function always inline, AFAIR. + gcc can force a function not to be inline, yes + but inline is still considered as a strong hint + + +## IRC, freenode, #hurd, 2012-07-05 + + braunr: hello. You've said that pager has to supply 2 values to + kernel to give it an advice how execute page fault. These two values + should be number of pages before and after the page where fault + occurred. But for sequential policy number of pager before makes no + sense. For random policy too. For normal policy it would be sane to make + readahead symmetric. Probably it would be sane to make pager supply + cluster_size (if it is necessary to supply any) that w + *that will be advice for kernel of least sane value? And maximal + value will be f(free_memory, map_entry_size)? + mcsim1: I doubt symmetric readahead would be a good default + policy... while it's hard to estimate an optimum over all typical use + cases, I'm pretty sure most situtations will benefit almost exclusively + from reading following pages, not preceeding ones + I'm not even sure it's useful to read preceding pages at all in + the default policy -- the use cases are probably so rare that the penalty + in all other use cases is not justified. I might be wrong on that + though... + I wonder how other systems handle that + antrik: if there is a mismatch between pages and the underlying + store, like why changing small bits of data on an ssd is slow? + mcsim1: i don't see why not + antrik: netbsd reads a few pages before too + actually, what netbsd does vary on the version, some only mapped + in resident pages, later versions started asynchronous transfers in the + hope those pages would be there + LarstiQ: not sure what you are trying to say + in linux : + 321 * MADV_NORMAL - the default behavior is to read clusters. + This + 322 * results in some read-ahead and read-behind. + not sure if it's actually what the implementation does + well, right -- it's probably always useful to read whole clusters + at a time, especially if they are the same size as pages... that doesn't + mean it always reads preceding pages; only if the read is in the middle + of the cluster AIUI + antrik: basically what braunr just pasted + and in most cases, we will want to read some *following* clusters + as well, but probably not preceding ones + * LarstiQ nods + antrik: the default policy is usually rather sequential + here are the numbers for netbsd + 166 static struct uvm_advice uvmadvice[] = { + 167 { MADV_NORMAL, 3, 4 }, + 168 { MADV_RANDOM, 0, 0 }, + 169 { MADV_SEQUENTIAL, 8, 7}, + 170 }; + struct uvm_advice { + int advice; + int nback; + int nforw; + }; + surprising isn't it ? + they may suggest sequential may be backwards too + makes sense + braunr: what are these numbers? pages? + yes + braunr: I suspect the idea behind SEQUENTIAL is that with typical + sequential access patterns, you will start at one end of the file, and + then go towards the other end -- so the extra clusters in the "wrong" + direction do not actually come into play + only situation where some extra clusters are actually read is when + you start in the middle of a file, and thus do not know yet in which + direction the sequential read will go... + yes, there are similar comments in the linux code + mcsim1: so having before and after numbers seems both + straightforward and in par with other implementations + I'm still surprised about the almost symmetrical policy for NORMAL + though + BTW, is it common to use heuristics for automatically recognizing + random and sequential patterns in the absence of explicit madise? + i don't know + netbsd doesn't use any, linux seems to have different behaviours + for anonymous and file memory + when KAM was working on this stuff, someone suggested that... + there is a file_ra_state struct in linux, for per file read-ahead + policy + now the structure is of course per file system, since they all use + the same address + (which is why i wanted it to be per pager in the first place) + mcsim1: as I said before, it might be useful for the pager to + supply cluster size, if it's different than page size. but right now I + don't think this is something worth bothering with... + I seriously doubt it would be useful for the pager to supply any + other kind of policy + braunr: I don't understand your remark about using the same + address... + braunr: pre-mapping seems the obvious way to implement readahead + policy + err... per-mapping + the ra_state (read ahead state) isn't the policy + the policy is per mapping, parts of the implementation of the + policy is per file system + braunr: How do you look at following implementation of NORMAL + policy: We have fault page that is current. Than we have maximal size of + readahead block. First we find first absent pages before and after + current. Than we try to fit block that will be readahead into this + range. Here could be following situations: in range RBS/2 (RBS -- size of + readahead block) there is no any page, so readahead will be symmetric; if + current page is first absent page than all + RBS block will consist of pages that are after current; on the + contrary if current page is last absent than readahead will go backwards. + Additionally if current page is approximately in the middle of the + range we can decrease RBS, supposing that access is random. + mcsim1: i think your gsoc project is about readahead, we're in + july, and you need to get the job done + mcsim1: grab one policy that works, pages before and after are + good enough + use sane default values, let the pagers decide if they want + something else + and concentrate on the real work now + braunr: I still don't see why pagers should mess with that... only + complicates matters IMHO + antrik: probably, since they almost all use the default + implementation + mcsim1: just use sane values inside the kernel :p + this simplifies things by only adding the new vm_advise call and + not change the existing external pager interface + + +## IRC, freenode, #hurd, 2012-07-12 + + mcsim: so, to begin with, tell us what state you've reached please + braunr: I'm writing code for hurd and gnumach. For gnumach I'm + implementing memory policies now. RANDOM and NORMAL seems work, but in + hurd I found error that I made during editing ext2fs. So for now ext2fs + does not work + policies ? + what about mechanism ? + also I moved some translators to new interface. + It works too + well that's impressive + braunr: I'm not sure yet that everything works + right, but that's already a very good step + i thought you were still working on the interfaces to be honest + And with mechanism I didn't implement moving pages to inactive + queue + what do you mean ? + ah you mean with the sequential policy ? + yes + you can consider this a secondary goal + sequential I was going to implement like you've said, but I still + want to support moving pages to inactive queue + i think you shouldn't + first get to a state where clustered transfers do work fine + policies are implemented in function calculate_clusters + then, you can try, and measure the difference + ok. I'm now working on fixing ext2fs + so, except from bug squashing, what's left to do ? + finish policies and ext2fs; move fatfs, ufs, isofs to new + interface; test this all; edit patches from debian repository, that + conflict with my changes; rearrange commits and fix code indentation; + update documentation; + think about measurements too + mcsim: Please don't spend a lot of time on ufs. No testing + required for that one. + and keep us informed about your progress on bug fixing, so we can + test soon + Forgot about moving system to new interfaces (I mean determine form + of vm_advise and memory_object_change_attributes) + s/determine/final/ + braunr: ok. + what do you mean "moving system to new interfaces" ? + braunr: I also pushed code changes to gnumach and hurd git + repositories + I met an issue with memory_object_change_attributes when I tried to + use it as I have to update all applications that use it. This includes + libc and translators that are not in hurd repository or use debian + patches. So I will not be able to run system with new + memory_object_change_attributes interface, until I update all software + that use this rpc + this is a bit like the problem i had with my change + the solution is : don't do it + i mean, don't change the interface in an incompatible way + if you can't change an existing call, add a new one + temporary I changed memory_object_set_attributes as it isn't used + any more. + braunr: ok. Adding new call is a good idea :) + + +## IRC, freenode, #hurd, 2012-07-16 + + mcsim: how did you deal with multiple page transfers towards the + default pager ? + braunr: hello. Didn't handle this yet, but AFAIR default pager + supports multiple page transfers. + mcsim: i'm almost sure it doesn't + braunr: indeed + braunr: So, I'll update it just other translators. + like other translators you mean ? + *just as + braunr: yes + ok + be aware also that it may need some support in vm_pageout.c in + gnumach + braunr: thank you + if you see anything strange in the default pager, don't hesitate + to talk about it + braunr: ok. I didn't finish with ext2fs yet. + so it's a good thing you're aware of it now, before you begin + working on it :) + braunr: I'm working on ext2 now. + yes i understand + i meant "before beginning work on the default pager" + ok + + mcsim: BTW, we were mostly talking about readahead (pagein) over + the past weeks, so I wonder what the status on clustered page*out* is?... + antrik: I don't work on this, but following, I think, is an example + of *clustered* pageout: _pager_seqnos_memory_object_data_return: object = + 113, seqno = 4, control = 120, start_address = 0, length = 8192, dirty = + 1. This is an example of debugging printout that shows that pageout + manipulates with chunks bigger than page sized. + antrik: Another one with bigger length + _pager_seqnos_memory_object_data_return: object = 125, seqno = 124, + control = 132, start_address = 131072, length = 126976, dirty = 1, kcopy + mcsim: that's odd -- I didn't know the functionality for that even + exists in our codebase... + my understanding was that Mach always sends individual pageout + requests for ever single page it wants cleaned... + (and this being the reason for the dreadful thread storms we are + facing...) + antrik: ok + antrik: yes that's what is happening + the thread storms aren't that much of a problem now + (by carefully throttling pageouts, which is a task i intend to + work on during the following months, this won't be an issue any more) + + +## IRC, freenode, #hurd, 2012-07-19 + + I moved fatfs, ufs, isofs to new interface, corrected some errors + in other that I already moved, moved kernel to new interface (renamed + vm_advice to vm_advise and added rpcs memory_object_set_advice and + memory_object_get_advice). Made some changes in mechanism and tried to + finish ext2 translator. + braunr: I've got an issue with fictitious pages... + When I determine bounds of cluster in external object I never know + its actual size. So, mo_data_request call could ask data that are behind + object bounds. The problem is that pager returns data that it has and + because of this fictitious pages that were allocated are not freed. + why don't you know the size ? + I see 2 solutions. First one is do not allocate fictitious pages at + all (but I think that there could be issues). Another lies in allocating + fictitious pages, but then freeing them with mo_data_lock. + braunr: Because pages does not inform kernel about object size. + i don't understand what you mean + I think that second way is better. + so how does it happen ? + you get a page fault + Don't you understand problem or solutions? + then a lookup in the map finds the map entry + and the map entry gives you the link to the underlying object + from vm_object.h: vm_size_t size; /* + Object size (only valid if internal) */ + mcsim: ugh + For external they are either 0x8000 or 0x20000... + and for internal ? + i'm very surprised to learn that + braunr: for internal size is actual + right sorry, wrong question + did you find what 0x8000 and 0x20000 are ? + for external I met only these 2 magic numbers when printed out + arguments of functions _pager_seqno_memory_object_... when they were + called. + yes but did you try to find out where they come from ? + braunr: no. I think that 0x2000(many zeros) is maximal possible + object size. + what's the exact value ? + can't tell exactly :/ My hurd box has broken again. + mcsim: how does the vm find the backing content then ? + braunr: Do you know if it is guaranteed that map_entry size will be + not bigger than external object size? + mcsim: i know it's not + but you can use the map entry boundaries though + braunr: vm asks pager + but if the page is already present + how does it know ? + it must be inside a vm_object .. + If I can use these boundaries than the problem, I described is not + actual. + good + it makes sense to use these boundaries, as the application can't + use data outside the mapping + I ask page with vm_page_lookup + it would matter for shared objects, but then they have their own + faults :p + ok + so the size is actually completely ignord + if it is present than I stop expansion of cluster. + which makes sense + braunr: yes, for external. + all right + use the mapping boundaries, it will do + mcsim: i have only one comment about what i could see + mcsim: there are 'advice' fields in both vm_map_entry and + vm_object + there should be something else in vm_object + i told you about pages before and after + mcsim: how are you using this per object "advice" currently ? + (in addition, using the same name twice for both mechanism and + policy is very sonfusing) + confusing* + braunr: I try to expand cluster as much as it possible, but not + much than limit + they both determine policy, but advice for entry has bigger + priority + that's wrong + mapping and content shouldn't compete for policy + the mapping tells the policy (=the advice) while the content tells + how to implement (e.g. how much content) + IMO, you could simply get rid of the per object "advice" field and + use default values for now + braunr: What sense these values for number of pages before and + after should have? + or use something well known, easy, and effective like preceding + and following pages + they give the vm the amount of content to ask the backing pager + braunr: maximal amount, minimal amount or exact amount? + neither + that's why i recommend you forget it for now + but + imagine you implement the three standard policies (normal, random, + sequential) + then the pager assigns preceding and following numbers for each of + them, say [5;5], [0;0], [15;15] respectively + these numbers would tell the vm how many pages to ask the pagers + in a single request and from where + braunr: but in fact there could be much more policies. + yes + also in kernel context there is no such unit as pager. + so there should be a call like memory_object_set_advice(int + advice, int preceding, int following); + for example + what ? + the pager is the memory manager + it does exist in kernel context + (or i don't understand what you mean) + there is only port, but port could be either pager or something + else + no, it's a pager + it's a port whose receive right is hold by a task implementing the + pager interface + either the default pager or an untrusted task + (or null if the object is anonymous memory not yet sent to the + default pager) + port is always pager? + the object port is, yes + struct ipc_port *pager; /* Where to get + data */ + So, you suggest to keep set of advices for each object? + i suggest you don't change anything in objects for now + keep the advice in the mappings only, and implement default + behaviour for the known policies + mcsim: if you understand this point, then i have nothing more to + say, and we should let nowhere_man present his work + braunr: ok. I'll implement only default behaviors for know policies + for now. + (actually, using the mapping boundaries is slightly unoptimal, as + we could have several mappings for the same content, e.g. a program with + read only executable mapping, then ro only) + mcsim: another way to know the "size" is to actually lookup for + pages in objects + hm no, that's not true + braunr: But if there is no page we have to ask it + and I don't understand why using mappings boundaries is unoptimal + here is bash + 0000000000400000 868K r-x-- /bin/bash + 00000000006d9000 36K rw--- /bin/bash + two entries, same file + (there is the anonymous memory layer for the second, but it would + matter for the first cow faults) + + +## IRC, freenode, #hurd, 2012-08-02 + + braunr: You said that I probably need some support in vm_pageout.c + to make defpager work with clustered page transfers, but TBH I thought + that I have to implement only pagein. Do you expect from me implementing + pageout either? Or I misunderstand role of vm_pageout.c? + no + you're expected to implement only pagins for now + pageins + well, I'm finishing merging of ext2fs patch for large stores and + work on defpager in parallel. + braunr: Also I didn't get your idea about configuring of paging + mechanism on behalf of pagers. + which one ? + braunr: You said that pager has somehow pass size of desired + clusters for different paging policies. + mcsim: i said not to care about that + and the wording isn't correct, it's not "on behalf of pagers" + servers? + pagers could tell the kernel what size (before and after a faulted + page) they prefer for each existing policy + but that's one way to do it + defaults work well too + as shown in other implementations + + +## IRC, freenode, #hurd, 2012-08-09 + + braunr: I'm still debugging ext2 with large storage patch + mcsim: tough problems ? + braunr: The same issues as I always meet when do debugging, but it + takes time. + mcsim: so nothing blocking so far ? + braunr: I can't tell you for sure that I will finish up to 13th of + August and this is unofficial pencil down date. + all right, but are you blocked ? + braunr: If you mean the issues that I can not even imagine how to + solve than there is no ones. + good + mcsim: i'll try to review your code again this week end + mcsim: make sure to commit everything even if it's messy + braunr: ok + braunr: I made changes to defpager, but I haven't tried + them. Commit them too? + mcsim: sure + mcsim: does it work fine without the large storage patch ? + braunr: looks fine, but TBH I can't even run such things like fsx, + because even without my changes it failed mightily at once. + +[[file_system_exerciser]]. + + mcsim: right, well, that will be part of another task :) + + +## IRC, freenode, #hurd, 2012-08-13 + + braunr: hello. Seems ext2fs with large store patch works. + + +## IRC, freenode, #hurd, 2012-08-19 + + hello. Consider such situation. There is a page fault and kernel + decided to request pager for several pages, but at the moment pager is + able to provide only first pages, the rest ones are not know yet. Is it + possible to supply only one page and regarding rest ones tell the kernel + something like: "Rest pages try again later"? + I tried pager_data_unavailable && pager_flush_some, but this seems + does not work. + Or I have to supply something anyway? + mcsim: better not provide them + the kernel only really needs one page + don't try to implement "try again later", the kernel will do that + if other page faults occur for those pages + braunr: No, translator just hangs + ? + braunr: And I even can't deattach it without reboot + hangs when what + ? + i mean, what happens when it hangs ? + If kernel request 2 pages and I provide one, than when page fault + occurs in second page translator hangs. + well that's a bug + clustered pager transfer is a mere optimization, you shouldn't + transfer more than you can just to satisfy some requested size + I think that it because I create fictitious pages before calling + mo_data_request + as placeholders ? + Yes. Is it correct if I will not grab fictitious pages? + no + i don't know the details well enough about fictitious pages + unfortunately, but it really feels wrong to use them where real physical + pages should be used instead + normally, an in-transfer page is simply marked busy + But If page is already marked busy kernel will not ask it another + time. + when the pager replies, you unbusy them + your bug may be that you incorrectly use pmap + you shouldn't create mmu mappings for pages you didn't receive + from the pagers + I don't create them + ok so you correctly get the second page fault + If pager supplies only first pages, when asked were two, than + second page will not become un-busy. + that's a bug + your code shouldn't assume the pager will provide all the pages it + was asked for + only the main one + Will it be ok if I will provide special attribute that will keep + information that page has been advised? + what for ? + i don't understand "page has been advised" + Advised page is page that is asked in cluster, but there wasn't a + page fault in it. + I need this attribute because if I don't inform kernel about this + page anyhow, than kernel will not change attributes of this page. + why would it change its attributes ? + But if page fault will occur in page that was asked than page will + be already busy by the moment. + and what attribute ? + advised + i'm lost + 08:53 < mcsim> I need this attribute because if I don't inform + kernel about this page anyhow, than kernel will not change attributes of + this page. + you need the advised attribute because if you don't inform the + kernel about this page, the kernel will not change the advised attribute + of this page ? + Not only advised, but busy as well. + And if page fault will occur in this page, kernel will not ask it + second time. Kernel will just block. + well that's normal + But if kernel will block and pager is not going to report somehow + about this page, than translator will hang. + but the pager is going to report + and in this report, there can be less pages then requested + braunr: You told not to report + the kernel can deduce it didn't receive all the pages, and mark + them unbusy anyway + i told not to transfer more than requested + but not sending data can be a form of communication + i mean, sending a message in which data is missing + it simply means its not there, but this info is sufficient for the + kernel + hmmm... Seems I understood you. Let me try something. + braunr: I informed kernel about missing page as follows: + pager_data_supply (pager, precious, writelock, i, 1, NULL, 0); Am I + right? + i don't know the interface well + what does it mean + ? + are you passing NULL as the data for a missing page ? + yes + i see + you shouldn't need a request for that though, avoiding useless ipc + is a good thing + i is number of page, 1 is quantity + but if you can't find a better way for now, it will do + But this does not work :( + that's a bug + in your code probably + braunr: supplying NULL as data returns MACH_SEND_INVALID_MEMORY + but why would it work ? + mach expects something + you have to change that + It's mig who refuses data. Mach does not even get the call. + hum + That's why I propose to provide new attribute, that will keep + information regarding whether the page was asked as advice or not. + i still don't understand why + why don't you fix mig so you can your null message instead ? + +send + braunr: because usually this is an error + the kernel will decide if it's an erro + r + what kinf of reply do you intend to send the kernel with for these + "advised" pages ? + no reply. But when page fault will occur in busy page and it will + be also advised, kernel will not block, but ask this page another time. + And how kernel will know that this is an error or not? + why ask another time ?! + you really don't want to flood pagers with useless messages + here is how it should be + 1/ the kernel requests pages from the pager + it know the range + 2/ the pager replies what it can, full range, subset of it, even + only one page + 3/ the kernel uses what the pager replied, and unbusies the other + pages + First time page was asked because page fault occurred in + neighborhood. And second time because PF occurred in page. + well it shouldn't + or it should, but then you have a segfault + But kernel does not keep bound of range, that it asked. + if the kernel can't find the main page, the one it needs to make + progress, it's a segfault + And this range could be supplied in several messages. + absolutely not + you defeat the purpose of clustered pageins if you use several + messages + But interface supports it + interface supported single page transfers, doesn't mean it's good + well, you could use several messages + as what we really want is less I/O + Noone keeps bounds of requested range, so it couldn't be checked + that range was split + but it would be so much better to do it all with as few messages + as possible + does the kernel knows the main page ? + know* + Splitting range is not optimal, but it's not an error. + i assume it does + doesn't it ? + no, that's why I want to provide new attribute. + i'm sorry i'm lost again + how does the kernel knows a page fault has been serviced ? + know* + It receives an interrupt + ? + let's not mix terms + oh.. I read as received. Sorry + It get mo_data_supply message. Than it replaces fictitious pages + with real ones. + so you get a message + and you kept track of the range using fictitious pages + use the busy flag instead, and another way to retain the range + I allocate fictitious pages to reserve place. Than if page fault + will occur in this page fictitious page kernel will not send another + mo_data_request call, it will wait until fictitious page unblocks. + i'll have to check the code but it looks unoptimal to me + we really don't want to allocate useless objects when a simple + busy flag would do + busy flag for what? There is no page yet + we're talking about mo_data_supply + actually we're talking about the whole page fault process + We can't mark nothing as busy, that's why kernel allocates + fictitious page and marks it as busy until real page would be supplied. + what do you mean "nothing" ? + VM_PAGE_NULL + uh ? + when are physical pages allocated ? + on request or on reply from the pager ? + i'm reading mo_data_supply, and it looks like the page is already + busy at that time + they are allocated by pager and than supplied in reply + Yes, but these pages are fictitious + show me please + in the master branch, not yours + that page is fictitious? + yes + i'm referring to the way mach currently does things + vm/vm_fault.c:582 + that's memory_object_lock_page + hm wait + my bad + ah that damn object chaining :/ + ok + the original code is stupid enough to use fictitious pages all the + time, you probably have to do the same + hm... Attributes will be useless, pager should tell something about + pages, that it is not going to supply. + yes + that's what null is for + Not null, null is error. + one problem i can think of is making sure the kernel doesn't + interpret missing as error + right + I think better have special value for mo_data_error + probably + + +### IRC, freenode, #hurd, 2012-08-20 + + braunr: I think it's useful to allow supplying the data in several + batches. the kernel should *not* assume that any data missing in the + first batch won't be supplied later. + antrik: it really depends + i personally prefer synchronous approaches + demanding that all data is supplied at once could actually turn + readahead into a performace killer + antrik: Why? The only drawback I see is higher response time for + page fault, but it also leads to reduced overhead. + that's why "it depends" + mcsim: it brings benefit only if enough preloaded pages are + actually used to compensate for the time it took the pager to provide + them + which is the case for many workloads (including sequential access, + which is the common case we want to optimize here) + mcsim: the overhead of an extra RPC is negligible compared to + increased latencies when dealing with slow backing stores (such as disk + or network) + antrik: also many replies lead to fragmentation, while in one reply + all data is gathered in one bunch. If all data is placed consecutively, + than it may be transferred next time faster. + mcsim: what kind of fragmentation ? + I really really don't think it's a good idea for the page to hold + back the first page (which is usually the one actually blocking) while + it's still loading some other pages (which will probably be needed only + in the future anyways, if at all) + err... for the pager to hold back + antrik: then all pagers should be changed to handle asynchronous + data supply + it's a bit late to change that now + there could be two cases of data placement in backing store: 1/ all + asked data is placed consecutively; 2/ it is spread among backing + store. If pager gets data in one message it more like place it + consecutively. So to have data consecutive in each pager, each pager has + to try send data in one message. Having data placed consecutive is + important, since reading of such data is much more faster. + mcsim: you're confusing things .. + or you're not telling them properly + Ok. Let me try one more time + since you're working *only* on pagein, not pageout, how do you + expect spread pages being sent in a single message be better than + multiple messages ? + braunr: I think about future :) + ok + but antrik is right, paging in too much can reduce performance + so the default policy should be adjusted for both the worst case + (one page) and the average/best (some/mane contiguous pages) + through measurement ideally + mcsim: BTW, I still think implementing clustered pageout has + higher priority than implementing madvise()... but if the latter is less + work, it might still make sense to do it first of course :-) + many* + there aren't many users of madvise, true + antrik: Implementing madvise I expect to be very simple. It should + just translate call to vm_advise + well, that part is easy of course :-) so you already implemented + vm_advise itself I take it? + antrik: Yes, that was also quite easy. + great :-) + in that case it would be silly of course to postpone implementing + the madvise() wrapper. in other words: never mind my remark about + priorities :-) + + +## IRC, freenode, #hurd, 2012-09-03 + + I try a test with ext2fs. It works, than I just recompile ext2fs + and it stops working, than I recompile it again several times and each + time the result is unpredictable. + sounds like a concurrency issue + I can run the same test several times and ext2 works until I + recompile it. That's the problem. Could that be concurrency too? + mcsim: without bad luck, yes, unless "several times" is a lot + like several dozens of tries + + +## IRC, freenode, #hurd, 2012-09-04 + + hello. I want to tell that ext2fs translator, that I work on, + replaced for my system old variant that processed only single pages + requests. And it works with partitions bigger than 2 Gb. + Probably I'm not for from the end. + But it's worth to mention that I didn't fix that nasty bug that I + told yesterday about. + braunr: That bug sometimes appears after recompilation of ext2fs + and always disappears after sync or reboot. Now I'm going to finish + defpager and test other translators. + + +## IRC, freenode, #hurd, 2012-09-17 + + braunr: hello. Do you remember that you said that pager has to + inform kernel about appropriate cluster size for readahead? + I don't understand how kernel store this information, because it + does not know about such unit as "pager". + Can you give me an advice about how this could be implemented? + mcsim: it can store it in the object + youpi: It too big overhead + youpi: at least from my pow + *pov + mcsim: we discussed this already + mcsim: there is no "pager" entity in the kernel, which is a defect + from my PoV + mcsim: the best you can do is follow what the kernel already does + that is, store this property per object$ + we don't care much about the overhead for now + my guess is there is already some padding, so the overhead is + likely to be amortized by this + like youpi said + I remember that discussion, but I didn't get than whether there + should be only one or two values for all policies. Or each policy should + have its own values? + braunr: ^ + each policy should have its own values, which means it can be + implemented with a simple static array somewhere + the information in each object is a policy selector, such as an + index in this static array + ok + mcsim: if you want to minimize the overhead, you can make this + selector a char, and place it near another char member, so that you use + space that was previously used as padding by the compiler + mcsim: do you see what i mean ? + yes + good + + +## IRC, freenode, #hurd, 2012-09-17 + + hello. May I add function krealloc to slab.c? + mcsim: what for ? + braunr: It is quite useful for creating dynamic arrays + you don't want dynamic arrays + why? + they're expensive + try other data structures + more expensive than linked lists? + depends + but linked lists aren't the only other alternative + that's why btrees and radix trees (basically trees of arrays) + exist + the best general purpose data structure we have in mach is the red + black tree currently + but always think about what you want to do with it + I want to store there sets of sizes for different memory + policies. I don't expect this array to be big. But for sure I can use + rbtree for it. + why not a static array ? + arrays are perfect for known data sizes + I expect from pager to supply its own sizes. So at the beginning in + this array is only default policy. When pager wants to supply it own + policy kernel lookups table of advice. If this policy is new set of sizes + then kernel creates new entry in table of advice. + that would mean one set of sizes for each object + why don't you make things simple first ? + Object stores only pointer to entry in this table. + but there is no pager object shared by memory objects in the + kernel + I mean struct vm_object + so that's what i'm saying, one set per object + it's useless overhead + i would really suggest using a global set of policies for now + Probably, I don't understand you. Where do you want to store this + static array? + it's a global one + "for now"? It is not a problem to implement a table for local + advice, using either rbtree or dynamic array. + it's useless overhead + and it's not a single integer, you want a whole container per + object + don't do anything fancy unless you know you really want it + i'll link the netbsd code again as a very good example of how to + implement global policies that work more than decently for every file + system in this OS + + http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/uvm/uvm_fault.c?rev=1.194&content-type=text/x-cvsweb-markup&only_with_tag=MAIN + look for uvmadvice + But different translators have different demands. Thus changing of + global policy for one translator would have impact on behavior of another + one. + i understand + this isn't l4, or anything experimental + we want something that works well for us + And this is acceptable? + until you're able to demonstrate we need different policies, i'd + recommend not making things more complicated than they already are and + need to be + why wouldn't it ? + we've been discussing this a long time :/ + because every process runs in isolated environment and the fact + that there is something outside this environment, that has no rights to + do that, does it surprises me. + ? + ok. let me dip in uvm code. Probably my questions disappear + i don't think it will + you're asking about the system design here, not implementation + details + with l4, there are as you'd expect well defined components + handling policies for address space allocation, or paging, or whatever + but this is mach + mach has a big shared global vm server with in kernel policies for + it + so it's ok to implement a global policy for this + and let's be pragmatic, if we don't need complicated stuff, why + would we waste time on this ? + It is not complicated. + retaining a whole container for each object, whereas they're all + going to contain exactly the same stuff for years to come seems overly + complicated for me + I'm not going to create separate container for each object. + i'm not following you then + how can pagers upload their sizes in the kernel ? + I'm going to create a new container only for combination of cluster + sizes that are not present in table of advice. + that's equivalent + you're ruling out the default set, but that's just an optimization + whenever a file system decides to use other sizes, the problem + will arise + Before creating a container I'm going to lookup a table. And only + than create + a table ? + But there will be the same container for a huge bunch of objects + how do you select it ? + if it's a per pager container, remember there is no shared pager + object in the kernel, only ports to external programs + I'll give an example + Suppose there are only two policies. At the beginning we have table + {{random = 4096, sequential = 8096}}. Than pager 1 wants to add new + policy where random cluster size is 8192. He asks kernel to create it and + after this table will be following: {{random = 4096, sequential = 8192}, + {random = 8192, sequential = 8192}}. If pager 2 wants to create the same + policy as pager 1, kernel will lockup table and will not create new + entry. So the table will be the same. + And each object has link to appropriate table entry + i'm not sure how this can work + how can pagers 1 and 2 know the sizes are the same for the same + policy ? + (and actually they shouldn't) + For faster lookup there will be create hash keys for each entry + what's the lookup key ? + They do not know + The kernel knows + then i really don't understand + and how do you select sizes based on the policy ? + and how do you remove unused entries ? + (ok this can be implemented with a simple ref counter) + "and how do you select sizes based on the policy ?" you mean at + page fault? + yes + entry or object keeps pointer to appropriate entry in the table + ok your per object data is a pointer to the table entry and the + policy is the index inside + so you really need a ref counter there + yes + and you need to maintain this table + for me it's uselessly complicated + but this keeps design clear + not for me + i don't see how this is clearer + it's just more powerful + a power we clearly don't need now + and in the following years + in addition, i'm very worried about the potential problems this + can introduce + In fact I don't feel comfortable from the thought that one + translator can impact on behavior of another. + simple example: the table is shared, it needs a lock, other data + structures you may have added in your patch may also need a lock + but our locks are noop for now, so you just can't be sure there is + no deadlock or other issues + and adding smp is a *lot* more important than being able to select + precisely policy sizes that we're very likely not to change a lot + what do you mean by "one translator can impact another" ? + As I understand your idea (I haven't read uvm code yet) that there + is a global table of cluster sizes for different policies. And every + translator can change values in this table. That is what I mean under one + translator will have an impact on another one. + absolutely not + translators *can't* change sizes + the sizes are completely static, assumed to be fit all + -be + it's not optimial but it's very simple and effective in practice + optimal* + and it's not a table of cluster sizes + it's a table of pages before/after the faulted one + this reflects the fact tha in mach, virtual memory (implementation + and policy) is in the kernel + translators must not be able to change that + let's talk about pagers here, not translators + Finally I got you. This is an acceptable tradeoff. + it took some time :) + just to clear something + 20:12 < mcsim> For faster lookup there will be create hash keys + for each entry + i'm not sure i understand you here + To found out if there is such policy (set of sizes) in the table we + can lookup every entry and compare each value. But it is better to create + a hash value for set and thus find equal policies. + first, i'm really not comfortable with hash tables + they really need careful configuration + next, as we don't expect many entries in this table, there is + probably no need for this overhead + remember that one property of tables is locality of reference + you access the first entry, the processor automatically fills a + whole cache line + so if your table fits on just a few, it's probably faster to + compare entries completely than to jump around in memory + But we can sort hash keys, and in this way find policies quickly. + cache misses are way slower than computation + so unless you have massive amounts of data, don't use an optimized + container + (20:38:53) braunr: that's why btrees and radix trees (basically + trees of arrays) exist + and what will be the key? + i'm not saying to use a tree instead of a hash table + i'm saying, unless you have many entries, just use a simple table + and since pagers don't add and remove entries from this table + often, it's on case reallocation is ok + one* + So here dynamic arrays fit the most? + probably + it really depends on the number of entries and the write ratio + keep in mind current processors have 32-bits or (more commonly) + 64-bits cache line sizes + bytes probably? + yes bytes + but i'm not willing to add a realloc like call to our general + purpose kernel allocator + i don't want to make it easy for people to rely on it, and i hope + the lack of it will make them think about other solutions instead :) + and if they really want to, they can just use alloc/free + Under "other solutions" you mean trees? + i mean anything else :) + lists are simple, trees are elegant (but add non negligible + overhead) + i like trees because they truely "gracefully" scale + but they're still O(log n) + a good hash table is O(1), but must be carefully measured and + adjusted + there are many other data structures, many of them you can find in + linux + but in mach we don't need a lot of them + Your favorite data structures are lists and trees. Next, what + should you claim, is that lisp is your favorite language :) + functional programming should eventually rule the world, yes + i wouldn't count lists are my favorite, which are really trees + as* + there is a reason why red black trees back higher level data + structures like vectors or maps in many common libraries ;) + mcsim: hum but just to make it clear, i asked this question about + hashing because i was curious about what you had in mind, i still think + it's best to use static predetermined values for policies + braunr: I understand this. + :) + braunr: Yeah. You should be cautious with me :) + + +## IRC, freenode, #hurd, 2012-09-21 + + mcsim: there is only one cluster size per object -- it depends on + the properties of the backing store, nothing else. + (while the readahead policies depend on the use pattern of the + application, and thus should be selected per mapping) + but I'm still not convinced it's worthwhile to bother with cluster + size at all. do other systems even do that?... + + +## IRC, freenode, #hurd, 2012-09-23 + + mcsim: how long do you think it will take you to polish your gsoc + work ? + (and when before you begin that part actually, because we'll to + review the whole stuff prior to polishing it) + braunr: I think about 2 weeks + But you may already start review it, if you're intended to do it + before I'll rearrange commits. + Gnumach, ext2fs and defpager are ready. I just have to polish the + code. + mcsim: i don't know when i'll be able to do that + so expect a few weeks on my (our) side too + ok + sorry for being slow, that's how hurd development is :) + What should I do with libc patch that adds madvise support? + Post it to bug-hurd? + hm probably the same i did for pthreads, create a topic branch in + glibc.git + there is only one commit + yes + (mine was a one liner :p) + ok + it will probably be a debian patch before going into glibc anyway, + just for making sure it works + But according to term. I expect that my study begins in a week and + I'll have to do some stuff then, so actually probably I'll need a week + more. + don't worry, that's expected + and that's the reason why we're slow + And what should I do with large store patch? + hm good question + what did you do for now ? + include it in your work ? + that's what i saw iirc + Yes. It consists of two parts. + the original part and the modificaionts ? + modifications* + i think youpi would know better about that + First (small) adds notification to libpager interface and second + one adds support for large stores. + i suppose we'll probably merge the large store patch at some point + anyway + Yes both original and modifications + good + I'll split these parts to different commits and I'll try to make + support for large stores independent from other work. + that would be best + if you can make it so that, by ommitting (or including) one patch, + we can add your patches to the debian package, it would be great + (only with regard to the large store change, not other potential + smaller conflicts) + braunr: I also found several bugs in defpager, that I haven't fixed + since winter. + oh + seems nobody hasn't expect them. + i'm very interested in those actually (not too soon because it + concerns my work on pageout, which is postponed after pthreads and + select) + ok. than I'll do it first. + + +## IRC, freenode, #hurd, 2012-09-24 + + mcsim: what is vm_get_advice_info ? + braunr: hello. It should supply some machine specific parameters + regarding clustered reading. At the moment it supplies only maximal + possible size of cluster. + mcsim: why such a need ? + It is used by defpager, as it can't allocate memory dynamically and + every thread has to allocate maximal size beforehand + mcsim: i see + + +## IRC, freenode, #hurd, 2012-10-05 + + braunr: I think it's not worth to separate large store patch for + ext2 and patch for moving it to new libpager interface. Am I right? + mcsim: it's worth separating, but not creating two versions + i'm not sure what you mean here + First, I applied large store patch, and than I was changing patched + code, to make it work with new libpager interface. So changes to make + ext2 work with new interface depend on large store patch. + braunr: ^ + mcsim: you're not forced to make each version resulting from a new + commit work + but don't make big commits + so if changing an interface requires its users to be updated + twice, it doesn't make sense to do that + just update the interface cleanly, you'll have one or more commits + that produce intermediate version that don't build, that's ok + then in another, separate commit, adjust the users + braunr: The only user now is ext2. And the problem with ext2 is + that I updated not the version from git repository, but the version, that + I've got after applying the large store patch. So in other words my + question is follows: should I make a commit that moves to new interface + version of ext2fs without large store patch? + you're asking if you can include the large store patch in your + work, and by extension, in the main branch + i would say yes, but this must be discussed with others + + +## IRC, freenode, #hurd, 2013-02-18 + + mcsim: so, currently reviewing gnumach + braunr: hello + mcsim: the review branch, right ? + braunr: yes + braunr: What do you start with? + memory refreshing + i see you added the advice twice, to vm_object and vm_map_entry + iirc, we agreed to only add it to map entries + am i wrong ? + let me see + the real question being: what do you use the object advice for ? + >iirc, we agreed to only add it to map entries + braunr: TBH, do not remember that. At some point we came to + conclusion that there should be only one advice. But I'm not sure if it + was final point. + maybe it wasn't, yes + that's why i've just reformulated the question + if (map_entry && (map_entry->advice != VM_ADVICE_DEFAULT)) + advice = map_entry->advice; + else + advice = object->advice; + ok + It just participates in determining actual advice + ok that's not a bad thing + let's keep it + please document VM_ADVICE_KEEP + and rephrase "How to handle page faults" in vm_object.h to + something like 'How to tune page fault handling" + mcsim: what's the point of VM_ADVICE_KEEP btw ? + braunr: Probably it is better to remove it? + well if it doesn't do anything, probably + braunr: advising was part of mo_set_attributes before + no it is redudant + i see + so yes, remove it + s/no/now + (don't waste time on a gcs-like changelog format for now) + i also suggest creating _vX branches + so we can compare the changes between each of your review branches + hm, minor coding style issues like switch(...) instead of switch + (...) + why does syscall_vm_advise return MACH_SEND_INTERRUPTED if the + target map is NULL ? + is it modelled after an existing behaviour ? + ah, it's the syscall version + braunr: every syscall does so + and the error is supposed to be used by user stubs to switch to + the rpc version + ok + hm + you've replaced obsolete port_set_select and port_set_backup calls + with your own + don't do that + instead, add your calls to the new gnumach interface + mcsim: out of curiosity, have you actually tried the syscall + version ? + braunr: Isn't it called by default? + i don't think so, no + than no + ok + you could name vm_get_advice_info vm_advice_info + regarding obsolete calls, did you say that only in regard of + port_set_* or all other calls too? + all of the + m + i missed one, yes + the idea is: don't change the existing interface + >you could name vm_get_advice_info vm_advice_info + could or should? i.e. rename? + i'd say should, to remain consistent with the existing similar + calls + ok + can you explain KERN_NO_DATA a bit more ? + i suppose it's what servers should answer for neighbour pages that + don't exist in the backend, right ? + kernel can ask server for some data to read them beforehand, but + server can be in situation when it does not know what data should be + prefetched + yes + ok + it is used by ext2 server + with large store patch + so its purpose is to allow the kernel to free the preallocated + pages that won't be used + do i get it right ? + no. + ext2 server has a buffer for pages and when kernel asks to read + pages ahead it specifies region of that buffer + ah ok + but consecutive pages in buffer does not correspond to consecutive + pages on disk + so, the kernel can only prefetch pages that were already read by + the server ? + no, it can ask a server to prefetch pages that were not read by + server + hum + ok + but in case with buffer, if buffer page is empty, server does not + know what to prefetch + i'm not sure i'm following + well, i'm sure i'm not following + what happens when the kernel requests data from a server, right + after a page fault ? + what does the message afk for ? + kernel is unaware regarding actual size of file where was page + fault because of buffer indirection, right? + i don't know what "buffer" refers to here + this is buffer in memory where ext2 server reads pages + with large store patch ext2 server does not map the whole disk, but + some of its pages + and it maps these pages in special buffer + that means that constructiveness of pages in memory does not mean + that they are consecutive on disk or logically (belong to the same file) + ok so it's a page pool + with unordered pages + but what do you mean when you say "server does not know what to + prefetch" + it normally has everything to determine that + For instance, page fault occurs that leads to reading of + 4k-file. But kernel does not know actual size of file and asks to + prefetch 16K bytes + yes + There is no sense to prefetch something that does not belong to + this file + yes but the server *knows* that + and server answers with KERN_NO_DATA + server should always say something about every page that was asked + then, again, isn't the purpose of KERN_NO_DATA to notify the + kernel it can release the preallocated pages meant for the non existing + data ? + (non existing or more generally non prefetchable) + yes + then + why did you answer no to + 15:46 < braunr> so its purpose is to allow the kernel to free the + preallocated pages that won't be used + is there something missing ? + (well obviously, notify the kernel it can go on with page fault + handling) + braunr: sorry, misunderstoo/misread + ok + so good, i got this right :) + i wonder if KERN_NO_DATA may be a bit too vague + people might confuse it with ENODATA + Actually, this is transformation of ENODATA + I was looking among POSIX error codes and thought that this is the + most appropriate + i'm not sure it is + first, it's about STREAMS, a commonly unused feature + and second, the code is obsolete + braunr: AFAIR purpose of KERN_NO_DATA is not only free + pages. Without this call something should hang + 15:59 < braunr> (well obviously, notify the kernel it can go on + with page fault handling) + yes + hm + sorry again + i don't see anything better for the error name for now + and it's really minor so let's keep it as it is + actually, ENODATA being obsolete helps here + ok, done for now, work calling + we'll continue later or tomorrow + braunr: ok + other than that, this looks ok on the kernel side for now + the next change is a bit larger so i'd like to take the time to + read it + braunr: ok + regarding moving calls in mach.defs, can I put them elsewhere? + gnumach.defs + you'll probably need to rebase your changes to get it + braunr: I'll rebase this later, when we finish with review + ok + keep the comments in a list then, not to forget + (logging irc is also useful) + + +## IRC, freenode, #hurd, 2013-02-20 + + mcsim: why does VM_ADVICE_DEFAULT have its own entry ? + braunr: this kind of fallback mode + i suppose that even random strategy could even read several pages + at once + yes + but then, why did you name it "default" ? + because it is assigned by default + ah + so you expect pagers to set something else + for all objects they create + yes + ok + why not, but add a comment please + at least until all pagers will support clustered reading + ok + even after that, it's ok + just say it's there to keep the previous behaviour by default + so people don't get the idea of changing it too easily + comment in vm_advice.h? + no, in vm_fault.C + right above the array + why does vm_calculate_clusters return two ranges ? + also, "Function PAGE_IS_NOT_ELIGIBLE is used to determine if", + PAGE_IS_NOT_ELIGIBLE doesn't look like a function + I thought make it possible not only prefetch range, but also free + some memory that is not used already + braunr: ^ + but didn't implement it :/ + don't overengineer it + reduce to what's needed + braunr: ok + braunr: do you think it's worth to implement? + no + braunr: it could be useful for sequential policy + describe what you have in mind a bit more please, i think i don't + have the complete picture + with sequential policy user supposed to read strictly in sequential + order, so pages that user is not supposed to read could be put in unused + list + what pages the user isn't supposed to read ? + if user read pages in increasing order than it is not supposed to + read pages that are right before the page where page fault occured + right ? + do you mean higher ? + that are before + before would be lower then + oh + "right before" + yes :) + why not ? + the initial assumption, that MADV_SEQUENTIAL expects *strict* + sequential access, looks wrong + remember it's just a hint + a user could just acces pages that are closer to one another and + still use MADV_SEQUENTIAL, expecting a speedup because pages are close + well ok, this wouldn't be wise + MADV_SEQUENTIAL should be optimized for true sequential access, + agreed + but i'm not sure i'm following you + but I'm not going to page these pages out. Just put in unused + list, and if they will be used later they will be move to active list + your optimization seem to be about freeing pages that were + prefetched and not actually accessed + what's the unused list ? + inactive list + ok + so that they're freed sooner + yes + well, i guess all neighbour pages should first be put in the + inactive list + iirc, pages in the inactive list aren't mapped + this would force another page fault, with a quick resolution, to + tell the vm system the page was actually used, and must become active, + and paged out later than other inactive pages + but i really think it's not worth doing it now + clustered pagins is about improving I/O + page faults without I/O are orders of magnitude faster than I/O + it wouldn't bring much right now + ok, I remove this, but put in TODO + I'm not sure that right list is inactive list, but the list that is + scanned to pageout pages to swap partition. There should be such list + both the active and inactive are + the active one is scanned when the inactive isn't large enough + (the current ratio of active pages is limited to 1/3) + (btw, we could try increasing it to 1/2) + iirc, linux uses 1/2 + your comment about unlock_request isn't obvious, i'll have to + reread again + i mean, the problem isn't obvious + ew, functions with so many indentation levels :/ + i forgot how ugly some parts of the mach vm were + mcsim: basically it's ok, i'll wait for the simplified version for + another pass + simplified? + 22:11 < braunr> reduce to what's needed + ok + and what comment? + your XXX in vm_fault.c + when calling vm_calculate_clusters + is m->unlock_request the same for all cluster or I should + recalculate it for every page? + s/all/whole + that's what i say, i'll have to come back to that later + after i have reviewed the userspace code i think + so i understand the interactions better + braunr: pushed v1 branch + braunr: "Move new calls to gnumach.defs file" and "Implement + putting pages in inactive list with sequential policy" are in my TODO + mcsim: ok + + +## IRC, freenode, #hurd, 2013-02-24 + + mcsim: where does the commit from neal (reworking libpager) come + from ? + (ok the question looks a little weird semantically but i think you + get my point) + braunr: you want me to give you a link to mail with this commit? + why not, yes + http://permalink.gmane.org/gmane.os.hurd.bugs/446 + ok so + http://lists.gnu.org/archive/html/bug-hurd/2012-06/msg00001.html + ok so, we actually have three things to review here + that libpager patch, the ext2fs large store one, and your work + mcsim: i suppose something in your work depends on neal's patch, + right ? + i mean, why did you work on top of it ? + Yes + All user level code + i see it adds some notifications + no + notifacations are for large store + ok + but the rest is for my work + but what does it do that you require ? + braunr: this patch adds support for multipage work. There were just + stubs that returned errors for chunks longer than one page before. + ok + for now, i'll just consider that it's ok, as well as the large + store patch + ok i've skipped all patches up to "Make mach-defpager process + multipage requests in m_o_data_request." since they're obvious + but this one isn't + mcsim: why is the offset member a vm_size_t in struct block ? + (these things matter for large file support on 32-bit systems) + braunr: It should be vm_offset_t, right? + yes + well + it seems so but + im not sure what offset is here + vm_offset is normally the offset inside a vm_object + and if we want large file support, it could become a 64-bit + integer + while vm_size_t is a size inside an address space, so it's either + 32 or 64-bit, depending on the address space size + but here, if offset is an offset inside an address space, + vm_size_t is fine + same question for send_range_parameters + braunr: TBH, I do not differ vm_size_t and vm_offset_t well + they can be easily confused yes + they're both offsets and sizes actually + they're integers + so here I used vm_offset_t because field name is offset + but vm_size_t is an offset/size inside an address space (a + vm_map), while vm_offset_t is an offset/size inside an object + braunr: I didn't know that + it's not clear at all + and it may not have been that clear in mach either + but i think it's best to consider them this way from now on + well, it's not that important anyway since we don't have large + file support, but we should some day :/ + i'm afraid we'll have it as a side effect of the 64-bit port + mcsim: just name them vm_offset_t when they're offsets for + consistency + but seems that I guessed, because I use vm_offset_t variables in + mo_ functions + well ok, but my question was about struct block + where you use vm_size_t + braunr: I consider this like a mistake + ok + moving on + in upload_range, there are two XXX comments + i'm not sure to understand + Second XXX I put because at the moment when I wrote this not all + hurd libraries and servers supported size different from vm_page_size + But then I fixed this and replaced vm_page_size with size in + page_read_file_direct + ok then update the comment accordingly + When I was adding third XXX, I tried to check everything. But I + still had felling that I forgot something. + No it is better to remove second and third XXX, since I didn't find + what I missed + well, that's what i mean by "update" :) + ok + and first XXX just an optimisation. Its idea is that there is no + case when the whole structure is used in one function. + ok + But I was not sure if was worth to do, because if there will appear + some bug in future it could be hard to find it. + I mean that maintainability decreases because of using union + So, I'd rather keep it like it is + how is struct send_range_parameters used ? + it doesn't looked to be something stored long + also, you're allowed to use GNU extensions + It is used to pass parameters from one function to another + which of them? + see + http://gcc.gnu.org/onlinedocs/gcc-4.4.7/gcc/Unnamed-Fields.html#Unnamed-Fields + mcsim: if it's used to pass parameters, it's likely always on the + stack + braunr: I use it when necessary + we really don't care much about a few extra words on the stack + the difference in size would + agree + matter + oops + the difference in size would matter if a lot of those were stored + in memory for long durations + that's not the case, so the size isn't a problem, and you should + remove the comment + ok + mcsim: if i get it right, the libpager rework patch changes some + parameters from byte offset to page frame numbers + braunr: yes + why don't you check errors in send_range ? + braunr: it was absent in original code, but you're right, I should + do this + i'm not sure how to handle any error there, but at least an assert + I found a place where pager just panics + for now it's ok + your work isn't about avoiding panics, but there must be a check, + so if we can debug it and reach that point, we'll know what went wrong + i don't understand the prototype change of default_read :/ + it looks like it doesn't return anything any more + has it become asynchronous ? + It was returning some status before, but now it handles this status + on its own + hum + how ? + how do you deal with errors ? + in old code default_read returned kr and this kr was used to + determine what m_o_ function will be used + now default_read calls m_o_ on its own + ok + + +## IRC, freenode, #hurd, 2013-03-06 + + braunr: hi, regarding memory policies. Should I create separate + policy that will do pageout or VM_ADVICE_SEQUENTIAL is good enough? + braunr: at the moment it is exactly like NORMAL + mcsim: i thought you only did pageins + braunr: yes, but I'm doing pageouts now + oh + i'd prefer you didn't :/ + if you want to improve paging, i have a suggestion i believe is a + lot better + and we have 3 patches concerning libpager that we need to review, + polish, and merge in + braunr: That's not hard, and I think I know what to do + yes i understand that + but it may change the interface and conflict with the pending + changes + braunr: What changes? + the large store patch, neal's libpager rework patch on top of + which you made your changes, and your changes + the idea i have in mind was writeback throttling + +[[hurd/translator/ext2fs]], [[hurd/libpager]]. + + i was planning on doing it myself but if you want to work on it, + feel free to + it would be a much better improvement at this time than clustered + pageouts + (which can then immediately follow + ) + braunr: ok + braunr: but this looks much more bigger task for me + we'll talk about the strategy i had in mind tomorrow + i hope you find it simple enough + on the other hand, clustered pageouts are very similar to pageins + and we have enough paging related changes to review that adding + another wouldn't be such a problem actually + so, add? + if that's what you want to do, ok + i'll think about your initial question tomorrow + + +## IRC, freenode, #hurd, 2013-09-30 + + talking about which... did the clustered I/O work ever get + concluded? + antrik: yes, mcsim was able to finish clustered pageins, and it's + still on my TODO list + it will get merged eventually, now that the large store patch has + also been applied + + +## IRC, freenode, #hurd, 2013-12-31 + + mcsim: do you think you'll have time during january to work out + your clustered pagein work again ? :) + braunr: hello. yes, I think. Depends how much time :) + shouldn't be much i guess + what exactly should be done there? + probably a rebase, and once the review and tests have been + completed, writing the full changelogs + ok + the libpager notification on eviction patch has been pushed in as + part of the merge of the ext2fs large store patch + i have to review neal's rework patch again, and merge it + and then i'll test your work and make debian packages for + darnassus + play with it a bit, see how itgoes + mcsim: i guess you could start with + 62004794b01e9e712af4943e02d889157ea9163f (Fix bugs and warnings in + mach-defpager) + rebase it, send it as a patch on bug-hurd, it should be + straightforward and short + + +## IRC, freenode, #hurd, 2014-03-04 + + btw, has mcsim worked on vectorized i/o ? there was someting you + wanted to integrate + not sure what + clustered pageins + but he seems busy + oh, pageins diff --git a/open_issues/performance/ipc_virtual_copy.mdwn b/open_issues/performance/ipc_virtual_copy.mdwn new file mode 100644 index 00000000..9708ab96 --- /dev/null +++ b/open_issues/performance/ipc_virtual_copy.mdwn @@ -0,0 +1,395 @@ +[[!meta copyright="Copyright © 2011 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +IRC, freenode, #hurd, 2011-09-02: + + what's the usual throughput for I/O operations (like "dd + if=/dev/zero of=/dev/null") in one of those Xen based Hurd machines + (*bber)? + good question + slpz: but don't use /dev/zero and /dev/null, as they don't have + anything to do with true I/O operations + braunr: in fact, I want to test the performance of IPC's virtual + copy operations + ok + braunr: sorry, the "I/O" was misleading + use bs=4096 then i guess + bs > 2k + ? + braunr: everything about 2k is copied by vm_map_copyin/copyout + s/about/above/ + braunr: MiG's stubs check for that value and generate complex (with + out_of_line memory) messages if datalen is above 2k, IIRC + ok + slpz: found it, thanks + tschwinge@strauss:~ $ dd if=/dev/zero of=/dev/null bs=4k & p=$! + && sleep 10 && kill -s INFO $p && sleep 1 && kill $p + [1] 13469 + 17091+0 records in + 17090+0 records out + 70000640 bytes (70 MB) copied, 17.1436 s, 4.1 MB/s + Note, however 10 s vs. 17 s! + And this is slow compared to heal hardware: + thomas@coulomb:~ $ dd if=/dev/zero of=/dev/null bs=4k & p=$! && + sleep 10 && kill -s INFO $p && sleep 1 && kill $p + [1] 28290 + 93611+0 records in + 93610+0 records out + 383426560 bytes (383 MB) copied, 9.99 s, 38.4 MB/s + tschwinge: is the first result on xen vm ? + I think so. + :/ + tschwinge: Thanks! Could you please try with a higher block size, + something like 128k or 256k? + strauss is on a machine that also hosts a buildd, I think. + oh ok + yes, aside either rossini or mozart + And I can confirm that with dd if=/dev/zero of=/dev/null bs=4k + running, a parallel sleep 10 takes about 20 s (on strauss). + +[[open_issues/time]] + + slpz: i'll set up xen hosts soon and can try those tests while + nothing else runs to have more accurate results + tschwinge@strauss:~ $ dd if=/dev/zero of=/dev/null bs=256k & + p=$! && sleep 10 && kill -s INFO $p && sleep 1 && kill $p + [1] 13482 + 4566+0 records in + 4565+0 records out + 1196687360 bytes (1.2 GB) copied, 13.6751 s, 87.5 MB/s + slpz: gains are logarithmic beyond the page size + thomas@coulomb:~ $ dd if=/dev/zero of=/dev/null bs=256k & p=$! + && sleep 10 && kill -s INFO $p && sleep 1 && kill $p + [1] 28295 + 6335+0 records in + 6334+0 records out + 1660420096 bytes (1.7 GB) copied, 9.99 s, 166 MB/s + This time a the sleep 10 decided to take 13.6 s. + ``Interesting.'' + tschwinge: Thanks again. The results for the Xen machine are not bad + though. I can't obtain a throughput over 50MB/s with KVM. + slpz: Want more data (bs)? Just tell. + slpz: i easily get more than that + slpz: what buffer size do you use ? + tschwinge: no, I just wanted to see if Xen has an upper limit beyond + KVM's. Thank you. + braunr: I try with different sizes until I find the maximum + throughput for a certain amount of requests (count) + braunr: are you working with KVM? + yes + slpz: my processor is a model name : Intel(R) Core(TM)2 Duo + CPU E7500 @ 2.93GHz + Linux silvermoon 2.6.32-5-amd64 #1 SMP Tue Jun 14 09:42:28 UTC + 2011 x86_64 GNU/Linux + (standard amd64 squeeze kernel) + braunr: and KVM's version? + squeeze (0.12.5) + bbl + 212467712 bytes (212 MB) copied, 9.95 s, 21.4 MB/s on kvm for me! + gnu_srs: which block size? + 4k, and 61.7 MB/s with 256k + gnu_srs: could you try with 512k and 1M? + 512k: 56.0 MB/s, 1024k: 40.2 MB/s Looks like the peak is around a + few 100k + gnu_srs: thanks! + I've just obtained 1.3GB/s with bs=512k on other (newer) machine + on which hw/vm ? + I knew this is a cpu-bound test, but I couldn't imagine faster + processors could make this difference + braunr: Intel(R) Core(TM) i5 CPU 650 @ 3.20GHz + braunr: KVM + ok + how much time did you wait before reading the result ? + that was 20x times better than the same test on my Intel(R) + Core(TM)2 Duo CPU T7500 @ 2.20GHz + braunr: I've repeated the test with a fixed "count" + My box is: Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz: Max + is 67 MB/s around 140k block size + yes but how much time did dd run ? + 10 s plus/minus a few fractions of a second, + try waiting 30s + braunr: didn't check, let me try again + my kvm peaks at 130 MiB/s with bs 512k / 1M + 2029690880 bytes (2.0 GB) copied, 30.02 s, 67.6 MB/s, bs=140k + gnu_srs: i'm very surprised with slpz's result of 1.3 GiB/s + braunr: over 60 s running, same performance + nice + i wonder what makes it so fast + how much cache ? + Me too, I cannot get better values than around 67 MB/s + gnu_srs: same questions + braunr: 4096KB, same as my laptop + slpz: l2 ? l3 ? + kvm: cache=writeback, CPU: 4096 KB + gnu_srs: this has nothing to do with the qemu option, it's about + the cpu + braunr: no idea, it's the first time I touch this machine. I going + to see if I find the model in processorfinder + under my host linux system, i get a similar plot, that is, + performance drops beyond bs=1M + braunr: OK, bu I gave you the cache size too, same as slpz. + i wonder what dd actually does + read() and writes i guess + braunr: read/write repeatedly, nothing fancy + slpz: i don't think it's a good test for virtual copy + io_read_request, vm_deallocate, io_write_request, right + slpz: i really wonder what it is about i5 that improves speed so + much + braunr: me too + braunr: L2: 2x256KB, L3: 4MB + and something calling "SmartCache" + slpz: where did you find these values? + gnu_srs: ark.intel.com and wikipedia + aha, cpuinfo just gives cache size. + that "SmartCache" thing seems to be just L2 cache sharing between + cores. Shouldn't make a different since we're using only one core, and I + don't see KVM hooping between them. + with bs=256k: 7004487680 bytes (7.0 GB) copied, 10 s, 700 MB/s + (qemu/kvm, 3 * Intel(R) Xeon(R) E5504 2GHz, cache size 4096 KB) + manuel: did you try with 512k/1M? + bs=512k: 7730626560 bytes (7.7 GB) copied, 10 s, 773 MB/s + bs=1M: 7896825856 bytes (7.9 GB) copied, 10 s, 790 MB/s + manuel: those are pretty good numbers too + xeon processor + lshw gave me: L1 Cache 256KiB, L2 cache 4MiB + sincerely, I've never seen Hurd running this fast. Just checked + "uname -a" to make sure I didn't take the wrong image :-) + for bs=256k, 60s: 40582250496 bytes (41 GB) copied, 60 s, 676 MB/s + slpz: i think you can assume processor differences alter raw + copies too much to get any valuable results about virtual copy operations + you need a specialized test program + and bs=512k, 60s, 753 MB/s + braunr: I'm using the mach_perf suite from OSFMach to do the + "serious" testing. I just wanted a non-synthetic test to confirm the + readings. + +[[!taglink open_issue_gnumach]] -- have a look at *mach_perf*. + + manuel: how much cache ? 2M ? + slpz: ok + manuel: hmno, more i guess + braunr: /proc/cpuinfo says cache size : 4096 KB + ok + manuel: performance should drop beyond bs=2M + but that's not relevant anyway + Linux: bs=1M, 10.8 GB/s + I think this difference is too big to be only due to a bigger amount + of CPU cycles... + slpz: clearly + gnu_srs: your host system has 64 or 32 bits? + braunr: I'm going to investigate a bit + but this accidental discovery just made my day. We're able to run + Hurd at decent speeds on newer hardware! + slpz: what result do you get with the same test on your host + system ? + interestingly, running it several times has made the performance + drop quite much (i'm getting 400-500MB/s with 1M now, compared to nearly + 800 fifteen minutes ago) + +[[Degradataion]]. + + braunr: probably an almost infinite throughput, but I don't consider + that a valid test, since in Linux, the write operation to "/dev/null" + doesn't involve memory copying/moving + manuel: i observed the same behaviour + slpz: Host system is 64 bit + slpz: it doesn't on the hurd either + slpz: (under 2k, that is) + over* + braunr: humm, you're right, as the null translator doesn't "touch" + the memory, CoW rules apply + slpz: the only thing which actually copies things around is dd + probably by simply calling read() + which gets its result from a VM copy operation, but copies the + content to the caller provided buffer + then vm_deallocate() the data from the storeio (zero) translator + if storeio isn't too dumb, it doesn't even touch the transfered + buffer (as anonymous vm_map()ped memory is already cleared) + +[[!taglink open_issue_documentation]] + + so this is a good test for measuring (profiling?) our ipc overhead + and possibly the vm mapping operations (which could partly explain + why the results get worse over time) + manuel: can you run vminfo | wc -l on your gnumach process ? + braunr: Yes, unless some special situation apply, like the source + address/offset being unaligned, or if the translator decides to return + the result in a different buffer (which I assume is not the case for + storeio/zero) + braunr: 35 + slpz: they can't be unaligned, the vm code asserts that + manuel: ok, this is normal + braunr: address/offset from read() + slpz: the caller provided buffer you mean ? + braunr: yes, and the offset of the memory_object, if it's a pager + based translator + slpz: highly unlikely, the compiler chooses appropriate alignments + for such buffers + braunr: in those cases, memcpy is used over vm_copy + slpz: and the glibc memcpy() optimized versions can usually deal + with that + slpz: i don't get your point about memory objects + slpz: requests on memory objects always have aligned values too + braunr: sure, but can't deal with the user requesting non + page-aligned sizes + slpz: we're considering our dd tests, for which we made sure sizes + were page aligned + braunr: oh, I was talking in a general sense, not just in this dd + tests, sorry + by the way, dd on the host tops at 12 GB/s with bs=2M + that's consistent with our other results + slpz: you mean, even on your i5 processor with 1.3 GiB/s on your + hurd kvm ? + braunr: yes, on the GNU/Linux which is running as host + slpz: well that's not consistent + braunr: consistent with what? + slpz: i get roughly the same result on my host, but ten times less + on my hurd kvm + slpz: what's your kernel/kvm versions ? + 2.6.32-5-amd64 (debian's build) 0.12.5 + same here + i'm a bit clueless + why do i only get 130 MiB/s where you get 1.3 .. ? :) + well, on my laptop, where Hurd on KVM tops on 50 MB/s, Linux gets a + bit more than 10 GB/s + see + slpz: reduce bs to 256k and test again if you have time please + braunr: on which system? + slpz: the fast one + (linux host) + braunr: Hurd? + ok + 12 GB/s + i get 13.3 + same for 128k, only at 64k starts dropping + maybe, on linux we're being limited by memory speed, while on Hurd's + this test is (much) more CPU-bound? + slpz: maybe + too bad processor stalls aren't easy to measure + braunr: that's very true. It's funny when you read a paper which + measures performance by cycles on an old RISC processor. That's almost + impossible to do (with reliability) nowadays :-/ + I wonder which throughput can achieve Hurd running bare-metal on + this machine... + both the Xeon and the i5 use cores based on the Nehalem + architecture + apparently Nehalem is where Intel first introduces nested page + tables + which pretty much explains the considerably lower overhead of VM + magic + antrik, what are nested page tables? (sounds like the 4-level page + tables we already have on amd64, or 2-level or 3-level on x86 pae) + page tables were always 2-level on x86 + that's unrelated + nested page tables means there is another layer of address + translation, so the VMM can do it's own translation and doesn't care what + the guest system does => no longer has to intercept all page table + manipulations + antrik: do you imply it only applies to virtualized systems ? + braunr: yes + antrik: Good guess. Looks like Intel's EPT are doing the trick by + allowing the guest OS deal with its own page faults + antrik: next monday, I'll try disabling EPT support in KVM on that + machine (the fast one). That should confirm your theory empirically. + this also means that there're too many page faults, as we should be + doing virtual copies of memory that is not being accessed + and looking at how the value of "page faults" in "vmstat" increases, + shows that page faults are directly proportional to the number of pages + we are asking from the translator + I've also tried doing a long read() directly, to be sure that "dd" + is not doing something weird, and it shows the same behaviour. + slpz: dd does copy buffers + slpz: i told you, it's not a good test case for pure virtual copy + evaluation + antrik: do you know if xen benefits from nested page tables ? + no idea + +[[!taglink open_issue_xen]] + + braunr: but my small program doesn't, and still provokes a lot of + page faults + slpz: are you certain it doesn't ? + braunr: looking at google, it looks like recent Xen > 3.4 supports + EPT + ok + i'm ordering my new server right now, core i5 :) + braunr: at least not explicitily. I need to look at MiG stubs again, + I don't remember if they do something weird. + braunr: sandybridge or nehalem? :-) + antrik: no idea + does it tell a model number? + not yet + but i don't have a choice for that, so i'll order it first, check + after + hehe + I'm not sure it makes all that much difference anyways for a + server... unless you are running it at 100% load ;-) + antrik: i'm planning on running xen guests suchs as new buildd + hm... note though that some of the nehalem-generation i5s were + dual-core, while all the new ones are quad + it's a quad + the newer generation has better performance per GHz and per + Watt... but considering that we are rather I/O-limited in most cases, it + probably won't make much difference + not sure whether there are further virtualisation improvements + that could be relevant... + buildds spend much time running gcc, so even such improvements + should help + there, server ordered :) + antrik: model name : Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz + +IRC, freenode, #hurd, 2011-09-06: + + youpi: what machines are being used for buildd? Do you know if they + have EPT/RVI? + we use PV Xen there + I think Xen could also take advantage of those technologies. Not + sure if only in HVM or with PV too. + only in HVM + in PV it does not make sense: the guest already provides the + translated page table + which is just faster than anything else + +IRC, freenode, #hurd, 2011-09-09: + + oh BTW, for another data point: dd zero->null gets around 225 MB/s + on my lowly 1 GHz Pentium3, with a blocksize of 32k + (but only half of that with 256k blocksize, and even less with 1M) + the system has been up for a while... don't know whether it's + faster on a freshly booted one + +IRC, freenode, #hurd, 2011-09-15: + + + http://www.reddit.com/r/gnu/comments/k68mb/how_intelamd_inadvertently_fixed_gnu_hurd/ + so is the dd command pointed to by that article a measure of io + performance? + sudoman: no, not really + it's basically the baseline of what is possible -- but the actual + slowness we experience is more due to very unoptimal disk access patterns + though using KVM with writeback caching does actually help with + that... + also note that the title of this post really makes no + sense... nested page tables should provide similar improvements for *any* + guest system doing VM manipulation -- it's not Hurd-specific at all + ok, that makes sense. thanks :) + +IRC, freenode, #hurd, 2011-09-16: + + antrik: I wrote that article (the one about How AMD/Intel fixed...) + antrik: It's obviously a bit of an exaggeration, but it's true that + nested pages supposes a great improvement in the performance of Hurd + running on virtual machines + antrik: and it's Hurd specific, as this system is more affected by + the cost of page faults + antrik: and as the impact of virtualization on the performance is + much higher than (almost) any other OS. + antrik: also, dd from /dev/zero to /dev/null it's a measure on how + fast OOL IPC is. diff --git a/open_issues/performance/microbenchmarks.mdwn b/open_issues/performance/microbenchmarks.mdwn new file mode 100644 index 00000000..de3a54b7 --- /dev/null +++ b/open_issues/performance/microbenchmarks.mdwn @@ -0,0 +1,13 @@ +[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +Microbenchmarks may give useful hints, or they may not. + + diff --git a/open_issues/performance/microkernel_multi-server.mdwn b/open_issues/performance/microkernel_multi-server.mdwn new file mode 100644 index 00000000..0382c835 --- /dev/null +++ b/open_issues/performance/microkernel_multi-server.mdwn @@ -0,0 +1,226 @@ +[[!meta copyright="Copyright © 2011, 2013 Free Software Foundation, Inc."]] + +[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable +id="license" text="Permission is granted to copy, distribute and/or modify this +document under the terms of the GNU Free Documentation License, Version 1.2 or +any later version published by the Free Software Foundation; with no Invariant +Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license +is included in the section entitled [[GNU Free Documentation +License|/fdl]]."]]"""]] + +[[!tag open_issue_documentation]] + +Performance issues due to the microkernel/multi-server system architecture? + + +# IRC, freenode, #hurd, 2011-07-26 + + < CTKArcher> I read that, because of its microkernel+servers design, the + hurd was slower than a monolithic kernel, is that confirmed ? + < youpi> the hurd is currently slower than current monolithic kernels, but + it's not due to the microkernel + servers design + < youpi> the microkernel+servers design makes the system call path longer + < youpi> but you're bound by disk and network speed + < youpi> so the extra overhead will not hurt so much + < youpi> except dumb applications keeping doing system calls all the time + of course, but they are usually considered bogus + < braunr> there may be some patterns (like applications using pipes + extensively, e.g. git-svn) which may suffer from the design, but still in + an acceptable range + < CTKArcher> so, you are saying that disk and network are more slowing the + system than the longer system call path and because of that, it wont + really matter ? + < youpi> braunr: they should sitll be fixed because they'll suffer (even if + less) on monolithic kernels + < youpi> CTKArcher: yes + < braunr> yes + < CTKArcher> mmh + < youpi> CTKArcher: you might want to listen to AST's talk at fosdem 10 + iirc, about minix + < youpi> they even go as far as using an IPC for each low-level in/out + < youpi> for security + < braunr> this has been expected for a long time + < braunr> which is what motivated research in microkernels + < CTKArcher> I've already downloaded the video :) + < youpi> and it has been more and more true with faster and faster cpus + < braunr> but in 95, processors weren't that fast compared to other + components as they are now + < youpi> while disk/mem haven't evovled so fast + + +# IRC, freenode, #hurd, 2013-09-30 + + ok.. i noticed when installing debian packages in X, the mouse + lagged a little bit + that takes me back to classic linux days + it could be a side effect of running under virtualisation who + knows + no + it's because of the difference of priorities between server and + client tasks + is it simple enough to increase the priority of the X server? + it does remind me of the early linux days.. people were more + interested in making things work, and making things not crash.. than + improving the desktop interactivity or responsiveness + very low priority :P + snadge: actually it's not the difference in priority, it's the + fact that some asynchronous processing is done at server side + the priority difference just gives more time overall to servers + for that processing + snadge: when i talk about servers, i mean system (hurd) servers, + no x + yeah.. linux is the same.. in the sense that, that was its + priority and focus + snadge: ? + servers + what are you talking about ? + going back 10 years or so.. linux had very poor desktop + performance + i'm not talking about priorities for developers + it has obviously improved significantly + i'm talking about things like nice values + right.. and some of the modifications that have been done to + improve interactivity of an X desktop, are not relevant to servers + not relevant at all since it's a hurd problem, not an x problem + yeah.. that was more of a linux problem too, some time ago was the + only real point i was making.. a redundant one :p + where i was going with that.. was desktop interactivity is not a + focus for hurd at this time + it's not "desktop interactivity" + it's just correct scheduling + is it "correct" though.. the scheduler in linux is configurable, + and selectable + depending on the type of workload you expect to be doing + not really + it can be interactive, for desktop loads.. or more batched, for + server type loads.. is my basic understanding + no + that's the scheduling policy + the scheduler is cfs currently + and that's the main difference + cfs means completely fair + whereas back in 2.4 and before, it was a multilevel feedback + scheduler + i.e. a scheduler with a lot of heuristics + the gnumach scheduler is similar, since it was the standard + practice from unix v6 at the time + (gnumach code base comes from bsd) + so 1/ we would need a completely fair scheduler too + and 2/ we need to remove asynchronous processing by using mostly + synchronous rpc + im just trying to appreciate the difference between async and sync + event processing + on unix, the only thing asynchronous is signals + on the hurd, simply cancelling select() can cause many + asynchronous notifications at the server to remove now unneeded resources + when i say cancelling select, i mean one or more fds now have + pending events, and the others must be cleaned + yep.. thats a pretty fundamental change though isnt it? .. if im + following you, you're talking about every X event.. so mouse move, + keyboard press etc etc etc + instead of being handled async.. you're polling for them at some + sort of timing interval? + never mind.. i just read about async and sync with regards to rpc, + and feel like a bit of a noob + async provides a callback, sync waits for the result.. got it :p + async is resource intensive on hurd for the above mentioned + reasons.. makes sense now + how about optimising the situation where a select is cancelled, + and deferring the signal to the server to clean up resources until a + later time? + so like java.. dont clean up, just make a mess + then spend lots of time later trying to clean it up.. sounds like + my life ;) + reuse stale objects instead of destroying and recreating them, and + all the problems associated with that + but if you're going to all these lengths to avoid sending messages + between processes + then you may as well just use linux? :P + im still trying to wrap my head around how converting X to use + synchronous rpc calls will improve responsiveness + what has X to do with it? + nothing wrong with X.. braunr just mentioned that hurd doesnt + really handle the async calls so well + there is more overhead.. that it would be more efficient on hurd, + if it uses sync rpc instead + and perhaps a different task scheduler would help also + ala cfs + but i dont think anyone is terribly motivated in turning hurd into + a desktop operating system just yet.. but i could be wrong ;) + i didn't say that + i misinterpreted what you said then .. im not surprised, im a + linux sysadmin by trade.. and have basic university OS understanding (ie + crap all) at a hobbyist level + i said there is asynchronous processing (i.e. server still have + work to do even when there is no client) + that processing mostly comes from select requests cancelling what + they installed + ie.e. you select fd 1 2 3, even on 2, you cancel on 1 and 3 + those cancellations aren't synchronous + the client deletes ports, and the server asynchronously receives + dead name notifications + since servers have a greater priority, these notifications are + processed before the client can continue + which is what makes you feel lag + X is actually a client here + when i say server, i mean hurd servers + the stuff implementing sockets and files + also, you don't need to turn the hurd into a desktop os + any correct way to do fair scheduling will do + can the X client be made to have a higher priority than the hurd + servers? + or perhaps something can be added to hurd to interface with X + well, the future is wayland + ufs .. unfair scheduling.. give priority to X over everything else + hurd almost seams ideal for that idea.. since the majority of the + system is seperated from the kernel + im likely very wrong though :p + snadge: the reason we elevated the priority of servers is to avoid + delaying the processing of notifications + because each notification can spawn a server thread + and this lead to cases where processing notifications was so slow + that spawning threads would occur more frequently, leading to the server + exhausting its address space because of thread stacks + cant it wait for X though? .. or does it lead to that situation + you just described + we should never need such special cases + we should remove async notifications + my logic is this.. if you're not running X then it doesnt + matter.. if you are, then it might.. its sort of up to you whether you + want priority over your desktop interface or whether it can wait for more + important things, which creates perceptible lag + snadge: no it doesn't + X is clearly not the only process involved + the whole chain should act synchronously + from the client through the server through the drivers, including + the file system and sockets, and everything that is required + it's a general problem, not specific to X + right.. from googling around, it looks like people get very + excited about asyncronous + there was a move to that for some reason.. it sounds great in + theory + continue processing something else whilst you wait for a + potentially time consuming process.. and continue processing that when + you get the result + its also the only way to improve performance with parallelism? + which is of no concern to hurd at this time + snadge: please don't much such statements when you don't know what + you're talking about + it is a concern + and yes, async processing is a way to improve performance + but don't mistake async rpc and async processing + async rpc simply means you can send and receive at any time + sync means you need to recv right after send, blocking until a + reply arrives + the key word here is *blocking*ù + okay sure.. that makes sense + what is the disadvantage to doing it that way? + you potentially have more processes that are blocking? + a system implementing posix such as the hurd needs signals + and some event handling facility like select + implementing them synchronously means a thread ready to service + these events + the hurd currently has such a message thread + but it's complicated and also a scalability concern + e.g. you have at least two thread per process + bbl -- cgit v1.2.3