From 2603401fa1f899a8ff60ec6a134d5bd511073a9d Mon Sep 17 00:00:00 2001 From: Thomas Schwinge Date: Tue, 7 Aug 2012 23:25:26 +0200 Subject: IRC. --- open_issues/performance/io_system/read-ahead.mdwn | 280 ++++++++++++++++++++++ 1 file changed, 280 insertions(+) (limited to 'open_issues/performance/io_system') diff --git a/open_issues/performance/io_system/read-ahead.mdwn b/open_issues/performance/io_system/read-ahead.mdwn index 710c746b..657318cd 100644 --- a/open_issues/performance/io_system/read-ahead.mdwn +++ b/open_issues/performance/io_system/read-ahead.mdwn @@ -1565,3 +1565,283 @@ License|/fdl]]."]]"""]] mcsim1: just use sane values inside the kernel :p this simplifies things by only adding the new vm_advise call and not change the existing external pager interface + + +## IRC, freenode, #hurd, 2012-07-12 + + mcsim: so, to begin with, tell us what state you've reached please + braunr: I'm writing code for hurd and gnumach. For gnumach I'm + implementing memory policies now. RANDOM and NORMAL seems work, but in + hurd I found error that I made during editing ext2fs. So for now ext2fs + does not work + policies ? + what about mechanism ? + also I moved some translators to new interface. + It works too + well that's impressive + braunr: I'm not sure yet that everything works + right, but that's already a very good step + i thought you were still working on the interfaces to be honest + And with mechanism I didn't implement moving pages to inactive + queue + what do you mean ? + ah you mean with the sequential policy ? + yes + you can consider this a secondary goal + sequential I was going to implement like you've said, but I still + want to support moving pages to inactive queue + i think you shouldn't + first get to a state where clustered transfers do work fine + policies are implemented in function calculate_clusters + then, you can try, and measure the difference + ok. I'm now working on fixing ext2fs + so, except from bug squashing, what's left to do ? + finish policies and ext2fs; move fatfs, ufs, isofs to new + interface; test this all; edit patches from debian repository, that + conflict with my changes; rearrange commits and fix code indentation; + update documentation; + think about measurements too + mcsim: Please don't spend a lot of time on ufs. No testing + required for that one. + and keep us informed about your progress on bug fixing, so we can + test soon + Forgot about moving system to new interfaces (I mean determine form + of vm_advise and memory_object_change_attributes) + s/determine/final/ + braunr: ok. + what do you mean "moving system to new interfaces" ? + braunr: I also pushed code changes to gnumach and hurd git + repositories + I met an issue with memory_object_change_attributes when I tried to + use it as I have to update all applications that use it. This includes + libc and translators that are not in hurd repository or use debian + patches. So I will not be able to run system with new + memory_object_change_attributes interface, until I update all software + that use this rpc + this is a bit like the problem i had with my change + the solution is : don't do it + i mean, don't change the interface in an incompatible way + if you can't change an existing call, add a new one + temporary I changed memory_object_set_attributes as it isn't used + any more. + braunr: ok. Adding new call is a good idea :) + + +## IRC, freenode, #hurd, 2012-07-16 + + mcsim: how did you deal with multiple page transfers towards the + default pager ? + braunr: hello. Didn't handle this yet, but AFAIR default pager + supports multiple page transfers. + mcsim: i'm almost sure it doesn't + braunr: indeed + braunr: So, I'll update it just other translators. + like other translators you mean ? + *just as + braunr: yes + ok + be aware also that it may need some support in vm_pageout.c in + gnumach + braunr: thank you + if you see anything strange in the default pager, don't hesitate + to talk about it + braunr: ok. I didn't finish with ext2fs yet. + so it's a good thing you're aware of it now, before you begin + working on it :) + braunr: I'm working on ext2 now. + yes i understand + i meant "before beginning work on the default pager" + ok + + mcsim: BTW, we were mostly talking about readahead (pagein) over + the past weeks, so I wonder what the status on clustered page*out* is?... + antrik: I don't work on this, but following, I think, is an example + of *clustered* pageout: _pager_seqnos_memory_object_data_return: object = + 113, seqno = 4, control = 120, start_address = 0, length = 8192, dirty = + 1. This is an example of debugging printout that shows that pageout + manipulates with chunks bigger than page sized. + antrik: Another one with bigger length + _pager_seqnos_memory_object_data_return: object = 125, seqno = 124, + control = 132, start_address = 131072, length = 126976, dirty = 1, kcopy + mcsim: that's odd -- I didn't know the functionality for that even + exists in our codebase... + my understanding was that Mach always sends individual pageout + requests for ever single page it wants cleaned... + (and this being the reason for the dreadful thread storms we are + facing...) + antrik: ok + antrik: yes that's what is happening + the thread storms aren't that much of a problem now + (by carefully throttling pageouts, which is a task i intend to + work on during the following months, this won't be an issue any more) + + +## IRC, freenode, #hurd, 2012-07-19 + + I moved fatfs, ufs, isofs to new interface, corrected some errors + in other that I already moved, moved kernel to new interface (renamed + vm_advice to vm_advise and added rpcs memory_object_set_advice and + memory_object_get_advice). Made some changes in mechanism and tried to + finish ext2 translator. + braunr: I've got an issue with fictitious pages... + When I determine bounds of cluster in external object I never know + its actual size. So, mo_data_request call could ask data that are behind + object bounds. The problem is that pager returns data that it has and + because of this fictitious pages that were allocated are not freed. + why don't you know the size ? + I see 2 solutions. First one is do not allocate fictitious pages at + all (but I think that there could be issues). Another lies in allocating + fictitious pages, but then freeing them with mo_data_lock. + braunr: Because pages does not inform kernel about object size. + i don't understand what you mean + I think that second way is better. + so how does it happen ? + you get a page fault + Don't you understand problem or solutions? + then a lookup in the map finds the map entry + and the map entry gives you the link to the underlying object + from vm_object.h: vm_size_t size; /* + Object size (only valid if internal) */ + mcsim: ugh + For external they are either 0x8000 or 0x20000... + and for internal ? + i'm very surprised to learn that + braunr: for internal size is actual + right sorry, wrong question + did you find what 0x8000 and 0x20000 are ? + for external I met only these 2 magic numbers when printed out + arguments of functions _pager_seqno_memory_object_... when they were + called. + yes but did you try to find out where they come from ? + braunr: no. I think that 0x2000(many zeros) is maximal possible + object size. + what's the exact value ? + can't tell exactly :/ My hurd box has broken again. + mcsim: how does the vm find the backing content then ? + braunr: Do you know if it is guaranteed that map_entry size will be + not bigger than external object size? + mcsim: i know it's not + but you can use the map entry boundaries though + braunr: vm asks pager + but if the page is already present + how does it know ? + it must be inside a vm_object .. + If I can use these boundaries than the problem, I described is not + actual. + good + it makes sense to use these boundaries, as the application can't + use data outside the mapping + I ask page with vm_page_lookup + it would matter for shared objects, but then they have their own + faults :p + ok + so the size is actually completely ignord + if it is present than I stop expansion of cluster. + which makes sense + braunr: yes, for external. + all right + use the mapping boundaries, it will do + mcsim: i have only one comment about what i could see + mcsim: there are 'advice' fields in both vm_map_entry and + vm_object + there should be something else in vm_object + i told you about pages before and after + mcsim: how are you using this per object "advice" currently ? + (in addition, using the same name twice for both mechanism and + policy is very sonfusing) + confusing* + braunr: I try to expand cluster as much as it possible, but not + much than limit + they both determine policy, but advice for entry has bigger + priority + that's wrong + mapping and content shouldn't compete for policy + the mapping tells the policy (=the advice) while the content tells + how to implement (e.g. how much content) + IMO, you could simply get rid of the per object "advice" field and + use default values for now + braunr: What sense these values for number of pages before and + after should have? + or use something well known, easy, and effective like preceding + and following pages + they give the vm the amount of content to ask the backing pager + braunr: maximal amount, minimal amount or exact amount? + neither + that's why i recommend you forget it for now + but + imagine you implement the three standard policies (normal, random, + sequential) + then the pager assigns preceding and following numbers for each of + them, say [5;5], [0;0], [15;15] respectively + these numbers would tell the vm how many pages to ask the pagers + in a single request and from where + braunr: but in fact there could be much more policies. + yes + also in kernel context there is no such unit as pager. + so there should be a call like memory_object_set_advice(int + advice, int preceding, int following); + for example + what ? + the pager is the memory manager + it does exist in kernel context + (or i don't understand what you mean) + there is only port, but port could be either pager or something + else + no, it's a pager + it's a port whose receive right is hold by a task implementing the + pager interface + either the default pager or an untrusted task + (or null if the object is anonymous memory not yet sent to the + default pager) + port is always pager? + the object port is, yes + struct ipc_port *pager; /* Where to get + data */ + So, you suggest to keep set of advices for each object? + i suggest you don't change anything in objects for now + keep the advice in the mappings only, and implement default + behaviour for the known policies + mcsim: if you understand this point, then i have nothing more to + say, and we should let nowhere_man present his work + braunr: ok. I'll implement only default behaviors for know policies + for now. + (actually, using the mapping boundaries is slightly unoptimal, as + we could have several mappings for the same content, e.g. a program with + read only executable mapping, then ro only) + mcsim: another way to know the "size" is to actually lookup for + pages in objects + hm no, that's not true + braunr: But if there is no page we have to ask it + and I don't understand why using mappings boundaries is unoptimal + here is bash + 0000000000400000 868K r-x-- /bin/bash + 00000000006d9000 36K rw--- /bin/bash + two entries, same file + (there is the anonymous memory layer for the second, but it would + matter for the first cow faults) + + +## IRC, freenode, #hurd, 2012-08-02 + + braunr: You said that I probably need some support in vm_pageout.c + to make defpager work with clustered page transfers, but TBH I thought + that I have to implement only pagein. Do you expect from me implementing + pageout either? Or I misunderstand role of vm_pageout.c? + no + you're expected to implement only pagins for now + pageins + well, I'm finishing merging of ext2fs patch for large stores and + work on defpager in parallel. + braunr: Also I didn't get your idea about configuring of paging + mechanism on behalf of pagers. + which one ? + braunr: You said that pager has somehow pass size of desired + clusters for different paging policies. + mcsim: i said not to care about that + and the wording isn't correct, it's not "on behalf of pagers" + servers? + pagers could tell the kernel what size (before and after a faulted + page) they prefer for each existing policy + but that's one way to do it + defaults work well too + as shown in other implementations -- cgit v1.2.3