[[!meta copyright="Copyright © 2010, 2011 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] [[!tag open_issue_gnumach open_issue_hurd]] # IRC, freenode, #hurd, 2010 humm... why does tmpfs try to use the default pager? that's a bad idea, and probably will never work correctly... * slpz is thinking about old issues tmpfs should create its own pagers, just like ext2fs, storeio... slopez@slp-hurd:~$ settrans -a tmp /hurd/tmpfs 10M slopez@slp-hurd:~$ echo "foo" > tmp/bar slopez@slp-hurd:~$ cat tmp/bar foo slopez@slp-hurd:~$ :-) slpz: woo you fixed it? pochu: well, it's WIP, but reading/writing works... I've replaced the use of default pager for the standard pager creation mechanism slpz: err... how is it supposed to use swap space if not using the default pager? slpz: or do you mean that it should act as a proxy, just allocating anonymous memory (backed by the default pager) itself? antrik: the kernel uses the default pager if the application pager isn't responsive enough antrik: it will just create memory objects and provide zerofilled pages when requested by the kernel (after a page fault) youpi: that makes sense I guess... but how is that relevant to the question at hand?... antrik: memory objects will contain the data by themselves antrik: as youpi said, when memory is scarce, GNU Mach will start paging out data from memory objects to the default pager antrik: that's the way in which pages will get into swap space (if needed) the thing being that the tmpfs pager has a chance to select pages he doesn't care any more about slpz: well, the point is that instead of writing the pages to a backing store, tmpfs will just keep them in anonymous memory, and let the default pager write them out when there is pressure, right? youpi: no idea what you are talking about. apparently I still don't really understand this stuff :-( ah, but tmpfs doesn't have pages he doesn't care about, does it? antrik: yes, but the term "anonymous memory" could be a bit confusing. antrik: in GNU Mach, anonymous memory is backed by a memory object without a pager. In tmpfs, nodes will be allocated in memory objects, and the pager for those memory objects will be tmpfs itself slpz: hm... I thought anynymous memory is backed by memory objects created from the default pager? yes, I understand that tmpfs is supposed to be the pager for the objects it provides. they are obviously not anonymoust -- they have inodes in the tmpfs name space but my understanding so far was that when Mach returns pages to the pager, they end up in anonymous memory allocated to the pager process; and then this pager is responsible for writing them back to the actual backing store am I totally off there?... (i.e. in my understanding the returned pages do not reside in the actual memory object the pager provides, but in an anonymous memory object) antrik: you're right. The trick here is, when does Mach return the pages? antrik: if we set the attribute "can_persist" in a memory object, Mach will keep it until object cache is full or memory is scarce or we change the attributes so it can no longer persist, of course without a backing store, if Mach starts sending us pages to be written, we're in trouble so we must do something about it. One option, could be creating another pager and copying the contents between objects. another pager? not sure what you mean BTW, you didn't really say why we can't use the default pager for tmpfs objects :-) well, there're two problems when using the default pager as backing store for translators 1) Mach relies on it to do swapping tasks, so meddling with it is not a good idea 2) There're problems with seqnos when trying to work with the default pager from tasks other the kernel itself (probably, the latter could be fixed) antrik: pager's terminology is a bit confusing. One can also say creating another memory object (though the function in libpager is "pager_create") not sure why "meddling" with it would be a problem... and yeah, I was vaguely aware that there is some seqno problem with tmpfs... though so far I didn't really understand what it was about :-) makes sense now anyways, AIUI now you are trying to come up with a mechanism where the default pager is not used for tmpfs objects directly, but without making it inefficient? slpz: still don't understand what you mean by creating another memory object/pager... (and yeat, the terminology is pretty mixed up even in Mach itself) antrik: I meant creating another pager, in terms of calling again to libpager's pager_create slpz: well, I understand what "create another pager" means... I just don't understand what this other pager would be, when you would create it, and what for... antrik: oh, ok, sorry antrik: creating another pager it's just a trick to avoid losing information when Mach's objects cache is full, and it decides to purge one of our objects anyway, IMHO object caching mechanism is obsolete and should be replaced I'm writting a comment to bug #28730 which says something about this antrik: just one more thing :-) if you look at the code, for most time of their lives, anonymous memory objects don't have a pager not even the default one only the pageout thread, when the system is running really low on memory, gives them a reference to the default pager by calling vm_object_pager_create this is not really important, but worth noting ;-) # IRC, freenode, #hurd, 2011-09-28 mcsim: "Fix tmpfs" task should be called "Fix default pager" :-) mcsim: I've been thinking about modifying tmpfs to actually have it's own storeio based backend, even if a tmpfs with storage sounds a bit stupid. mcsim: but I don't like the idea of having translators messing up with the default pager... slpz: messing up?... antrik: in the sense of creating a number of arbitrarily sized objects slpz: well, it doesn't really matter much whether a process indirectly eats up arbitrary amounts of swap through tmpfs, or directly through vm_allocate()... though admittedly it's harder to implement resource limits with tmpfs antrik: but I've talked about having its own storeio device as backend. This way Mach can pageout memory to tmpfs if it's needed. Do I understand correctly that the goal of tmpfs task is to create tmpfs in RAM? mcsim: It is. But it also needs some kind of backend, just in case it's ordered to page out data to free some system's memory. mcsim: Nowadays, this backend is another translator that acts as default pager for the whole system slpz: pageout memory to tmpfs? not sure what you mean antrik: I mean tmpfs acting as its own pager slpz: you mean tmpfs not using the swap partition, but some other backing store? antrik: Yes. See also: [[open_issues/resource_management_problems/pagers]]. slpz: I don't think an extra backing store for tmpfs is a good idea. the whole point of tmpfs is not having a backing store... TBH, I'd even like to see a single backing store for anonymous memory and named files antrik: But you need a backing store, even if it's the default pager :-) antrik: The question is, Should users share the same backing store (swap space) or provide their own? slpz: not sure what you mean by "users" in this context :-) antrik: Real users with the ability of setting tmpfs translators essentially, I'd like to have a single partition that contains both swap space and the main filesystem (at least /tmp, but probably also all of /run, and possibly even /home...) but that's a bit off-topic :-) well, ideally all storage should be accounted to a user, regardless whether it's swapped out anonymous storage, temporary named files, or permanent files antrik: you could use a file as backend for tmpfs slpz: what's the point of using tmpfs then? :-) (and then store the file in another tmpfs) antrik: mach-defpager could be modified to use storeio instead of Mach's device_* operations, but by the way things work right now, that could be dangerous, IMHO pinotree: hehe .. recursive tmpfs'es ;) slpz: hm, sounds interesting antrik: tmpfs would try to keep data in memory always it's possible (not calling m_o_lock_request would do the trick), but if memory is scarce an Mach starts paging out, it would write it to that file/device/whatever ideally, all storage used by system tasks for swapped out anonymous memory as well as temporary named files would end up on the /run partition; while all storage used by users would end up in /home/* if users share a partition, some explicit storage accounting would be useful too... slpz: is that any different from what "normal" filesystems do?... (and *should* it be different?...) antrik: Yes, as most FS try to synchronize to disk at a reasonable rate, to prevent data losses. antrik: tmpfs would be a FS that wouldn't synchronize until it's forced to do that (which, by the way, it's what's currently happening with everyone that uses the default pager). slpz: hm, good point... antrik: Also, metadata in never written to disk, only kept in memory (which saves a lot of I/O, too). antrik: In fact, we would be doing the same as every other kernel does, but doing it explicitly :-) I see the use in separating precious data (in permanent named files) from temporary state (anonymous memory and temporary named files) -- but I'm not sure whether having a completely separate FS for the temporary data is the right approach for that... antrik: And giving the user the option to specify its own storage, so we don't limit him to the size established for swap by the super-user. either way, that would be a rather radical change... still would be good to fix tmpfs as it is first if possible as for limited swap, that's precisely why I'd prefer not to have an extra swap partition at all... antrik: It's not much o fa change, it's how it works right now, with the exception of replacing the default pager with its own. antrik: I think it's just a matter of 10-20 hours, as much. Including testing. antrik: It could be forked with another name, though :-) slpz: I don't mean radical change in the implementation... but a radical change in the way it would be used antrik: I suggest "almosttmpfs" as the name for the forked one :-P hehe how about lazyfs? antrik: That sound good to me, but probably we should use a more descriptive name :-) ## 2011-09-29 slpz, antrik: There is a defpager in the Hurd code. It is not currently being used, and likely incomplete. It is backed by libstore. I have never looked at it.