[[!meta copyright="Copyright © 2008, 2009, 2013, 2014, 2016 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] [[!tag open_issue_documentation]] The following text needs to be updated. # Original Project Description As a consequence, there is currently no easy way to obtain a listing of all mounted file systems. This also means that commands like `df` can only work on explicitly specified mountpoints, instead of displaying the usual listing. One possible solution to this would be for the translator startup mechanism to update the `mtab` on any `mount`/`umount`, like in traditional systems. However, there are some problems with this approach. Most notably: what to do with passive translators, i.e., translators that are not presently running, but set up to be started automatically whenever the node is accessed? Probably these should be counted among the mounted filesystems; but how to handle the `mtab` updates for a translator that is not started yet? Generally, being centralized and event-based, this is a pretty inelegant, non-hurdish solution. A more promising approach is to have `mtab` exported by a special translator, which gathers the necessary information on demand. This could work by traversing the tree of translators, asking each one for mount points attached to it. (Theoretically, it could also be done by just traversing *all* nodes, checking each one for attached translators. That would be very inefficient, though. Thus a special interface is probably required, that allows asking a translator to list mount points only.) There are also some other issues to keep in mind. Traversing arbitrary translators set by other users can be quite dangerous -- and it's probably not very interesting anyways what private filesystems some other user has mounted. But what about the global `/etc/mtab`? Should it list only root-owned filesystems? Or should it create different listings depending on what user contacts it?... That leads to a more generic question: which translators should be actually listed? There are different kinds of translators: ranging from traditional filesystems ([[disks|hurd/libdiskfs]] and other actual [[stores|hurd/translator/storeio]]), but also purely virtual filesystems like [[hurd/translator/ftpfs]] or [[hurd/translator/unionfs]], and even things that have very little to do with a traditional filesystem, like a [[gzip_translator|hurd/translator/storeio]], [[mbox_translator|hurd/translator/mboxfs]], [[xml_translator|hurd/translator/xmlfs]], or various device file translators... Listing all of these in `/etc/mtab` would be pretty pointless, so some kind of classification mechanism is necessary. By default it probably should list only translators that claim to be real filesystems, though alternative views with other filtering rules might be desirable. After taking decisions on the outstanding design questions, the student will implement both the actual mtab translator, and the necessary interface(s) for gathering the data. It requires getting a good understanding of the translator mechanism and Hurd interfaces in general. # IRC, freenode, #hurd, 2013-04-17 thinking how to get the listing. traversing would be ineffecient, trying to come up with something better what listing ? and traversing what ? mtab well i assumed so be more precise please when the translator is done initalized are written to /etc/mtab will be provided by the translator, and when some one want to read the info just read it this way if their is some credentials like ftp sites pass username can be masked by the translator if some trans dont want to list them, no need to write to file | while unmounting (sorry i couldnt find the right word) , it will pass the mount node address | will have special structure to remove/add mounts example "a /mount-to /mount-from" = add , "r /mount-to" = remove here "/mount-to" will be unique for every mount this have a draw back , we would have to trust trans for the listed data | also "/mount-to" + "/mount-from" could be used a combination for making sure that other trans unable remove others trans mount data sorry but "also "/mount-to" + "/mount-from" could be used a combination for making sure that other trans unable remove others trans mount data" this is a bad idea if we had to print the whole thing braunr, whats ur opinion? you don't need a mtab to "unmount" things on hurd kuldeepdhaka: hum, have you read the project idea ? A more promising approach is to have mtab exported by a special translator, which gathers the necessary information on demand. This could work by traversing the tree of translators, asking each one for mount points attached to it. pinotree, not to unmount, i mean is to remove the for a first implementation, i'd suggest a recursive traversal of root-owned translators braunr, hum, but it did stated it as inefficient where ? para 5 , line 3 and line 6 no traversing "all" nodes would be inefficient translators which host the nodes of other translators could maintain a simple list of active translators ext2fs, e.g. (if that's not already the case) could keep the list of the translators it started we can already see that list with pstree for example but this new list would only retain those relevant for mtab i.e. root-owned ones i would not limit to those though and then filter on their type (e.g. file system ones) pinotree: why ? this way you could have proper per-user /proc/$pid/mounts info we could also very easily have a denial of service but how will the mount point and source point will be listed? they're returned by the translator k you ask /, it returns its store and its options, and asks its children recursively a /home translator would return its store and its options etc.. each translator would build the complete path before returning it sort of, it's very basic but that would be a very hurdish way to do it shall /etc/mtab should be made seek-able and what should be the filesize? content are generated on demand so, it could arise problem (fsize:0 , seek-able:no), ur opinions? kuldeepdhaka: it should have all the properties of a regular file the filesize would be determined after it's generated being empty doesn't imply it's not seekable content is generated on demand so, could cause problem while seeking and filesize, shall i still program as regular file? in two different read, it could generate different content, though same seek pos is used... what ? the content is generated on open ooh, ok # IRC, freenode, #hurd, 2013-06-04 how to see list of all connected translators? you can't directly you can use ps to list processes and guess which are translators (e.g. everything starting with /hurd/) a recursive call to obtain such a list would be useful similar to what's needed to implement /proc/mounts # IRC, freenode, #hurd, 2013-06-25 In context of [[open_issues/mig_portable_rpc_declarations]]. should I go for an iterator like interface instead? btw, what's the expected roundtrip time? don't think that way consider the round trip delay as varying y, is it that bad? no but the less there is the better we think the same with system calls even if they're faster the delay itself isn't the real issue look at how proc provides information (in procfs for example) ## IRC, freenode, #hurd, 2013-06-26 so tell me about the more hurdish way of dealing with that issue creating a specialized translator for this? you need to avoid thinking with centralization in mind the hurd is a distributed system in practice i think proc is the only centralized component in there braunr: would having an mtab translator and having fs translators register to thae be acceptable? that* teythoon: why do you want to centralize it ? translators already register themselves when they get attached to a node we don't want an additional registration have you read the link we gave you ? I did and I got the message, but isn't the concept of /proc/mounts or a mtab file already a centralized one? that doesn't mean the implementation has to be and no, i don't think it's centralized actually it's just a file you can build a file from many sources or if we do it your way recursing on fs translators *but* restricting this to root owned translators also suffering from centralization wrt to the root user? I mean the concept of all mounted filesystems does not apply cleanly to the hurd i don't understand restricting to the root user doesn't mean it's centralized trust has nothing to do with being centralized I guess I'm not used to thinking this way teythoon: i guess that's the main reason why so few developers work on the hurd also the way fs notification is done is also centralized, that could also be done recursively what doyou call fs notification ? and the information I need could just be stuffed into the same mechanism fs translators being notified of system shutdown right that gets a bit complicated because the kernel is also a centralized component it knows every memory object and their pagers it manages all virtual memory there are two different issues here syncing memory and shutting down file systems the latter could be done recursively, yes i wonder if the former could be delegated to external pagers as well teythoon: but that's not the focus of your work aiui, it would take much time sure, but missing an mtab file or better yet /proc/mounts could be an issue for me, at least a cosmetic one, if not a functional one i understand and hacking up a quick solution for that seemed like a good exercise i suggest you discuss it with your mentors they might agree to a temporary centralized solution although i don't think it's much simpler than the recursive one braunr: would that be implemented in libdiskfs and friends? teythoon: i'm not sure, it might be a generic fs operation libnetfs etc.. are also mount points so where would it go if it was generic? libfshelp perhaps translator startup is handled in start-translator-long.c, so in case a startup is successful, I'd add it to a list? i'd say so, yes would that cover all cases, passive and active translators? that's another question do we consider passive translators as mounted ? ah, that was not what i meant i know but it's related start b/c of accessing a passive one vs. starting an active one using settrans start_translator_xxx only spawn active translators it's the same ok the definition of a passive translator is that it starts the active translator on access yeah I can see how that wouldn't be hard to implement i think we want to include passive translators in the mount table so registration must happen before starting the active one so it's a) keeping a list of active translators and b) add an interface to query fs translators for this list and c) an interface to query mtab style information? keeping a list of all translators attached and yes well a is easy b is the real work c would be procfs using b oh? I thought recursing on the translators and querying info would be separate operations? why so ? the point is querying recursively :) and when i say recursively, it's only a logical view ok, yes, it can be implemented this way, so we construct the list while recursing on the translators i think it would be better to implement it the way looking up a node is done in a loop, using a stack? iteratively a translator would provide information about itself (if supported), and referrences to translators locally registered to it could you point me to the node lookup? ah, yes eg., you ask /, it tells you it's on /dev/hd0, read-write, with options, and send rights to /home, /proc, etc.. well rights, references it could be the path itself rights as in a port to the translators? i think the path would be better but i'm not sure it would also allow you to check the permissions of the node before querying path would be nicer in the presence of stacked translators and obviously you'd have the path right away, no need to provide it in the reply true braunr: if we want to list passive translators (and I agree, we should), it isn't sufficient to touch libfshelp, as setting a passive translator is not handled there, only the startup teythoon: doesn't mean you can't add something there that other libraries will use so yes, not sufficient ## IRC, freenode, #hurd, 2013-06-29 braunr: diskfs_S_fsys_set_options uses diskfs_node_iterate to recurse on active translators if do_children is given braunr: I wonder how fast that is in practice braunr: if it's fast enough, there might not even be a need for a new function in fsys.defs and no need to keep a list of translators for that reason braunr: if it's not fast enough, then diskfs_S_fsys_set_options could use the list to speed this up teythoon: on all nodes ? braunr: i believe so, yes, see libdiskfs/fsys-options.c teythoon: well, if it really is all node, you clearly don't want that ## IRC, freenode, #hurd, 2013-07-01 I've ment to ask, the shiny new fsys_get_translators interface, should it return the options for the queried translator or not? i don't think it should ok let's walk through why it shouldn't may I assume that the last argument returned by fsys_get_options is the "source"? how would you know these options ? the source ? I wouldn't actually yes, you wouldn't you'd have to ask the translators for that so the only thing you can do is point to them well, the device to include in the mtab file and the client asks i don't know fsys_get_options tbh well, both tmpfs and ext2fs print an appropriate value for "device" as last argument looks like a bad interface to me options should be options there should be a specific call for the device but if everyone agrees with the options order, you can do it that way for now i guess one that could be used to recreate the "mount" using either mount or settrans just comment it where appropriate I thought that'd be the point? ? % fsysopts tmp /hurd/tmpfs --writable --no-inherit-dir-group --no-sync 48K where is the device ? % settrans -ca tmp $(fsysopts tmp) 15:56 < teythoon> well, both tmpfs and ext2fs print an appropriate value for "device" as last argument 48K i don't see it really ? yes what about ext2fs ? hm ok i see % fsysopts / ext2fs --writable --no-inherit-dir-group --sync=10 --store-type=typed device:hd0s1 i don't think you should consider that as devices but really translator specific options agree I don't ;) b/c the translator calling convention is hardcoded in the mount utility ? I think it's reasonable to assume that this mapping can be reversed theorically you can write a translator that takes no arguments, but just options the 48K string for tmpfs is completely meaningless in fstab, it should be none "tmpfs" the linux equivalent is the size option no, none it's totally ignored and it's recommended to set none rather than the type to avoid confusion u sure? % settrans -cga tmp /hurd/tmpfs --mode=666 6M % settrans -cga tmp /hurd/tmpfs --mode=666 6M % fsysopts tmp /hurd/tmpfs --writable --no-inherit-dir-group --no-sync 6M i've not explained myself clearly it's not ignored by the translator but in fstab, it should be in the options field it's not the source clearly not ah now i'm talking about fstab, but iirc the format is similar in mtab/mounts close, but not the same yes, close ok, so I'll put a method into libfshelp so that translators can explicitly set a device and patch all existing translators to do so? teythoon: what i meant is that, for virtual vile systems (actually file systems with no underlying devices), the device field is normally ignored teythoon: why do you need that for exactly right do they even have a "device" field? (i can see why but i'd like more visibility) pinotree: not yet pinotree: that's what he wants to add but i'd like to see if there is another way to get the information 16:05 < braunr> teythoon: why do you need that for exactly well if I'm constructing a mtab entry I need a value for the device field do we actually need it to be valid ? not necessarily I guess discuss it with your mentors then it has to be valid for e2fsck checks etc. doesn't e2fsck check fstab actually ? i.e. actually for the cases where it's trivial fstab doesn't tell it whether it's mounted I mean fsck checking whether it's mounted not fsck -a oh couldn't we ask the device instead ? looks twisted too that'd mean patching a lot of applications which do similar checks yes teythoon: propose an interface for that with your mentors then yeah, but couldn't you lay it out a little, I mean would it be one procedure or like three? 16:04 < teythoon> ok, so I'll put a method into libfshelp so that translators can explicitly set a device and patch all existing translators to do so? ok why three ? no, I mean when adding stuff to fsys.defs i understood that but why three ? :) it'd be more generic really ? please show a quick example of what you have in mind i honestly don't know, thus I'm asking ;) well first, this device thing bothers me when you look at how we set up our ext2fs translators, you can see they use device:xxx and not /dev/xxx but ok, let's assume it's harmless ok, but isn't the first way actually better? i think it ends up being the same ideally, that's what we want to use as device path but you can recreate a storeio translator using the device:xxx info, the node is useless for that so that we don't need to explicitely set it ? what do you mean ? well, fsysopts / tells currently tells me device:hd0s1 for /, there isn't much choice /dev isn't there yet ah, got it that's why it differs... differs ? from what ? other ext2fs translators are set the same way by the debian installer for example % fsysopts /media/scratch /hurd/ext2fs --writable --no-inherit-dir-group /dev/hd1s1 here it uses the path to the node that's weird was that done by the debian installer ? ah no, that was me :p $ fsysopts /home /hurd/ext2fs --writable --no-inherit-dir-group --store-type=device hd0s6 so as you can see, it's not that simple to infer the device path oho, yet another way ;) right then isn't device:hd0s1 as shortcut for specifying the store type, as done with --store-type=device hd0s1? but perhaps we don't need to yes it is iirc it's something libstore does, per-store prefixes ah that sucks teythoon: you may need to normalize those strings so that they match what's in fstab i.e. unix /dev paths otherwise e2fsck still won't be able to find the translators mounting the device well, if it's mounted actually it just needs to find the matching line in mtab aiui so perhaps a libfshelp function for that, yes braunr: so you suggest adding a normalizing function to libfshelp that creates a /dev/path? yes used by the call you intend to add, which returns that device string as found in fstab found in fstab? so this would only work for translators managed by fstab? no ah a string like the ones found in fstab? yes so that fsck and friends are able to know whether a device is mounted or not i don't see any other purpose for that string in mtab you'd take regular paths as they are, convert device:xxx to /dev/xxx, and return "none" for the rest i suppose ok i'm not even sure it's right youpi: are you sure it's required ? well it's a start and I think it's not too much work aiui, e2fsck may simply find the mount point in fstab, and ask the translator if it's mounted we can refine this later on maybe? or rather, init scripts, using mountpoint, before starting e2fsck teythoon: sure there's this mountpoint issue... I need to run fsysopts / --update early in the boot process otherwise the device ids returned by stat(2)ing / are wrong and mountpoint misbehaves i guess b/c it's the rootfs device ids ? % stat / | grep Device Device: 3h/3d Inode: 2 Links: 22 do you mean the major/minor identifiers ? I do. if I don't do the --update i get seemingly random values i guess that's expected we don't have major/minor values well, they're emulated well, if that's fixable, that'd be really nice ;) we'll never have major/minor values yeah, I understand that but they could be fixed by MAKEDEV when creating device nodes but not having to call fsys_set_options on the rootfs to get the emulation up to speed try doing it from grub not sure it's possible but worth checking by means of an ext2fs flag? yes if there is one i don't know the --update flag, is it new from your work ? braunr: no, it's been there before. -oremount gets mapped to that it's documented by fsysopts, but not by the ext2fs translators libdiskfs source says something about flushing buffers iirc -s what does it do ? teythoon: ok braunr: so the plan is to automatically generate a device path from the translators argz vector but to provide the functionality so translators can set a more appropriate value? did I get the last part of the discussion right? not set, return yeah return from the procedure but settable using libfshelp? why settable ? you'd have a fsys call to obtain the dev string, and the server side would call libfshelp on the fly to obtain a normalized value and return it ah, make a function overrideable that returns an appropriate response? overrideable ? like netfs_append_args you wouldn't change the command line, no isn't that done using weak references or something? no I know sorry i'm lost then never mind, I'll propose a patch early to get your feedback braunr: am I sure that _what_ is required? the device? e2fsck surely needs it, yes a valid device path, yes it can't rely only on fstab yes since users may mount things by hand i've used strace on it and it does perform lookups there (although i also saw uuid magic that i guess wouldn't work yet on the hurd) ## IRC, freenode, #hurd, 2013-07-03 I added a procedure to fsys.defs, added a server stub to my tmpfs translator and wrote a simple client, but something hasn't picked up the new message yet % ./mtab tmp ./mtab: get_translator_info: (ipc/mig) bad request message ID I guess it's libhurduser.so from glibc, not sure though... glibc would only have the client calls what is "% ./mtab tmp" ? mtab is my mtab tool/soon to be a translator testing thing, tmp is an active tmpfs with the appropriate server stub so mtab has the client call, right ? yes then tmpfs doesn't so what to do about it? i set LD_LIBRARY_PATH to my hurd builds lib dir, is that preserved by settrans -a? not really not at all there is a wiki entry about that iirc http://darnassus.sceen.net/~hurd-web/hurd/debugging/translator/ yeah, I read it too once ah on the other hand, using export to set the environment should do the work yes, that did the trick, thanks :) * teythoon got his EOPNOPSUPP... *nomnomnom ? same error ? well I stubbed it out oh no, that's what I've been expecting ;) great :) yes that's better than "mig can't find it" braunr: in that list of active and passive translators that will have to be maintained, do you expect it should carry more information other than the relative path to that translator? like what ? dunno, maybe like a port to any active translator there should we care if any active translator dies and remove the entry if there's no passive translator that could restart it again? don't add anything until you see it's necessary or really useful yes think of something like sshfs when you kill it, it's not reported by mount any more well, for a dynamically allocated list of strings I could use the argz stuff, but if we'd ever add anything else, we'd need a linked list or something, maybe a hash table yes, I thought that'd be useful use libihash for no now braunr: but what would I use as keys? the relative path should be unique (unless translators are stacked... hmmm), but that's the value I'd like to store and ihash keys are pointers stacked translators are an kinda interesting case for mtab anyways... why not store the string address ? i suppose that, for stacked translators, the code querying information would only return the topmost translator since this is the one which matters for regular clients (if i'm right) wouldn't that map strings that are equal but stored at different locations to different values? that'd defeat the point I suppose so, yes then add a layer that looks for existing strings before adding the list should normally be small so a linear lookup is fine yeah sure, but then there's little advantage of using ihash in the first place, isn't it? over what ? over not using it at all how would you store the list then ? it's either ll or ll+ihash uh no let me check there is ihash_iterate so you don't need a linked list so how do I store my list of strings to deduplicate the keys? you store pointers and on addition, you iterate over all entries, making sure none matches the new one and if it does, you replace it i guess depending on how you design the rest in an dynamically allocated region of memory? i don't understand your strings should be dynmaically allocate, yes no the array of char * your data structure being managed by libihash, you don't care about allocation what array ? ah, got it... right. there is only one structure here, an ihash of char * yes, I got the picture ;) goo d actually, the lookup wouldn't be linear since usually, hash tables have stale entries heh... what forest?!? but that's ok teythoon: ? the one I couldn't make out b/c of all the trees... ? ah, it's not important. there is this saying over here, not sure if there's an english equivalent ok got it we have the same in french I ran into a problem with my prototype if an translator is set in e. g. diskfs_S_file_set_translator, how do I get the path to that node? I believe there cannot be a way to do that, b/c the mapping is not bijective it doesn't have to be ok, so how do I get *a* path for this node? that's another question do you see how the node is obtained ? np = cred->po->np; yes the translation occurred earlier you need to find where then perhaps, you'll need to carry the path along or if you're lucky, it will still be there somewhere the translation from path to node? yes doesn't that happen in the client? and the client hands a file_t to the file_set_translator routine? the relative lookup can't happen in the client the server can (and often does) retain information between two RPCs uh, I can access information from a previous rpc? is that considered safe? think of open() then read() a simple int doesn't carry enough information that's why it's a descriptor ah, the server retains some state, sure what it refers to is the state retained between several calls the object being invoked by clients teythoon: what is the "passive" parameter passed to diskfs_S_file_set_translator ? braunr: argz vector of the passive translator so it is a name but we also want active translators and what is active ? not the name of the node though active is the port (?) to the active translator I guess fsys_t, looks that way yes i suppose you could add the path to the peropen structure ok see diskfs_make_peropen braunr: but translation happens in dir_lookup in all places I've seen diskfs_make_peropen used, the path is not available why did you point me to diskfs_make_peropen? s/dir_lookup/diskfs_lookup/ diskfs_lookup operates on struct node, so the path would have to be stored there, right? teythoon: dir_lookup should call diskfs_make_peropen at least diskfs_S_dir_lookup does and the path is present there braunr: right hrm... I added a path field to struct peropen and initialize it properly in diskfs_make_peropen, but some bogus values keep creeping in :/ first of all, make it a dynamically allocated string it is not a fixed sized embedded array good yes if you really need help debugging what's happening, feel free to post your current changes somewhere there is a struct literal in fsys-getroot.c, but i fixed that as well % ./mtab tmp none tmp ../tmpfs/tmpfs writable,no-inherit-dir-group,no-sync 0 0 none tmp/bar ../tmpfs/tmpfs writable,no-inherit-dir-group,no-sync 0 0 none tmp/foo ../tmpfs/tmpfs writable,no-inherit-dir-group,no-sync 0 0 none tmp/foo/bar ../tmpfs/tmpfs writable,no-inherit-dir-group,no-sync 0 0 :) ## IRC, freenode, #hurd, 2013-07-10 btw, I read getcwd.c and got the idea however this situation is different afaict getcwd has a port to the current working directory, right? so they can do open_dir with .. as relative path but all I've got is a port referencing the node the translator is being attached to s/open_dir/dir_lookup/ and that is not necessarily a directory, so dir_lookup fails with not a directory as far as I can see it is not possible to get the directory a node is in from a port referencing that node dir_lookup has to be handled by all nodes, not just directories but file nodes only support "looking up" the empty string not empty, but null: This call is required to be supported by all files (even non-directories) if the filename is null, and should function in that case as a re-open of the file. */ why do you want the directory ? 10:40 < teythoon> as far as I can see it is not possible to get the directory a node is in from a port referencing that node to readdir(3) it and figure out the name of the node the translator is bound to similar to what getcwd does that's out of the question wasn't that was youpi was suggesting? you may have a lot of nodes in there, such a lookup shouldn't be done i didn't see that detail "│ Concerning storing the path, it's a bit sad to have to do that, and │ it'll become wrong if one moves the mount points. Another way would │ be to make the client figure it out by itself from a port to the mount │ point, much like glibc/sysdeps/mach/hurd/getcwd.c. It'll be slower, but │ should be safer. The RPC would thus return an array of ports to the │ mount points instead of an array of strings. yes i remember that but i didn't understand well how getcwd work s another scalability issue not a big one though, we rarely have translators in directories with thousands of nodes so why not teythoon: do it as youpi suggested well if you can eh if not, i don't know 10:47 < teythoon> │ it'll become wrong if one moves the mount points. Another way would yes, I know... :/ well, I'm not even sure it is possible to get the directory a node is in from the port referencing the node as in, I'm not sure if the information is even there b/c a filesystem is a tree, directories are nodes and files are leafs all non-leaf nodes reference their parent to allow traversing the tree starting from any directory but why would a leaf reference its parent(s - in case of hardlinks)? uh, for the same reason ? sure, it would be nice to do that, but I dont think this is possible on unixy systems ? you cannot say fchdir(2) to a fd that references a file do you mean /path/to/file/../ ? yes only that /path/to/file is given as fd or port when i pasted 10:49 < braunr> 10:47 < teythoon> │ it'll become wrong if one moves the mount points. Another way would i was actually wondering if it was true ah why can't the path be updated at the same time ? it's a relative path anyway completely managed by the parent translator ah right it's still kind of hacky, but I cannot see how to do this properly hacky ? but yes, updating the path should work I guess or sad what i find hacky is to set translators in two passes otherwise we'd only keep the translator paths not all paths true but then, it only concerns open nodes and again, there shouldn't be too many of them so actually it's ok braunr: I understand the struct nodes are cached in libdiskfs, so wouldn't it be easier to attach the path to that struct instead of struct peropen so that all peropen objects reference the same node object? so that the path can be updated if anyone dir_renames it *all peropen objects derived from the same file name that is teythoon: i'm not sure nodes could be real nodes (i.e. inodes) there can be several paths for the same inode braunr: I'm aware of that, but didn't we agree the other day that any path would do? i don't remember we did i don't know the details well, but i don't think setting a translator on a hard link should set the translator at the inode level on the other hand, if a new inode is created to replace the previous one (or stack over it), then storing the path there should be fine braunr: I don't think I can update the paths if they're stored in the peropen struct how would I get a reference to all those peropen objects? ? first, what's the context when you talkb about updating paths ? well, youpi was concerned about renaming a mount point and you implied that this could be managed can we actually do that btw ? what? renaming a mount point yep, just tried i mean, on a regular unix system like linux $ mv test blah mv: cannot move `test' to `blah': Device or resource busy (using sshfs so YMMV) do you have anything (shells, open files, etc) inside it? no i'll try with an empty loop-mounted ext4 I was testing on the Hurd, worked fine there even with a shell inside same thing i consider it a bug we may want to check what posix says about it o_O and decide not to support renaming why? start a discussion in ml, maybe roland can chime in it complicates things ah yes sure, but I can move or rename a directory, why should it be different with a mount point? because it's two of them they're stacked if we do want to support that, we must be very careful about atomically updating all the stack ok braunr: I'm trying to detect dying translators to remove them from the list of translators what port can I use for that purpose? if I use the bootstrap port, can I then use the same method as init/init.c uses? just defining a do_mach_notify_dead_name function and the multiplexer will call this? teythoon: possibly braunr: we'll see shortly... I get KERN_INVALID_CAPABILITY indicating that my bootstrap port is invalid when calling mach_port_request_notification to get the dead name notification I mean is the translator already started when you do that ? yes, at least I think so, I'm hooking into diskfs_S_file_set_translator and that gets an active translators port also the mach docs suggests that the notification port is invalid, not the name port referencing the translator i guess it shouldn't oh please show the code but beware, if the translator is started, assume it could die immediately braunr: http://paste.debian.net/15371/ line 87 teythoon: notify can't be bootstrap what do you have in mind when writing this ? i'm not sure i follow I want to be notified if an active translator goes away to remove it from the list of translators ok but then create a send-once right and wait on it also, why do you want to be notified ? isn't this already done ? or can't do it lazily on access attempt ? +you in the client? in the parent server what happens currently when a translator dies is the parent notified ? or does it give an invalid right ? ah, i think so then you don't need to do it again right, I overlooked that ## IRC, freenode, #hurd, 2013-07-12 recursively traversing all translators from / turns out to be more dangerous than I expected ... if done by a translator bound somewhere below /... my interpretation is that the mtab translator tries to talk to itself and deadlocks (and as a side effect the whole system kinda just stops...) ## IRC, freenode, #hurd, 2013-07-15 teythoon: did you discuss with braunr about returning port vs path in fsys_get_children? youpi: we did as I wrote I looked at the getcwd source you pointed me at and I started to code up something similar but as far as I can see there's no way to tell from a port referencing a file the directory this file is located in ah, right, there was a [0] mail teythoon: because it doesn't have a "..", right about Neals concerns, he's right about not covering passive translators very well but the solution he proposed was similar to what I tried to do first I don't like half-covering passive translators at all, to be honest :) either covering them completely, or not at all, would be fine and then braunr convinced me that the "recursive" approach is more elegant and hurdish, and I came to agree with him youpi: one could scan the filesystem at translator startup and populate the list by "Neal's solution", you mean an mtab registry? yes so, let's see what linux does when renaming parent directories mount points you mean? teythoon: browsing the whole filesystem just to find passive translators is costly teythoon, braunr: and that won't prevent the user from unexpectedly starting other translators at will scary youpi: but that requires the privilege to open the device the fact that a passive translator is set is nothing more than a user having the intent of starting a translator linux retains the original path in the mount table heh youpi: any unprivileged user can trigger a translator startup sure, but root can do that too and expect the system to behave nicely but if I'm root and want to fsck something, I won't start translators accessing the device just before that but if there's a passive translator targetting the device, someone else might do that root does not always completely control what he's doing linux for instance does prevent from mounting a filesystem being checked but still, including passive translators in the list would at least prevent anyone starting an translator by accident, isn't that worth doing then? if there's a way to prevent root too, that's better than having a half-support for something which we don't necessarily really want (i.e. an exclusive lock on the underlying device) right, that would also do the trick btw, some programs or scripts seem to hardcode /proc/mounts and procfs and I cannot bind a translator to /proc/mounts since it is read-only and the node does not exist IMHO automatically starting translators is a generic feature, and passive translator is just a specific instance of it; but we could very well have, like an "autofs" that automatically start translators in tar archives and iso images, allowing to cd into any tar/iso on the system; implementing such things is part of the Hurd flexibility, the "core system" shouldn't be too aware on how translators are started so in the end, storing where the active translator was started first seems okayish according to what linux has been exposing for decades kilobug: indeed it could serve a mounts with a passive translator by default, or a link to /run/mtab, or an simple file so we could bind a translator to that node I'd tend to think that /proc/mounts should be a passive translator and /run/mtab / /etc/mtab a symlink to it not being to choose the translator is a concern however ok, I'll look into that it could be an empty file, and people be able to set a translator on it if it had a passive translator, people still could bind their own translator to it later on, right? afaics the issue currently is mostly, that there is no mounts node and it is not possible to create one right cool so with the actual path, you can even check for caller's permission to read the path i.e. not provide any more information than the user would be able to get from browsing by hand sure, that concern of Neil's is easy to address I'm not so much concerned by stale paths being shown in mtab the worst that can happen is a user not being able to umount the path but he can settrans -g it (which he can't on linux ;) ) yes, and the device information is still valid yes despite the parent dir being renamed, linux is still able to umount the new path and so is our current umount good (if one uses the mount point as argument) what's the current plan concerning /proc/mounts ? serving a node with a passive translator record ? so that /hurd/mtab / is started on access i mean, still planning on using the recursive approach instead of a registry ? ah I do not feel confident enough to decide this, but I agree with you, it feels elegant and it works :) modulo the translator deadlocking if it talks to itself, any thoughts on that? it is a non-threaded translator I guess? currently yes making it threaded should fix the issue I tried to make the mtab translator multithreaded but that didn't help that's odd maybe I did it wrong i don't find it surprising well, not that surprising :p on what lock does it block ? as far as i can see the only difference of hello and hellot-mt is that it uses a different dispatcher and has lot's of locking, right? braunr: I'm not sure, partly because that wrecked havoc on the whole system it just freezes but it wasn't permanent. once i let it running and it recovered consider using a subhurd ah right, I ment to set up one anyway, but my first attempts were not successful, not sure why anyway, is there a way to prevent this in the first place? if one could compare ports that'd be helpful Mmm, did you try to simply compare the number? with the bootstrap port I presume? Mmm, no, the send port and the receive port would be different no, with the receive port ah comparing the numbers should work youpi: no they should be the same braunr: ah, then it should work yes that's why there are user ref counts ok only send-once rights have their own names btw, I'll push my work to darnassus from now on, e.g. http://darnassus.sceen.net/cgit/?p=teythoon/hurd.git;a=shortlog;h=refs/heads/feature-mtab-translator-v3-wip ### IRC, freenode, #hurd, 2013-11-20 teythoon: ah thanks for making mtab multithreaded :) i forgot about that ## [[open_issues/libnetfs_passive_translators]] ## IRC, freenode, #hurd, 2013-07-16 which port is the receive port of a translator? I mean, how is it called in the source, there is no port in sight named receive anywhere I looked. teythoon: what is the "receive port of a translator" ? braunr: we talked yesterday about preventing the mtab deadlock by comparing ports I asked which one to use for the comparison, youpi said the receive port i'm not sure what he meant it could be the receive port used for the RPC but i don't think it's exported past mig stub code weird, I just reread it. I asked if i should use the bootstrap port, and he said receive port, but it might have been addressed to you? you were talking about send and receive ports being singletons or not umm no i answered him he was wondering if the receive port could actually be used for comparison i said it can but still, i'm not sure what port if it's urgent, send him a mail no, my pipeline is full of stuff I can do instead ;) :) ## IRC, freenode, #hurd, 2013-07-17 braunr: btw, comparing ports solved the deadlock in the mtab translator rather easily :) which port then ? currently I'm stuck though, I'm not sure how to address Neals concern wrt to access permission checks I believe it's called control port ok the one one gets from doing the handshake with the parent i thought it was the bootstrap port but i don't know the details so i may be wrong anyway yes what is the permission problem again ? 871u73j4zp.wl%neal@walfield.org well, you could perform a lookup on the stored path as if opening the node if I look at any server implementation of a procedure from fs.defs (say libtrivfs/file-chmod.c [bad example though, that looks wrong to me]), there is permission checking being done any server implementation of a procedure from fsys.defs lacks permission checks, so I guess it's being done somewhere else i must say i'm a bit lost in this discussion i don't know :/ can *you* sum up the permission problem please ? i mean here, now, in just a few words ? ok, so I'm extending the fsys api with the get_children procedure that one should not return any children x/y if the user doing the request has no read permissions on x really ? why so ? the same way ls x would not reveal the existence of y i could also say unlike cat /proc/mounts i can see why we would want that i also can see why we could let this behaviour in place let's admit we do want it true, but I thought this could easily be addressed what you could do is now I'm not sure b/c I cannot even find the permission checking code for any fsys_* function for each element in the list of child translators perform a lookup on the stored path on behalf of the user and add to the returned list if permission checks pass teythoon: note that i said lookup on the path, which is an fs interface i assume there is no permission checking for the fsys interface because it's done at the file (fs) level i think so too, yes sure, if I only knew who made the request in the first place the file-* options have a convenient credential handle passed in as first parameter s/options/procedures/ surely the fsys-* procedures also have a means of retrieving that information, I just don't know how mig magic teythoon: see file_t in hurd_types.defs there is the macro FILE_INTRAN which is defined in subdirectories (or not) ah, retrieving the control port requires permissions, and the fsys-* operations then operate on the control port? see libdiskfs/fsmutations.h for example uh yes but that's for < braunr> i assume there is no permission checking for the fsys interface because it's done at the file (fs) level i'm answering < teythoon> sure, if I only knew who made the request in the first place teythoon: do we understand each other or is there still something fuzzy ? braunr: thanks for the pointers, I'll read up on that a bit later teythoon: ok ## IRC, freenode, #hurd, 2013-07-18 braunr: back to the permission checking problem for the fsys_get_children interface I can see how this could be easily implemented in the mtab translator, it asks the translator for the list of children and then checks if the user has permission to read the parent dir but that is pointless, it has to be implemented in the fsys_get_children server function yes why is it pointless ? because one could circumvent the restriction by doing the fsys_get_children call w/o the mtab translator uh no you got it wrong what i suggested is that fsys_get_children does it before returning a list the problem is that the mtab translator has a different identity from the users accessing it yes, but I cannot see how to do this, b/c at this point I do not have the user credentials get them how? 16:14 < braunr> mig magic 16:15 < braunr> teythoon: see file_t in hurd_types.defs 16:16 < braunr> there is the macro FILE_INTRAN which is defined in subdirectories (or not) 16:16 < braunr> see libdiskfs/fsmutations.h for example i saw that is there a problem i don't see then ? i suppose you should define FSYS_INTRAN rather but the idea is the same won't that change all the function signatures of the fsys-* family? that's probably the only reason not to implement this feature right now then again, that change is probably easy and mechanic in nature, might be an excuse to play around with coccinelle why not if you have the time right, if this can be done, the mtab translator (if run as root) could get credentials matching the users credentials to make that request, right? i suppose i'm not sure it's easy to make servers do requests on behalf of users on the hurd which makes me wonder if the mtab functionality shouldn't be implemented in glibc eheheh .... but probably not well, I'll try out the mig magic thing and see how painful it is to fix everything ;) good luck honestly, i'm starting to think it's deviating too much from your initial goal i'd be fine with a linux-like /proc/mounts with a TODO concerning permissions ok, fine with me :) confirm it with the other mentors please we have to agree quickly on this y? braunr: I actually believe that the permission issue can be addressed cleanly and unobstrusively braunr: would you still be opposed to the get_children approach if that is solved? the filesystem is a tree and the translators "creating" that tree are a more coarse version of that tree having a method to traverse that tree seems natural to me teythoon: it is natural i'm just worried it's a bit too complicated, unnecessary, and out-of-scope for the problem at hand (which is /proc/mounts, not to forget it) ## IRC, freenode, #hurd, 2013-07-19 braunr: I think you could be a bit more optimistic and supportive of the decentralized approach I know the dark side has cookies and strong language and it's mighty tempting but both are bad for you :p ## IRC, freenode, #hurd, 2013-07-22 teythoon: AIUI, you should be able to run the mtab translator as no-user (i.e. no uid) youpi: yes, that works fine teythoon: so there is actually no need to define FSYS_INTRAN, doing it by hand as you did is fine, right? (/me backlogs mails...) youpi: yes, the main challenge was to figure out what mig does and how the cpp is involved heh :) my patch does exactly the same, but only for this one server function youpi: I'm confused by your mail, why are read permissions on all path components necessary? teythoon: only execution normally teythoon: to avoid letting a user discover a translator running on a hidden directory braunr: exactly, and that is tested e.g. ~/home/foo is o+x, but o-r and I have a translator running on ~/home/foo/aZeRtYuyU I don't want that to show up on /proc/mounts youpi: i don't understand either: why isn't execution permission enough ? youpi: but that requires testing for read on the *last* component of the *dirname* of your translator, and that is tested let me take another example :) e.g. ~/home/foo/aZeRtYuyU is o+x, but o-r and I have a translator running on ~/home/foo/aZeRtYuyU/foo ergl sorry, I meant this actually: yes, that won't show up then in the mtab for users that are not you and not root e.g. ~/home/foo is o+x, but o-r and I have a translator running on ~/home/foo/aZeRtYuyU/foo ah hmm, good point ? * braunr still confused well, qwfpgjlu is the secret and that is revealed by the fsys_get_children procedure then i didn't understand the description of the call right > + /* check_access performs the same permission check as is normally > + done, i.e. it checks that all but the last path components are > + executable by the requesting user and that the last component is > + readable. */ braunr: youpi argues that this is not enough in this case from that, it looks ok to me the function and the documentation agree, yes but that's not what we want and that's where i fail to understand again, see my example i am 10:43 < youpi> e.g. ~/home/foo is o+x, but o-r ok so the user is not supposed to find out the secret then your example isn't enough to describe what's wron g checking read permission only on ~/home/foo/aZeRtYuyU will not garantee that ah i thought foo was the last component no, that's why I changed my example hum 10:43 < youpi> e.g. ~/home/foo is o+x, but o-r 10:43 < youpi> and I have a translator running on ~/home/foo/aZeRtYuyU/foo i meant, the last foo still, this is easily fixed sure just has to be :) youpi, braunr: so do you think that this approach will work? I believe so i still don't see the problem, so don't ask me :) i've been sick all week end and hardly slept, which might explain in the example, "all but the last path components" is "~/home/foo/aZeRtYuyU" right ? braunr: well, I haven't looked at the details but be it the last, or but-last doesn't change the issue if my ~/hidden is o-r,o+x and I have a translator on ~/hidden/a/b/c/d/e checking only +x on hidden is not ok but won't the call also check a b c d ? yes, but that's not what matters what matters is that hidden is o-r hm so the mtab translator is not supposed to reveal that there is an "a" in there ok i'm starting to understand so r must be checked on all components too yes right to simulate the user doing ls, cd, ls, cd, etc. well, not cd ah for being able to do ls, you have to be able to do cd as an ordered list of commands ok agreed. can you think of any more issues? so both x and r must be checked so in the end this RPC is really a shortcut for a find + fsysopts script teythoon: I don't see any teythoon: i couldn't take a clear look at the patch but do you perform a lookup on all nodes ? yes, all nodes on the path from the root to the one specified by the mount point entry in the active translator list let me rephrase do you at some point do a lookup, similar to a find, on all nodes of a translator ? no good yes iirc, neal raised that concern once and I'll also fix settrans --recursive not to iterate over *all* nodes either great :) fsys_set_options with do_children=1 currently does that (I've only looked at the diskfs version) ## IRC, freenode, #hurd, 2013-07-27 youpi: ah, I just found msg_get_init_port, that should make the translator detection feasible ## IRC, freenode, #hurd, 2013-07-31 braunr: can I discover the sender of an rpc message? teythoon: no teythoon: what do you mean by "sender" ? braunr: well, I'm trying to do permission checks in the S_proc_mark_essential server function ok so, the sending user that should be doable I've got a struct proc *p courtesy of a mig intran mutation and a port lookup but that is not necessarily the sender, right? proc is really the server i know the least :/ there is permission checking for signals it does work you should look there yes, there are permission checks there but the only argument the rpc has is a mach_port_t refering to an object in the proc server yes anyone can obtain such a handle for any process, no? can you tell where it is exactly please ? i don't think so, no what? 14:42 < teythoon> but the only argument the rpc has is a mach_port_t refering to an object in the proc server ah the code you're referring to a common way to give privileges to public objects is to provide different types of rights a public (usually read-only) right and a privileged one, like host_priv which you may have seen acting on (modifying) a remote object normally requires the latter http://paste.debian.net/20795/ i thought you were referring to existing code well, there is existing code doing permission checks the same way I'm doing it there where is it please ? mgt.c ? proc/mgt.c (S_proc_setowner) for example yes that's different but anyone can obtain such a reference by doing proc_pid2proc the sender is explicitely giving the new uid yes but not anyone is already an owner of the target process (although it may look like anyone has the right to clear the owner oO) see, that's what made me worry, it is not checked who's the sender of the message unless i'm missing something here ah I am pid2proc returns EPERM if one is not the owner of the process in question all is well ok it still requires the caller process though what? see check_owner the only occurrence i find in the hurd is in libps/procstat.c MGET(PSTAT_PROCESS, PSTAT_PID, proc_pid2proc (server, ps->pid, &ps->process)); server being the proc server AIUI yes, most likely but pid2proc describes this first argument to be the caller process ah but it is ? mig magic :p MIGSFLAGS="-DPROCESS_INTRAN=pstruct_t reqport_find (process_t)" \ MIGSFLAGS="-DPROCESS_INTRAN=pstruct_t reqport_find (process_t)" \ ah nice hum no this just looks up the proc object from a port name, which is obvious what i mean is 14:53 < braunr> MGET(PSTAT_PROCESS, PSTAT_PID, proc_pid2proc (server, ps->pid, &ps->process)); this is done in libps which can be used by any process server is the proc server for this process (it defines the process namespace) yes, but isn't the port to the proc server different for each process? no, the port is the same (the name changes only) ports are global non-first class objects and the proc server can thus tell with the lookup which process it is talking to? that's the thing from pid2proc : S_proc_pid2proc (struct proc *callerp [...] if (! check_owner (callerp, p)) check_owner (struct proc *proc1, struct proc *proc2) "Returns true if PROC1 has `owner' privileges over PROC2 (and can thus get its task port &c)." callerp looks like it should be the caller process but in libps, it seems to be the proc server this looks strange to me yep, to me too, hence my confusion could be a bug that allows anyone to perform pid2proc braunr: well, proc_pid2proc (getproc (), 1, ...) fails with EPERM as expected for me ofc it does with getproc() but what forces a process to pass itself as the first argument ? braunr: nothing, but what else would it pass there? 14:53 < braunr> MGET(PSTAT_PROCESS, PSTAT_PID, proc_pid2proc (server, ps->pid, &ps->process)); everyone knows the proc server ok now, that's weird teythoon: does getproc() return the proc server ? I think so, yes damn those distributed systems, all of their sources are so distributed too i suspect there is another layer of dark glue in the way I cannot even find getproc :/ hurdports.c:GETSET (process_t, proc, PROC) that's the dark glue :p ah, so it must be true that the ports to the proc server are indeed process specific, right? ? well, it is not one port to the proc server that everyone knows it is what makes you think it's not ? proc_pid2proc (getproc (), 1, ...) fails with EPERM for anyone not being root, but succeeds for root hm right if getproc () were to return the same port, the proc server couldn't distinguish these indeed in which case getproc() actually returns the caller's process object at its proc server yes, that is better worded teythoon: i'm not sure it's true actually :/ braunr: well, exploit or it didn't happen teythoon: getproc() apparently returns a bootstrap port we must find the code that sets this port i have a hard time doing that :/ isn't part of the stuff which is passed to a new process by exec? braunr: I know that feeling pinotree: probably still hard to find .. search in glibc braunr: exec/exec.c:1654 asks the proc server for the proc object to use for the new process so how much of hurd do I have to rebuild once i changed struct procinfo in hurd_types.h? oh noez, glibc uses it too :/ ## IRC, freenode, #hurd, 2013-08-01 I need some pointers on building the libc, specifically how to point libcs build system to my modified hurd headers nlightnfotis: hi nlightnfotis: you rebuild the libc right? do you have any hurd specific pointers for doing so? teythoon, I have not yet rebuild the libc (I was planning to, but I followed other courses of action) Thomas had pointed me to some resources on the Hurd website. I can look them up for you teythoon, here are the instructions http://darnassus.sceen.net/~hurd-web/open_issues/glibc/debian/ and the eglibc snapshot is here http://snapshot.debian.org/package/eglibc/ nlightnfotis: yeah, I found those. the thing is I changed a struct in the hurd_types.h header, so now I want to rebuild the libc with that header and I cannot figure out how to point libcs build system to my hurd headers :/ can you patch eglibc and build that one instead? teythoon: put your header in the appropriate /usr/include/ dir pochu: is there no other way? iirc nope teythoon: you may be able to pass some flag to configure, but I don't know if that will work in this specific case ouch >,< that explains why I haven't found one check ./configure --help, it's usually FOO_CFLAGS (so something like HURD_CFLAGS maybe) but then you may need _LIBS as well depending on how you changed the header... so in the end it's just easier to put the header in /usr/include/ teythoon: did you find the info for your libc build ? braunr: well, i firmlinked my hurd_types.h into /usr/include/... ew i recommend building debian packages but the build was not successful, looks unrelated to my changes though I tried that last week and the process took more than eight hours and did not finish use darnassus it takes about 6 hours on it I shall try again and skip the unused variants i also suggest you use ./debian/rules build and then interrupt the build process one you see it's building object files go to the hurd-libc-i386 build dir, and use make lib others make lib builds libc, others is for companion libraries lik libpthread actually building libc takes less than an hour so once you validate your build this way, you know building the whole debian package will succedd succeed* so how do I get the build system to pick up my hurd_types.h? sorry if this is obvious to you, you might be more familiar with debian than i am :) patch the hurd package append your own version string like +teythoon.hurd.1 install it then build libc i'll reboot darnassus so you have a fresh and fast build env almost a month of uptime without any major issue :) err, but I cannot install my hurd package on darnassus, can I? I don't think that'd be wise even if it were possible teythoon: rebooted, enjoy why not ? i often do it for my own developments teythoon: screen is normally available teythoon: be aware that fakeroot-tcp is known to hang when pfinet is out of ports (that's a bug) it takes more time to reach that bug since a patch that got in less than a year ago, but it still happens the hurd packages are quick to build, and they should only provide the new header, right ? you can include the functionality too in the packages if you're confident enough but my latest work on the killing of essential processes issues involves patching hurd_types.h and that in a way that breaks the ABI, hence the need to rebuild the libc (afaiui) teythoon: yes, this isn't uncommon braunr: this is much more intrusive than anything I've done so far, so I'm not so confident in my changes for now teythoon: show me the patch please braunr: it's not split up yet, so kind of messy: http://paste.debian.net/21403/ teythoon: did you make sure to add RPCs at the end of defs files ? yes, I got burned by this one on my very first attempt, you pointed out that mistake :) ok you're changing struct procinfo this really breaks the abi yes i.e. you can't do that I cannot put it at the end b/c of that variable length array you probably should add another interface that'd be easier, sure, but this will slow down procfs even more, no? that's secondary it won't be easier, breaking the abi may break updates in which case it's impossible another way would be to ues a new procinfo struct like struct procinfo2 but then you need a transition step so that all users switch to that new version which is the best way to deal with these issues imo, but this time not the easiest :) ok, so I'll introduce another rpc and make sure that one is extensible hum no this usually involves using a version anyway no? but it is likely that we need to save more addresses of this kind in the future in which case it will be hanlded as an independant problem with a true solution such as the one i mentioned it could return an array of vm_address_ts with a length indicating how many items were returned it's ugly the code is already confusing enough keep names around for clarity ok, point taken really, don't mind additional RPCs when first adding new features once the interface is stable, a new and improved version becomes a new development of its own you're invited to work on that after gsoc :) but during gsoc, it just seems like an unnecessary burden ok cool, I really like that way of extending Hurd, it's really easy and feels so natural i share your concern about performances, and had a similar problem when adding page cache information to gnumach in the end, i'll have to rework that again because i tried to extend it beyond what i needed true, I see how that could happen easily the real problem is mig mig limits subsystems to 100 calls it's clearly not enough in x15, i intend to use 16 bits for subsystems and 16 bits for RPCs, which should be plenty that limit seems rather artificial, it's not a power of two yes it is so let's fix it mach had many artificial static limits eh :D not easy replies are encoded by taking the request ID and adding 100 uh "uh" indeed so we need an intermediate version of mig that accepts both id+100 and dunno id+2^x as replies for id or -id - 1 that would completely break the abi braunr: how so? the change would be in the *_server functions and be compatible with the old id scheme how do you make sure id+2^x doesn't conflict with another id ? oh, the id is added to the subsystem id? to obtain a global message id? yes ah, I see ah, but the hurd subsystems are 1000 ids apart so id+100 or id +500 would work we need to make sure it's true always true so how many bits do we have for the message id in mach? (mig?) mach shouldn't care, it's entirely a mig thing well yes and no mach defines the message header, which includes the message id see mach/message.h mach_msg_id_t msgh_id; typedef integer_t mach_msg_id_t; well, if that is like a 32 bit integer, then allow -id-1 as reply and forbid ids > 2^x / 2 yes seems reasonable that'd give us an smooth upgrade path, no? i think so ## IRC, freenode, #hurd, 2013-08-28 teythoon: Mmm, your patch series does not make e.g. ext2fs provide a diskfs_get_source, does it? ## IRC, freenode, #hurd, 2013-08-29 youpi: that is correct teythoon: Mmm, I must be missing something then: as such the patch series introduces an RPC, but only EOPNOTSUPP is ever returned in all cases for now? ah /* Guess based on the last argument. */ since ext2fs & such report their options with store last, it seems ok indeed it still seems a bit lame not to return that information in get_source yes well, if it had been just for me, I would not have created that rpc, but only guessing was frowned uppon iirc then again, maybe this should be used and then the mtab translator could skip any translators that do not provide this information to filter out non-"filesystem" translators guessing is usually trap-prone, yes if it is to be used by mtab, then maybe it should be documented as being used by mtab otherwise symlink would set a source, for instance while we don't really want it here why would the symlink translator answer to such requests? it is not a filesystem-like translator no, but the name & documentation of the RPC doesn't tell it's only for filesystem-like translators well, the documentation does say "filesystem" but it does not clearly specify that one shouldn't implement get_source if one is not a filesystme "If the concept of a source is applicable" works for a symlink that could be the same for eth-filter, etc. right Mmm, that said it's fsys.defs not io.defs teythoon: it is the fact that we get EOPNOTSUPP (i.e. fsys interface supported, just not that call), and not MIG_BAD_ID (i.e. fsys interface not supported), that filters out symlink & such, right? that's what I was thinking, but that's based on my interpretation of EOPNOPSUPP of course ;) teythoon: I believe that for whatever is a bit questionable, even if you put yourself on the side that people will probably agree on, the discussion will still take place so we make sure it's the right side :) (re: start/end_code) I'm not sure I follow youpi: /proc/pid/stat seems to be used a lot: http://codesearch.debian.net/search?q=%22%2Fproc%2F.*%2Fstat%22 that does not mean that start/endcode is used, but still it seems like a good thing to mimic Linux closely stat is used a lot for cpu usage for instance, yes start/endcode, I really wonder who is using it using it for kernel thread detection looks weird to me :) (questionable): I mean that even if you take the time to put yourself on the side that people will probably agree on, the discussion will happen it has to happen so people know they agree on it I've seen that a lot in various projects (not only CS-related) ok, I think I got it it's to document the reasons for (not) doing something? something like this, yes even if you look right, people will try to poke holes just to make sure :) btw, I think it's rather unusual that our storeio experiments would produce such different results you're right about the block device, no idea why I got a character file there I used settrans -ca /tmp/hello.unzipped /hurd/storeio -T gunzip:file /tmp/hello also I tried stacking the translator on /tmp/hello directly, from what I've gathered that should be possible, but I failed ftr I use the exec server with all my patches, so the unzipping code has been removed from it ah, I probably still have it it shouldn't matter here, though I agree how would you stack it? I've never had a look at that I'm not sure attaching the translator to the node is done before or after the translator has a change to open its target right but it could be done, if storeio used the reference to the underlying node, no? yes btw, you had said at some point that you had issues with running remap. Was the issue what you fixed with your patches? * youpi realizes that he should have shown the remap.c source code during his presentation well, I tried to remap /servers/exec (iirc) and that failed then again, I recently played with remap and all seemed fine but I'm sure it has nothing to do with my patches ok those I came up with investigating fakeroot-hurd and I saw that this also aplies to remap.sh *while yep, they're basically the same [[open_issues/virtualization/remap_root_translator]], [[open_issues/virtualization/fakeroot]]. ## IRC, freenode, #hurd, 2013-09-06 teythoon: do you personally prefer /proc being able to implement /proc/self on its own, or using the magic server to tell clients to resolve those specific cases themselves ? imho solving the "who's the sender of an rpc" could solve both the SCM_CREDS implementation and the self case in procfs [[open_issues/SENDMSG_SCM_CREDS]], [[hurd/translator/procfs/jkoenig/discussion]], *`/proc/self`*. pinotree: yes but that would require servers impersonating users to some extent and this seems against the hurd philosophy and there was also the fact that you could create a fake/different port when sending an rpc to fake what ? the sender identiy *identity what ? you mean intermediate servers can do that braunr: I don't know if I understand all the implications of your question, but the magic server is the only hurd server that actually implements fsys_forward (afaics), so why not use that? teythoon: my question was rather about the principle do people find it acceptable to entrust a server with their authority or not on the hurd, it's clearly wrong but then it means you need special cases everywhere, usually handled by glibc and that's something i find wrong too it restricts extensibility the user can always change its libc at runtime, but in practice, it's harder to perform than simply doing it in the server braunr: then I think I didn't get the question at all teythoon: it's kind of the same issue that you had with the mtab translator about showing or not some entries the user normally doesn't have access to this problem occurs when there is more than one server on the execution path and the servers beyond the first one need credentials to reply something meaningful the /proc/self case is a perfect one (conceptually, it's client -> procfs -> symlink) 1/ procfs tells the client it needs to handle this specially, which is what the hurd does with magic 2/ procfs assumes the identity of the client and the symlink translator can act as expected because of that teythoon: what way do you find better ? braunr: by "procfs assumes the identity" you mean procfs impersonating the user? yes braunr: tbh I still do not see how this can be implemented at all b/c the /proc/self symlink is not about identity (which can be derived from the peropen struct initially created by fsys_getroot) but the pid of the callee (which afaics is nowhere to be found) s/callee/caller/ the one doing the rpc impersonating the user isn't only about identity actually, it's impersonating the client yes, client is the term >,< so basically, asking proc about the properties of the process being impersonated proc o_O it's not hard, it's just a big turn in the way the system would function teythoon: ? you lost me somewhere the client is the process not the user in order to implement /proc/self properly, one has to get the process id of the process doing the /proc/self lookup, right? yes actually, we would even slice it more and have the client be a thread so how do you get to that piece of information at all? the server inherits a special port designating the client, which allows it to query proc about its properties, and assume it's identity in servers such as auth its* ah, but that kind of functionality isn't there at the moment, is it? it's not, by design right, hence my confusion instead, servers use the magic translator to send a "retry with special handling" message to clients right, so the procfs could bounce that back to the libc handler that of course knows its pid yes right, so now at last I got the whole question :) :) ugh, I just found the FS_RETRY_MAGICAL handler in the libc :-/ [[interface/dir_lookup]]. ? why "ugh" ? well, I'm inclined to think this is the bad kind of magic ;) do i need to look at the code to understand ? ok, so I think option 1/ is easily implemented, option 2/ has consequences that I cannot fully comprehend same for me no, but you yourself said that you do not like that kind of logic being implemented in the libc well easily i'm not so sure it's easy to code, but i assume checking for magic replies has its cost why not? the code is doing a big switch over the retryname supplied by the server we could stuff getpid() logic in there 14:50 < braunr> it's easy to code, but i assume checking for magic replies has its cost what kind of cost? computational cost? yes the big switch you mentioned run every time a client gets a reply (unless i'm mistaken) a only for RETRY_MAGICAL replies but you need to test for it switch (retryname[0]) { case '/': ... that should compile to a jump table, so the cost of adding another case should be minimal, no? yes but it's even less than that the real cost is checking for RETRY_MAGICAL 14:55 < teythoon> a only for RETRY_MAGICAL replies so it's basically a if one if, right ? no, it's switch'ing over doretry you should pull up the code and see for yourself. it's in hurd/lookup-retry.c ok well no, that's not what i'm looking for it's not o_O i'm looking for what triggers the call to lookup_retry teythoon: hm ok, it's for lookups only, that's decent teythoon: 1/ has the least security implications yes it could slightly be improved with e.g. a well defined interface so a user could preload a library to extend it extend the whole magic lookup thing? yes but that is no immediate concern, you are trying to fix /proc/self, right? no, i'm thinking about the big picture for x15/propel, keeping the current design or doing something else oh, okay solving /proc/self looks actually very easy well, I'd say this depends a lot on your trust model then do you consider servers trusted? (btw, will there be mutual authentication of clients/servers in propel?) there were very interesting discussions about that during the l4hurd project iirc, shapiro insisted that using a server without trusting it (and there were specific terminology about trusting/relying/etc..) is nonsense teythoon: i haven't thought too much about that yet, for now it's supposed to be similar to what the hurd does hm, then again trust is not an on/off thing imho ? trusting someone to impersonate yourself is a very high level of trust s/is/requires/ the mobile code paper suggests that mutual authentication might be a good thing, and I tend to agree i'll have to read that again teythoon: for now (well, when i have time to work on it again .. :)) i'm focusing on the low level stuff, in a way that won't disturb such high level features teythoon: have you found something related to a thread-specific port in the proc server ? hurd/process.defs:297: /* You are not expected to understand this. */ \o/ braunr: no, why would I (the thread related question) braunr: yes, that comment also cought my eye :/ teythoon: because you read a lot of the proc code lately so maybe your view of it is better detailed than mine ## IRC, freenode, #hurd, 2013-09-13 * youpi crosses fingers yay, still boots teythoon: I'm getting a few spurious entries in /proc/mounts none /servers/socket/26 /hurd/pfinet interface=/dev/eth0, etc. /dev/ttyp0 /dev/ttyp0 /hurd/term name,/dev/ptyp0,type,pty-master 0 0 /dev/sd1 /dev/cons ext2fs writable,no-atime,no-inherit-dir-group,store-type=typed 0 0 fortunately mount drops most of them but not /dev/cons spurious entries in df are getting more and more common on linux too anyway... ah, after a console restart, I don't have it any more I'm getting df: `/dev/cons': Operation not supported instead ## IRC, freenode, #hurd, 2013-09-16 teythoon: e2fsck does not seem to be seeing that a given filesystem is mounted /dev/sd0s1 on /boot type ext2 (rw,no-inherit-dir-group) and still # e2fsck -C 0 /dev/sd0s1 e2fsck 1.42.8 (20-Jun-2013) /dev/sd0s1 was not cleanly unmounted, check forced. (yes, both /etc/mtab and /run/mtab point to /proc/mounts) Yes, that is a "known" problem. tschwinge: no, it's supposed to be fixed by the mtab translator :) youpi: glibc's paths.h points to /var/run/mtab (for us) youpi: Oh. But this is by means of mtab presence, and not by proper locking? (Which is at least something, of course!) /var/run points to /run tschwinge: yes anyway, got to run ## Memory Leak Fixed in 2013-09-28 Hurd commit a81c0c28ea606b0d0a2ad5eeb74071c746b7cdeb (libdiskfs), and 2013-10-04 Hurd commit 98b6f846b628e858acbae9258bac78cf54126d27 (libnetfs). ### IRC, freenode, #hurd, 2013-09-18 ext2fs is using a ginormous amount of memory on darnassus since i last updated the hurd package :/ i wonder if my ext2fs large store patches rework have introduced a regression the order of magnitude here is around 1.5G virtual space :/ it used to take up to 3 times less before that looks like my patches didn't make it into the latest hurd package teythoon: looks like there definitely is a new leak in ext2fs :/ memory only the number of ports looks stable relative to file system usage braunr: I tested my patches on my development machine, it's up for 14 days (yay libvirt :) and never encountered problems like this i've been building glibc to reach that state hm, that's a heavy load indeed could be the file name tracking stuff, I tried to make sure that everything is freed, but I might have missed something teythoon: simply running htop run shows a slight, regular increase in physical memory usage in ext2fs old procfs stikes again? :) braunr: I see that as well... curious... 16:46 < teythoon> could be the file name tracking stuff, I tried to make sure that everything is freed, but I might have missed something how knows, maybe completely unrelated the tracking patch isn't that big, I've gone over it twice today and it still seems reasonable to me hm ### IRC, freenode, #hurd, 2013-09-25 seems like a small leak per file access but htop makes it obvious because it makes lots of them shouldn't be too hard to find since it might also come from the large store patch, i'll take a look at it ### IRC, freenode, #hurd, 2013-09-27 teythoon: found the leak :) although its origin is weird braunr: where is it? i'm still building packages to make sure that's it see https://git.sceen.net/hurd/hurd.git/plain/libdiskfs/dir-lookup.c which you changed in https://git.sceen.net/hurd/hurd.git/commit/?id=06d49cdadd9e96361f3fe49b9c940b88bb869284 line 306 is "return error" instead of "goto out" has been so since 1994 what is unclear is why this code path is now run patch is here: http://darnassus.sceen.net/~rbraun/0001-Fix-memory-leak-in-libdiskfs.patch I see, weird indeed teythoon: the system also feels slower somehow such errors might have introduced unexpected retries i think it's possible to write a coccinelle patch to find such errors ### IRC, freenode, #hurd, 2013-09-28 braunr: bah, I havent noticed the leak on my box, even after building eglibc & hurd several times that's weird are you sure it's up to date ? also, is procfs correctly attached to /proc ? that's what seems to trigger it yes, 20130924-2, with procfs on /proc braunr: that turned out to be the leak indeed? and somehow my changes triggered it? did you discover why? teythoon: yes, yes, no but youpi didn't see the leak on his system ^^ cool that you found it I did oh yes you mean you saw the leak yes ### IRC, freenode, #hurd, 2013-10-01 the fix i did in libdiskfs might have fixed other issues apparently, it's the code path taken when error isn't ENOENT, including no error (translator started) the memory leak fix, you mean? yes it might haved fixed reference counting too although i'm not sure if we actually ever run into that issue in the past the weird thing is, that path is taken when starting a passive translator (i think) (it might be any kind of translator, and just doing nothing if alcready active) already* anyway, the fact that the leak was so visible means this code was run very often which doesn't make sense hm ok, it seems that code was run every time actually but the leak became visible when it concerned memory which side-effects did the old code produce? teythoon added a dynamically allocated path that wasn't freed reference leaks which might explain the assertion on reference we sometimes see with ext2fs when a counter overflows and becomes 0 [[open_issues/ext2fs_libports_reference_counting_assertion]]? hmm which is why i'm mentioning it :) i'll try to reproduce the assertion pinotree: actually, after a more in-depth look, reference counting looks valid before the fix too ok, thanks for checking pinotree: the assertion affects the root translator, and is triggered by a test that stresses memory memory as in ram, or as in disk storage? malloc ok i suspect the code doesn't handle memory failure well ### IRC, freenode, #hurd, 2013-10-02 braunr: btw, did you fix the leak? yes https://git.sceen.net/hurd/hurd.git/commit/?id=a81c0c28ea606b0d0a2ad5eeb74071c746b7cdeb 1h after tagging 0.5 ( :( ah yes, I've seen that commit I just wanted to know whether this settled the issue it does :) good i still can't figure out why youpi didn't had it the code path is run when no error (actually error != ENOENT) which explains why the leak was so visible so my patch exposed this b/c of the allocation I added, makes sense it's funny actually, b/c this wasn't an issue for me as well, I had my development vm running on that patches for two weeks ### IRC, freenode, #hurd, 2013-10-02 libnetfs suffers from the same leak as libdiskfs when looking up a translator i'll fix it too ## Multiple mtab Translators Spawned Fixed in 2013-10-05 procfs commit c87688a05a8b49479ee10127470cc60acebead4a? ### IRC, freenode, #hurd, 2013-09-20 teythoon: how come i see three mtab translators running ? 6 now oO looks like df -h spawns a few every time yes, weird... accessing /proc/mounts does actually teythoon: more bug fixing for you :) ### IRC, freenode, #hurd, 2013-09-23 so it might be a problem with either libnetfs (which afaics has never supported passive translator records before) or procfs, but tbh I haven't investigated this yet [[open_issues/libnetfs_passive_translators]]. ### [[!debbug 724868]] ### IRC, freenode, #hurd, 2013-10-03 i can't manage to find out where the hurd stores information about active translators ... there is this transbox per node but where are nodes stored ? what if they are are dropped ? braunr: iirc, see libfshelp well i have i still can't find it i fear that it works for ext2fs because that particular translator implements a cache of open nodes whereas things like procfs drop and recreate nodes per open which would be the root cause for the multiple mtab bug doesn't tmpfs support translators? good idea although it's still a libdiskfs based one no problem for tmpfs, so it would be a netfs/procfs issue better than what i feared :) now, how is libdiskfs able to find active translators .. ah, there is a name cache in libdiskfs .. nope, looks fine ### IRC, freenode, #hurd, 2013-10-04 nodes with a translator seem to keep a reference in libdiskfs and not in libnetfs mhmmpf oh great .. each libdiskfs that "works" seems to implement its own diskfs_cached_lookup function so both ext2fs and tmpfs actually maintain a list of nodes, keeping a reference on those with a translator while procfs simply doesn't teythoon: ^ *sigh* braunr: ok, thanks, I'll look into that i'm not sure how to fix it we can either fix node destruction to cleanly shut down translators but this would mean starting mtab on each access or we could implement a custom cache in procfs or perhaps a very custom change in the lookup callback for mounts i'll try the latter err, shouldn't we try to fix this in lib*fs? unless you really want to work on it i dont' know ah, so the node is destroyed but the translator is kept running? that's what you mean by the above? and ext2fs makes an effort of killing it in its node cleanup code? yes grmbl, i'm lagging a lot i'm not sure ext2fs maintains it with ext2fs, translators can only be explicitely removed i mean, ext2fs keeps all node descriptors alive once accessed while procfs doesn't teythoon: ok, looks like i have a working patch that merely caches the node for mounts i installed my fixed procfs on darnassus, only one mtab :) nice :) i have a fix for the multiple mtab issue, will send a patch tonight ## `/home` Missing ### IRC, freenode, #hurd, 2013-10-04 now, why is there no /home in df output ? not sure note how /dev/tty* end up in /proc/mounts, those are passive translators too, no? yes but that's a good thing i guess or was mounts intended for file systems only ? well, in the unix traditional meaning I think its nice too, yes but why are they fine and your /home is not... that's weirder also, mounts actually doesn't show passive translators teythoon: does your code perform any kind of comparison ? i see /servers/socket/26 but not /servers/socket/2 s/comparison/filter/g hmm well, yes, try /hurd/mtab --insecure / (I cannot connect to darnassus from here...) ok but that looks unrelated both /servers/socket/26 and /servers/socket/2 refer to the same translator i was wondering if mtab was filtering similar entries based on that no that's weird too then, isn't it ? yes ;) ok btw, how is that done with the same traanslator being bound to two nodes? settrans cannot do that, can it? no it can't the translator does it when started ah (which means there is a race if both are started simulatneously, although it's very rare and not hard to solve) a weird beaving translator then :) teythoon: if ext2fs is set active, mtab output reports it teythoon: looks like this bug is what allows mtab not to deadlock teythoon: when i attach it as an active translator, cat freezes teythoon: if (control && control->pi.port_right == fsys) that's the filtering i was previously talking about oh please don't name global variables "path" ... ### IRC, freenode, #hurd, 2013-10-06 teythoon: pty-s also bind to two nodes, not only pfinet ### IRC, freenode, #hurd, 2013-10-07 teythoon: please tell us when you're available, we need to work out the last mtab issues braunr: I'm available now :) I'm sorry, I've been very busy the last two weeks, but I've plenty of time now great :) did you see youpi's mail ? i have the exact same question I did it seems your code registers active translators but parent translators don't seem to register them when they're created from passive translators or am i mistaken ? I'll need a moment to get my hurd machine and myself up to speed... braunr: I concur with youpi, hooking into fshelp_fetch_root should do just fine I'll just try that ok how do you deal with mtab reporting itself ? o_O does it do that? no, but it should when i set it as an active translator, i get a deadlock hm teythoon: before you change libfshelp, i'd like you to try something else use more appropriate names for global variables in mtab.c in particular, the variable path clashes with local names noted teythoon: as a side note (i'm not asking to rewrite anything) i strongly recommend a very explicit object oriented style of coding (or data-oriented as it's sometimes called) use prefixes for all your interfaces so they can be made public if needed (which acts as a namespace and avoids lots of collisions naturally) use "constructors" and "destructors" (functions that both allocate and initialize) this helps avoiding leaks a lot too hm, I thought I did that, could you be more specific? ok didn't see the comment /* XXX split up */ error_t mtab_populate (... :) as a better example, see your code in libfshelp/translator-list.c struct translator should have been treated as an object this would probably have completely avoided any leaks in the first place braunr: right, I deviated from that style there teythoon: these are minor details, don't mind them too much, i just find it helps me a lot braunr: sure, I appreciate the feedback :) ### IRC, freenode, #hurd, 2013-10-08 braunr: I'm on to the passive translator not getting registered issue however, removing them from the list if the active translator is killed does not work as expected... I still need to fiddle with the notifications to get this right ok ### IRC, freenode, #hurd, 2013-10-16 braunr: btw, I fixed the 'passive translator not showing up in proc/mounts'-issue but 4 ports do leak each time a translator is killed and reinstalled this happens with passive ones as well as active ones teythoon: is that issue tied to your changed ? changes* I'm not sure tbh, testing that is on my list of things to do ok first thing to know i guess yes ### IRC, freenode, #hurd, 2013-11-06 by the way, did you fix mtab for passive translators ? or make any progress ? I just cleaned up the patch series ah good I'm still trying to decide whether I leak any ports ### IRC, freenode, #hurd, 2013-11-15 btw, I haven't forgotten about the passive translator not showing up in /proc/mounts I have a patch series, the first patch seems fine, but if I build hurd packages with the other two (those that actually hook into dir-lookup), strange things happen on the first boot, everything is fine, passive translators showing up in /proc/mounts, nice but when I reboot, the system kind-of comes back, but something is very wrong with many (passive?) translators the system never recovers, I have no idea whats going on there ok push that work in a branch somewhere for review please right, thanks braunr: http://darnassus.sceen.net/cgit/teythoon/hurd.git/shortlog/refs/heads/feature-translatorslist-detect-passive-translators ### IRC, freenode, #hurd, 2014-01-04 teythoon: did you eventually get any idea about why /proc/mounts is missing mounts? e.g. I have /boot as a separate partition, it doesn't show up ### IRC, freenode, #hurd, 2014-01-05 youpi: yes, passive translators are not currently handled youpi: i have patches for that, but they produce weird results, braunr promised to take a look teythoon: hum ^^ i thought they were pending review ! where are they again ? http://darnassus.sceen.net/cgit/teythoon/hurd.git/shortlog/refs/heads/feature-translatorslist-detect-passive-translators ok they are reasonably straight-forward but cause this funny issue, after the first reboot with the patched hurd everything is fine after the second reboot, everything is weird and broken badly right interesting :) ### IRC, freenode, #hurd, 2014-01-06 teythoon: is it normal that, if ext2fs is started as an active translator for /home, and then for another directory inside my home, mtab only reports / and /home and not this third file system ? (with the hurd master version) (the translator for /home is run by root, but the one inside my home is started with my uid) (and every component on the path is readable/crossable) ### IRC, freenode, #hurd, 2014-01-07 braunr: well yes, the mtab tool/translator does not follow translators bound to nodes not owned by root braunr: try /hurd/mtab --insecure /home ok good thinking braunr: did you encounter any problems after the second reboot ? i'm not yet there work still making me busy and i'm trying to isolate the problem first that is, restrict tests to a single leaf translator you reproduced the weirdness ? teythoon: attempting teythoon: first i'm trying to check a few things teythoon: i'm running my leaf translator as root now active and i still don't see it :/ one of the components in the path is not own by root is that a problem ? no, but maybe the mtab translator should check for that... does it check for the node ownership or the process rights ? credentials, rather node ownership ah, ok ok i see it now oh, also, it looks like settrans -g on an active translator doesn't work, i get ebusy all the time :/ oh ? that could explain the fs corruption i saw did i break that ? i don't know maybe i think it was possible in the past teythoon: when isolated, your code works fine i'll try applying it globally btw, how did you do that ? through debian packages ? i'm trying to get to the point, but i'm still not there with our uber-buildbot setup i picture myself pushing to a git repo, wait a little and get the packages from my deb repo... for now, i have my hurdtest tool that i used during gsoc with that i can just drop files into a directory and have that overlayed ontop of my test image hum so you replaced the lib*fs libraries directly ? and the static rootfs ? yes * teythoon checks hm, maybe i just missed a file, not sure anymore teythoon: with the new libraries, df -h doesn't see passive translators :/ braunr: see /proc/mounts please same or /hurd/mtab / weird no :/ b/c when i developed it, i used a test suite to check that every combination of tmpfs/ext2fs active/passive and every way to get rid of any translator produced the desired results i'll look into this teythoon: when i remove the active translator i set on /home, i get /dev/hd0s1 8.1G 2.0G 5.8G 26% /home hd0s1 being used for / :/ this does need reviewing this is how i expect the system to react currently oh w/o these patches hm ok well i'm currently using them including the root fs ? yes hmpf i have to mention that i merged master into it i did the same currently compiling... i only changed libdiskfs, libnetfs and libfshelp is there something else that should be changed ? (i.e. because of inlining ?) i guess i should rebuild a hurd package just to be sure braunr: isn't the translator for / statically linked ? if so it needs to be rebuilt (sorry if I'm saying silly things by popping out of nowhere) yes i took care of that teythoon: i've built a hurd package with the three patches and i don't see passive translators at all :/ well actually i see a lot of the them just not /home braunr: please see /home/teythoon/build/hurd-upstream/test.{bash,log} the latter is a log i just created on darnassus it shows no failures do you have another translator that is started from a passive translator record ? besides /home, if so, does that show up ? well, as i just said, i can actually see many of them weird http://darnassus.sceen.net/~rbraun/mounts hm let's try without --sync= ah, now i can see it can you give me the command line you used before for the /home translator /hurd/ext2fs --sync=30 /dev/hd0s6 but hm i can only see /home when the passive translator is started is that intentional ? yes ok and it doesn't work with --sync=30 o_O so, actually, mtab doesn't report passive translators no it reports active ones only, whether they're started manually or from passive translators yes ok that's good enough reporting passive translators wouldprobably require a complete scan yes, that was deemed not feasible right i can't reproduce any weirdness good it seems to just work well except this option parsing problem thanks for looking into this yeah sur e thanks for reminding me :) the actual fix is implementing fsys_get_source i forgot that "pending review" was my review here he which should actually be file_get_source teythoon: why does mtab report errors for /proc/swaps and /xconsole ? not sure btw, i build hurd packages with my patches and it reliably wreaks havoc on my test vms ... now that's really weird are you certain everything has been cleaned since your manual replacement of the libraries ? yes, i use mainly throwaway-vms for such experiments ok did you include the debian patches ? yes so did i i based my work on my own packages with thread destruction i'll redo it from the sid ones but before, i guess i should share mine with you so you can test them in your vms we may simply have different configurations yes something we might have missed just like the --sync parameter ### IRC, freenode, #hurd, 2014-01-08 teythoon: hello see http://darnassus.sceen.net/~rbraun/teythoon/ for the custom hurd packages they need thread destruction so get the latest gnumach package from unstable, and the libc packages from my repository first ### IRC, freenode, #hurd, 2014-01-09 braunr: your packages indeed seem to work and with mine i encounter a different problem, the proc server crashes *very* early this is w/o a rebuilt libc different as in not the weird one i remember having back when i wrote those patches hum you're using all my glibc/hurd packages right ? your packages work fine i wonder if your bug is caused by 2c9422595f41635e2f4f7ef1afb7eece9001feae mine don't (or rather, not having it) look at the patches i've added it's included, and i remember i really needed it althouhg it was just about a leak hm i don't know :/ but i strongly suspect your patches are ok and something else is wrong why would my packages miss that patch ? hum are you testing with your packages built from upstream sources ? always upstream hurd against debian glibc ? yes check the patches again what patches ? the debian ones that you don't include another thing you can do is err get the latest debian hurd package add your patch only rebuild i use hurd upstream + all the debian stuff that's weird but hurd upstream has received quite a lot of patches please try from debian hurd + your patch only braunr: i'm afraid it might be your patch 3a3fcc81 that breaks proc [[open_issues/libpthread_set_stack_size]]. teythoon: but anyway, it does look like your patches are actually fine yes i'll install my packages on darnassus and test a bit more there is an issue however ah grep sd2s1 /proc/mounts /dev/sd2s1 /dev/sd2s1 /hurd/storeio writable 0 0 that makes fsck think that /dev/sd2s1 is mounted hmpf which makes debians fsck magic (when using sysvinit) drop to a root shell at boot time why does it report a mount point ? or even a device why not none /dev/sd2s1 ? b/c of the heuristic it uses and i know, you told me it's a bad idea i did ? probably hm we said so many things i don't remember but that doesn't look too hard to fix well, i'll just have to make translators provide meaningful get_source responses and get rid of the heuristic ok so if you use passive translators for the mounts, and not /etc/fstab, you should be fine my "traditional" hurd systems are teythoon: i'll wait a bit before deplying it on darnassus then ### IRC, freenode, #hurd, 2014-01-13 teythoon: does your latest patch series take care of --sync ? yeah, i finally got why the hurd would react strangely after an reboot, it was umount --all removing vital passive translator records i never had any issues with --sync ah, umount -a hm do you recall i did :) ? umount -a was only run with the sysvinit scripts, that's why you didn't see that issue, only me yes, i do yes, i see however, i'm also using --sync 30 on my fs ok so you couldn't reproduce that particular issue hum try --sync=30 showtrans /media/scratch /hurd/ext2fs --sync=30 /dev/sd1s1 works fine now that's weird :) then again, previously there was another bug but then the patches i've tested are not the complete series you sent in the dir lookup function, if a passive translator is started say, if i first access /proc/foo, /proc/foo would be recorded, not /proc i fixed that yesterday ok maybe it was b/c of that but hm why would /proc be recorded ? now ? i mean, /proc should be recorded at /, and /foo at /proc right ? yes and that wasn't the case ? but what happened was that proc/foo was recorded as a child translator for / ohh yes i see well, i'm not sure it's that bug, since the translator involved was on /home same problem it's unlikely that the translator was started b/c of a /home lookup much more likely that the first lookup is /home/someone yes but mtab correctly reported /home without --sync, and not when the option was present and that part doesn't quite make sense to me how did you trigger the startup ? ssh i believe hm anyways, i could not reproduce this issue do you have packages somewhere i can test ? yes oh and btw http://darnassus.sceen.net/~teythoon/hurd-ci/ something you could do to deal with umount -a but i guess that's what you did already is to only shut the active translator down https://git.sceen.net/hurd/hurd.git/commit/?id=0033d20449b3bb558f2ea470983018db39b572aa yeah, i thought about that but i believe that is not the right hting to do yes i know but i'm not sure that's the right approach hm :) b/c if on linux you do umount /foo && ls /foo, foo will be empty yours is probably more posix-friendly if the passive translator lookup is left, /foo will be restarted ok s/lookup/record/ that's one reason i'm not very fond of passive translators tbh yep i'd reserve them as a user-oriented hurd-specific feature anything that must behave as mount/umount expects has to be active anyway ok let's give a quick shot at your packages :) teythoon: works fine :) mtab still reports console entries though is tha texpected ? braunr: kind of braunr: /bin/console is a netfs-based translator, probably for multiplexing, dunno exactly i see ### IRC, freenode, #hurd, 2014-01-14 teythoon: passive translators do work fine in mtab now, thanks :) indeed :) ### IRC, freenode, #hurd, 2014-02-11 another topic: what's the rationale again behind umount removing passive translators?... antrik: umount is for compatibility with unixoid systems consider umount /foo; ls /foo if umount would leave the passive translator record on /foo, /foo would be started again but mount never creates passive translators, right? so why would umount remove them? they are none of its business... sure, you can see it this way still, i like the way it is now, hence i implemented it this way ;) teythoon: but then umount -a unmounts all passive translators include ~joe/http:/ s/include/including/ I tend to agree with antrik i won't oppose a change of course and yes, we have seen problems b/c of that. otoh those can be fixed (and they are, i just sent a patch fixing that) teythoon: well, it's not only about http:, joe user may want to mount its own iso image, or whatever thats true no it's not /proc/mounts does not contain translators bound to nodes that do not belong to root but sure, we can change umount i agree active translators can be viewed as unix mounts passive translators are an entirely hurdish feature but then, servers such as pflocal and pfinet shouldn't probably not be passive translators braunr: shouldn't not? what are you trying to say? :-) woops i'm not trying to make this unconfusingly clearer :p pflocal and pfinet should probably be active translators why? hum wait i had a reason weeks ago but now it looks the opposite is better actually :p so that they don't appear in mounts but aiui, there is another property that is tested to make translators appear in mounts hm... I know this question has been discussed when first talking about an mtab translator years ago... but don't remember whether there was any conclusion I think one of the ideas was that translators would opt in for being considered as mounts... it makes sense to only have file systems in mounts anyway instead of going by the translator type, another option might be ignoring anything that is backed by a passive translator?... or have a startup option (perhaps with some "smart" defaults) to request a translator to manifest as a mount or not so many ideas... ;-) ## coreutils' `df` ### IRC, freenode, #hurd, 2014-01-24 gnu_srs: "df: Warning: cannot read table of mounted file systems: No such file or directory" you should be able to fix that easily ### IRC, OFTC, #debian-hurd, 2014-02-03 /run/mtab also seem to be missing. df do not work. that's a libc bug ### IRC, OFTC, #debian-hurd, 2014-02-05 i had to ln -s /proc/mounts /var/run/mtab to make df work, what's the proper fix? (here sysvinit+openrc) gg0: the proper fix for df is to fix the coreutils build I checked the mtab problem, and /etc/hurd/runsystem.gnu via /etc/hurd/rc fixes the symlink, while runsystem.sysv do not. I suspect /etc/init.d/checkroot.sh should fix the symlink too. well, atm df looks at /var/run/mtab instead of /etc/mtab only because it hasn't been rebuilt against a recent glibc, that's all but we can brownpaperbag fix it, yes right. so perhaps that is the bug to fix? yes, it is it depends on coreutils actually building youpi: i thought the proper way to fix the /var/run/mtab issue is to patch the libc ? it is the libc defines some macro, on linux it expands to /etc/mtab, on hurd to /var/run/mtab but you need to rebuild coreutils to get a fixed df ok, right should it be /var/run/mtab or /etc/mtab on hurd? the former seem more correct, but /run/mtab give more sense given that it should be available also before /var/ is mounted. to be honest, I don't really care and I thus tend to agree on sticking with what linux does to avoid issues as in: keep debian working mostly the same on all kernels, to avoid issue s well, linux really should move that file away from /etc/ too. :) pere: ok, but let's move at the same time rather than hitting bugs ourselves perhaps df should use /proc/mount instead, and get rid of the problem completely... that can be a way, too I believe is a good fix for the mtab problem. WRT /etc/mtab the main reason for keeping it is solely for compatibility; there's no reason not to use /proc/mounts directly (or /run/mtab if we want a kernel-agnostic location). Whether it's worth doing something like that is debatable. The main issue with doing stuff like this nowadays is that you get shouted at by all the systemd people for making changes... ## Memory Leak in `translator_ihash_cleanup` ### IRC, freenode, #hurd, 2013-10-04 teythoon: isn't there a leak in translator_ihash_cleanup ? braunr: looks like, yes braunr: I probably forgot to add the free (element->name) when I added the name field teythoon: ok teythoon: i let you fix that :p braunr: sure ;) ## `chroot` ### IRC, OFTC, #debian-hurd, 2014-03-05 In context of [[open_issues/nightly_builds_deb_packages]]. installer gets stuck running os-prober, seems because /target/proc/mounts gets unreadable, sometimes Resource lost sometimes it gets stuck reading it there are two proc translators: one /proc, one /target/proc anything known that could have broken /target one? anything else to check? what's unusual is that cat'ing /proc takes few seconds before giving output. does initrd slow down things? */proc/mounts what does "Resource lost" mean? i get it since the beginning of install ### IRC, freenode, #hurd, 2014-03-05 debian installer mounts installation disk under /target then probably it debootstraps on it we have 2 procfs: one outside, one inside /target outside it works fine, inside cat'ing /proc/mounts it gives Resource lost the first time. the second one it gets stuck oh there's only one /hurd/mtab though. inside one might have bought the farm, correct? ? two /proc/mounts means you have to have two mtab translators afaiu i can't reproduce on a clean chroot gg0: two mtab instances should be fine also, the one under /target should only list translators "under "under" /target teythoon: yeah i restart the dead one but at first access i get "Resource lost" again and mtab dies hm wait "Resource lost" means mtab is not running are you sure ? the mtab translator queries other translators that can lead to the delay you described and severe enough errors are handed back to the client, e.g. if mtab itself gets resource lost on the very first translator it queries i'd like to clean up chroot proc and mtab stuff but settrans -fg gets stuck or it gives settrans: /proc/mounts: Computer bought the farm settrans: /proc/mounts: Operation not supported gg0: try settrans --recursive ... then again, i never used this and it should work without, no ? same behavior root@hurd01:~# settrans --recursive -fg sid-chroot/proc/mounts settrans: sid-chroot/proc/mounts: Computer bought the farm root@hurd01:~# settrans --recursive -fg sid-chroot/proc/mounts settrans: sid-chroot/proc/mounts: Operation not supported root@hurd01:~# settrans --recursive -fg sid-chroot/proc/mounts hm (last got stuck) eh, i thought of settrans --recursive -fg sid-chroot i doesn't work. it gives no output but showtrans shows translators didn't go away showtrans shows passive translator records ah better use fsysopts to query the active one both still active, even with fsysopts hm i recently improved the mtab translator, but that change has not made its way into debian yet * gg0 adding hurd-ci [[open_issues/nightly_builds_deb_packages]]. btw Resource lost/mtab dying i can't reproduce on hurd01 seems breaking debian installer teythoon: dist-upgraded hurd-ci, it behaves pretty same way besides settrans -fg sid-chroot/proc/mounts gives "Operation not supported" right away, without "Computer..farm" first following attempts it simply gets stuck time to quickdraw gdb :p teythoon: i think easy to reproduce, just need to debootstrap anything stronger than -fg? kill well mtab has been already manually killed i could kill procfs too yes, but then it seems to me that the translator linking in netfs is buggy kill -KILL :p :D aka double kill ok by killing chroot procfs then accessing sid-chroot/proc/mounts makes both restarting ok, even more likely to be a netfs issue then which is to be expected afaik procfs is the first netfs based translator to support passive translator records it is did the same on debian installer machine: both restarted, cat /target/proc/mounts -> Resource lost and mtab died Computer wanted to buy the farm but it was out of resources ### IRC, OFTC, #debian-hurd, 2014-03-05 btw problem is cat /target/proc/mounts gives Resource lost, what package could cause that? and mtab dies Resource lost often means the same as Computer bought the farm: some translator has died somehow or dropped something procfs+mtab outside /target work well. inside mtab crashes http://postimg.org/image/vgz541z81/ tbh i unmounted a couple, they are not all fs mounted by install, a couple missing could any of them cause mtab crash? see ldd /hurd/mtab it only uses hurd libraries and pthread hurd libs and pthread _outside_ chroot, right? in principle it's the outside /hurd/mtab which gets triggered since the /target ext2fs doesn't know people use it chrooted do you have two processes? (mtab processes I mean) yes two mtabs children of their two procfs k and does setting up an mtab translator by hand gets the same? to restart mtab i have to kill its father procfs tell that to teythoon, not me, he implemented the thing :) then first access to /proc/mounts should make procfs and mtab restart yeah there's already been some troubleshooting on #hurd youpi: i suspect some bugs in netfs translator linking procfs is the first netfs based translator to make use of passive translators you can see dead mtab and os-prober trying to grep /proc/mounts http://postimg.org/image/z4fm981p7/ well, there can't be problems on /target if there's not /target yet