[[!meta copyright="Copyright © 2010, 2013, 2014 Free Software Foundation, Inc."]] [[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable id="license" text="Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled [[GNU Free Documentation License|/fdl]]."]]"""]] [[!tag open_issue_hurd]] # IRC, freenode, #hurd, 2013-02-26 btw, about fakeroot-hurd the remaining issue I see is with argv[0] (yes, again...) ## IRC, freenode, #hurd, 2013-04-03 btw, I believe our fakeroot-hurd is close to working actually it's just a argv[0] issue supposed to be fixed by exec_file_name but apparently not fixed in that case, for some reason ## IRC, freenode, #hurd, 2013-08-26 < teythoon> also I looked into the fakeroot issue, aiui the problem is that scripts are not handled correctly, right? < teythoon> the exec server fails to locate the scripts file name, and so it hands the file_t to the interpreter process and passes /dev/fds/3 as script name < teythoon> afaics that breaks e.g. python < youpi> yes < youpi> pinotree's exec_file_name is supposed to fix that, but for some reason it doesn't work here < pinotree> it was pochu's, not mine < youpi> ah, right < teythoon> ah I see, I was wondering about that < pochu> it was working for a long time, wasn't it? < pochu> and only stopped working recently < youpi> did it completely stop? < youpi> I have indeed seen odd issues < youpi> I haven't actually checked whether it has completely stopped working < youpi> probably worth looking there first < pinotree> gtk+3.0 fails, but other stuff like glib2.0 and gtester-using stuff works < teythoon> huh? I created tests like "#!/bin/sh\necho $0" and that says /dev/fd..., and a python script doing the same doesn't even run, so how can it work for a package build? < youpi> it works for me in plain bash < youpi> #!/bin/sh < youpi> echo $0 < youpi> € $PWD/test.sh < youpi> /home/samy/test.sh < teythoon> it does !? < youpi> yes < youpi> not in fakeroot-hurd however, as we said < teythoon> well, obviously it works when not being run under fakeroot-hurd, yes < youpi> ok, so we weren't talking about the same thing < youpi> a mere shell script doesn't work in fakeroot-hurd indeed < youpi> that's why we still use fakeroot-sysv < teythoon> right < youpi> err, -tcp ## IRC, freenode, #hurd, 2013-11-18 I believe I figured out the argv[0] issue with fakeroot-hurd but I'm not sure how to fix this first of all, Emilios file_exec_file_name patch set works fine but not with fakeroot http://git.sceen.net/hurd/hurd.git/blob/HEAD:/exec/hashexec.c#l300 check_hashexec tries to locate the script file using a heuristic Emilios patch improves the situation with just providing this information but then, the identity port of the "discovered" file is compared with the id port of the script file to verify if the heuristic found the right file but when using fakeroot-hurd, /hurd/fakeroot proxies all requests but the exec server is outside of the /hurd/fakeroot environment, so it gets the id port from the real filesystem we could skip that test if the script name is explicitly provided though that test was meant to see whether a search through $PATH turned up the right file teythoon: nice braunr: thanks :) unfortunately, dpkg-buildpackaging hurd with it still fails for some reason but it is faster than fakeroot-tcp :) even chown ? or chmod ? dunno in detail, but the whole build is faster if you can try it, i'm interested because chown/chmod is also slow on linux with fakeroot-tcp i can try... so it's probably not a hurd bug braunr: yes, it really is no i mean chown/chmod being slow with fakeroot-tcp is probably not a hurd bug but a fakeroot-tcp bug chowning all files in /usr/bin takes 5.930s with fakeroot-hurd (6.09 with startup overhead) vs 26.42s (26.59s) with fakeroot-tcp but try it on linux (fakeroot-tcp i mean) although you may want to do it on something you don't care much about :p) ## IRC, freenode, #hurd, 2013-12-03 * teythoon is gonna hunt a fakeroot bug ... % fakeroot-hurd /bin/sh -c ":> /tmp/some_file" /bin/sh: 1: cannot create /tmp/some_file: Is a directory ah fakeroot-hurd prevents installing stuff with /bin/install sure fakeroot-hurd, why would i work on the slow one ? i don't know because it makes chmod/chown/maybe others horrenddously slow ? yes, fixing this involves fixing fakeroot-hurd are you sure ? i prefer repeating just in case: i saw that problem on linux as well with fakeroot-sysv so ? i'm almost certain it's a pure fakeroot bug, not a hurd bug so even if this is fixed, it still has to pay the socket communication overhead fixing fakeroot-hurd so that i can be used instead of fakeroot-tcp is a very good thing to do, obviously it* but it won't solve the chown/chmod speed (or, probably won't) huh, why not ? 15:53 < braunr> i'm almost certain it's a pure fakeroot bug, not a hurd bug when i say it's slow, i should be more precise it doesn't show up in top yes, but why would fakeroot-hurd suffer from the same issue ? the cpu is almost idle oh right, it's a completely different tool my bad right, right, the proper way to implement fakeroot actually :) yes this will bring near-native speed ## IRC, freenode, #hurd, 2013-12-05 fakeroot-hurd just successfully built mig :) hangs in dh_gencontrol when building gnumach or hurd though i believe it hangs waiting for a lock lock like in file lock that is braunr: no more room for vm_map_find_entry in 80220a40 80220a40 <- is that a task ? or a vm_map, not sure probably a vm_map ## IRC, freenode, #hurd, 2013-12-06 well, aren't threads a source of endless entertainment ... ? well, I found three more bugs in fakeroot-hurd one of them requires fixing the locking used in fakeroot ouch the current code does some lock cycling to aquire a lock out of order cycling ? in the netfs_node_norefs function release and reaquire i see which imho should be better solved with a weak reference working on it, it no longer deadlocks but i broke something else ... endless fun ;) such things could have been done right in the beginning ... yes, I wonder libports has weak references but pflocal is the only user hm none of the lib*fs support that didn't i add one in libdiskfs too ? anyway, irrelevant weak references are a nice feature teythoon: i don't see the cycling you mentioned only netfs_node_refcnt_lock being dropped temporarily yep, that one line 145 note that due to another bug this code is currently never run how surprising .. the note about some leak actually gave a hint about that yeah, that leak I think i'm actually very close it's just so frustrating, i thought i got it last night good luck then thanks :) ## IRC, freenode, #hurd, 2013-12-09 sweet, i fixed fakeroot-hurd :) /clap what was the problem ? lots i see it's amazing it actually run as well as it did mess strikes again i hate messy code .. * teythoon is building half a hurd package using this ... stay tuned ;) teythoon: is this going to make building faster as well? most likely, yes fakeroot-tcp is known to be slow, even on linux teythoon: are you sure about the transparent retry patch ? pretty sure, why ? it's about a more general issue that we didn't fix yet our last discussions about it lead us to agree that clients should check the identity of a server before interacting with it braunr: i don't understand, what's the problem here ? teythoon: fakeroot does the lookup itself, doesn't it ? yes teythoon: but was that also the case before your patch ? braunr: yes teythoon: then ok teythoon: i guess fakeroot handles requests only for a specific set of calls right ? and for others, requests are directly relayed braunr: yes and that still is the case, right ? yes ok looks right since it only affects lookups ok then well, fakeroot-hurd built half a hurd package in less than 70 minutes a new record for my box compared to how much before ? (and why half of it ?) unfortunately it hung after signing the packages... some perl process with a /usr/bin/tee child killing tee made it succeed though braunr: i don't build the udeb package oh ok braunr: compared with ~75 with fakeroot-tcp and my demuxer rework, ~80 before teythoon: nice ## IRC, freenode, #hurd, 2013-12-18 there, i fixed the last fakeroot-hurd bug *whee* :) i thought so many times that i got the last fakeroot bug ... last as in it's in a good enough shape to compile the hurd package that is but now it is :) this will make glibc and others so much faster to build ## IRC, freenode, #hurd, 2013-12-19 teythoon_: hum, you should make the behaviour of fakeroot-hurd on the last client exiting optional y? fakeroot-tcp does the very same thing fakeroot-hurd is different it's part of the file system yes users may want it to stay around and reuse it without checking it's actually there but once the last client is gone, who is ever getting another port to it ? no that cannot happen really ? yes i thought it was like remap since remap is based on it the same thing applies to remap only settrans has the control port hum and uses it once to get a protid for the working dir of the initial process started inside the chrooted environment you may not want to chroot inside so ? then, you get another protid i'll make an example i create a myroot directory implemented by fakeroot populate it leave and do something else, i might want to return to it later ah ok, so you are not using settrans --chroot or maybe i'm confusing the fakeroot translator and fakeroot-hurd 10:48 < braunr> you may not want to chroot inside yes hm ok, so the patch could be changed to check whether the last control port is gone too i have no idea of any practical use, but i don't see a valid reason to make a translator go away just because it has no client except for resource usage and if it's installed as a passive translator although that would make fakeroot loose its state though remap state is on the command line so it would be fine for it see what i mean ? yes i do fakeroot state could be saved in some db one day so it may apply, if anyone feels the need so what about checking for control ports too ? i'm not too familiar with those who has the control port of a passive translator ? the parent ? that should cover the use case you described for the parent translator for fsys_getroot requests it has to keep it around and for more fsys stuff too and if active ? settrans ? who just looses it ? if settrans is used to start an active translator, the parent fs still gets a right to the control port ok i don't have a clear view of what this implies for fakeroot-hurd we'd want fakeroot-hurd to clean all resources including the fakeroot translator on exit for fakeroot-hurd (or any child translator) this means that a port from the control port class will still exists so we do not exit oh, you're speaking of fakeroot.sh ? the wrapper script ? probably for me, fakeroot-hurd is the command line too, similar to fakeroot-sysv and fakeroot-tcp and fakeroot is the translator yes, agreed fakeroot-hurd could use settrans --force --chroot ... to force fakeroot to exit if the main chrooted process dies but that'd kill anything that outlives that process that might be legitimate, say a process daemonized so detecting that noone uses fakeroot is the much cleaner solution ok also, that's what fakeroot-tcp does which is why i suggested an option for that why add an option if we can do the right thing without troubling the user ? ah, if we can, good i think we can I'll rework the patch, thanks for the hint so just to be clear the way you intend it to work is wait for all clients and the control port to drop before shutting down the control port is dropped when dettaching the translator, right ? yes but hm what if clients spawn other processes ? they won't find the translator any more then, that client get's a port to fakeroot at least for it's working dir so another protid is created ah yes, it's usually choorted for such uses chrooted so fakeroot will stick around but clients, even from fakeroot, might simply use absolute paths so ? in which case they won't find fakeroot it will hit fakeroots dir_lookup sure how so ? if the path is absolute, it will trigger a magic retry of some kind so the client uses it's root dir port i thought the lookup would be done straight from the root fs port .. which points to fakeroot of course ah, chrooted again that's the whole point so this implies clients are chrooted they are even if you do another chroot what i mean is that root port also points to a fakeroot port if we detach the translator, and clients outside the chroot spawn processes, say shell scripts, they won't find the fakeroot tree now, i wonder if we want to actually handle that i'm just uncomfortable with a translator silently shutting down because it has no client if fakeroot is detached, how are clients outside the chroot ever supposed to get a handle to files inside the fakerooted env ? it makes sense for fakeroot, so the expected behaviours here aer conflicting they had those before fakeroot being detached then fakeroot wouldn't go away right unless there is a race but i don't think there is there isn't i call netfs_shutdown clients get the rights before the parent has a chance to terminate and only shutdown if it doesn't return ebusy makes sense ok go ahead :) cool, thanks for the walk-through ;) on the other hand .. that's a complicated topic left unfinished by the original authors one of many having translators automatically go away when there is no client may be a good feature but it only makes sense for passive translators and this should be automated the lib*fs libraries should be able to handle it or, we could go for proper persistence instead stay around if active, leave after a while when no more clients if passive why ? clean solution persistence looks much more expensive to me other benefits i mean persistence looks so expensive it doesn't make sense in a general purpose system sure, we could make our *fs libs handle this smarter at a much lower cost don't we get a handle to the underlying file ? i think we do yes if that's actually a file and not a directory, we could store data into it many translators are read-only so ? well, when we can write, we can use passive translators instead normally yes depends on the fs type actually but you're right, we could use regular files or a special type of file, i don't know braunr: BTW, I agree that active translators should only go away when no ports are open anymore, while passive ones can exit when control ports are still open but no protids antrik: you mean as a general rule ? that leaves the question how the translator distinguishes between having a passive translator record and not having one I believe I already arrived at that conclusion in some design discussion, probaly regarding namespace-based translator selection teythoon: yeah, as a general rule interesting currently there are command line arguments controling timeouts, but they don't consider control ports IIRC i thought there are problems with shutting down translators in general (also, command line arguments seem inconvenient to distinguish the passive vs. active case...) yeah, but we disregard the timeouts in the debian flavor of hurd teythoon: err... no we don't. at least not last time I knew. are you confusing this with thread timeouts? simple test: do ls -l on /dev, wait a few minutes, compare what do you expect will happen ? the unused translators should go away no that must be new then might be, yes http://darnassus.sceen.net/gitweb/teythoon/packaging/hurd.git/blame/HEAD:/debian/patches/libports_stability.patch antrik: debian currently disables both the global and thread timeouts in libports my work on thread destruction consists in part in reenabling thread timeouts, and my binary packages do that well so far :) braunr: any idea why the global timeouts were disabled? ## IRC, freenode, #hurd, 2013-12-20 antrik: not sure but i suspect there could be races if a message arrives while the server is going away, i'm not sure the client can determine this and retry transparently good point... not sure how that is supposed to work exactly ## IRC, freenode, #hurd, 2013-12-31 btw, we should remove the libports_stability patch and directly change the upstream code if you agree, i can force the global timeout to 0 (because we're still not sure what can go wrong when a translator goes away while a message is being delivered to it) i didn't experience any slowdown with thread destruction however so i'm tempted to set that to an actual reasonable timeout value of 30-300 seconds braunr: if you do, please introduce a macro for the default value so it can be changed easily teythoon: yes i don't understand why these are left as parameters tbh true 30 seconds seems to be plenty enough ## IRC, freenode, #hurd, 2014-01-17 time to give fakeroot-hurd a shot http://darnassus.sceen.net/~rbraun/darnassus_fakeroot_hurd_assert braunr: (wrt fakeroot-hurd) well in my book that shouldn't happen that's why i put the assertion there ;) i assumed so :) then again, /me does not agree with "threads" as concurrency model >,<, and that feeling seems to be mutual :p ? well, obviously, the threads do not agree with me wrt to that assertion the threads ? well, fakeroot is a multithreaded server teythoon: i'm not sure i get the point, are you saying you're not comfortable with threads ? that's exactly what i'm saying ok coroutines/functional i guess ? csp functional not so much ## IRC, freenode, #hurd, 2014-01-20 [[open_issues/libpthread]], [[open_issues/libpthread/t/fix_have_kernel_resources]]. teythoon: it's perfectly possible that the bug i had with fakeroot-hurd have been caused by my own glibc thread related patches has* ok *phew* :p :) i wonder if youpi could reproduce his issue on his machine what issue ? i must have missed something some package failed but he didn't gave any details he wanted to try it on his vm first ok ## IRC, freenode, #hurd, 2014-01-21 teythoon: i still get the same assertion failure with fakeroot-hurd will take a look at that sometimes too braunr: hrm :/ teythoon: don't worry, i'm sure it's nothing big in the mean time, there are updated hurd and glibc packages on my repository with fixed tls and thread destruction cool :) ## IRC, freenode, #hurd, 2014-01-23 teythoon: can you briefly explain this fake reference thing in fakeroot when you have some time please ? braunr: fakeroot creates ports to hand out to clients every port represents a node and references a real node fakeroot allows one to set attributes, e.g. file permissions on any node as if the client was root those faked attributes are stored in the node objects let's focus on fake_reference please once some attribute is faked, that node has to be kept alive otherwise, that faked information is lost so if the last peropen object is closed and some information is faked, a fake reference is kept as indicated by a flag hm in dir lookup, if a node is looked-up that has a fake reference, it is recycled, i.e. the flag cleared and the referecne count is not incremented so every time fakeroot_netfs_release_protid is called b/c, the node in question should not have the fake reference flag set what's the relation between the number of hard links and this fake reference ? i don' i don't think fakeroot has a notion of 'hard links' it does the fake reference is added on nodes with a hard link count greater than 0 but i guess that just means the underlying node still exists ah yes right currently, if the real node is deleted, the fake node is still kept around let's say it's ok for now that's what the comment is talking about, the one that indicates that garbage collection could help here yes properly fixing this is difficult agreed it would require something like inotify anyway b/c of the way file deletion works let's just ignore the issue, that's not what i'm hunting agreed the assertion i have is telling us that we're dropping a fake reference are we certain this isn't possible ?