diff options
Diffstat (limited to 'service_solahart_jakarta_selatan__082122541663/virtualization')
5 files changed, 0 insertions, 1621 deletions
diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/capsicum.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/capsicum.mdwn deleted file mode 100644 index 44503e34..00000000 --- a/service_solahart_jakarta_selatan__082122541663/virtualization/capsicum.mdwn +++ /dev/null @@ -1,22 +0,0 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -*Capsicum - practical capabilities for UNIX* - -<http://www.cl.cam.ac.uk/research/security/capsicum/> - -<http://www.lightbluetouchpaper.org/2010/08/12/capsicum-practical-capabilities-for-unix/> -(server disappeared; [Google -cache](http://webcache.googleusercontent.com/search?q=cache:cCAqjWOhhksJ:www.lightbluetouchpaper.org/2010/08/12/capsicum-practical-capabilities-for-unix/)) - -<http://lackingrhoticity.blogspot.com/2010/10/process-descriptors-in-freebsd-capsicum.html> - -<http://www.cl.cam.ac.uk/research/security/capsicum/slides/20100811-usenix-capsicum.pdf> -/ <http://www.youtube.com/watch?v=raNx9L4VH2k> diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/fakeroot.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/fakeroot.mdwn deleted file mode 100644 index 441d5c13..00000000 --- a/service_solahart_jakarta_selatan__082122541663/virtualization/fakeroot.mdwn +++ /dev/null @@ -1,1330 +0,0 @@ -[[!meta copyright="Copyright © 2010, 2013, 2014 Free Software Foundation, -Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_hurd]] - - -# IRC, freenode, #hurd, 2013-02-26 - - <youpi> btw, about fakeroot-hurd - <youpi> the remaining issue I see is with argv[0] (yes, again...) - - -## IRC, freenode, #hurd, 2013-04-03 - - <youpi> btw, I believe our fakeroot-hurd is close to working actually - <youpi> it's just a argv[0] issue supposed to be fixed by exec_file_name - but apparently not fixed in that case, for some reason - -[[glibc#execve_relative_paths]]. - - -## IRC, freenode, #hurd, 2013-08-26 - - < teythoon> also I looked into the fakeroot issue, aiui the problem is that - scripts are not handled correctly, right? - < teythoon> the exec server fails to locate the scripts file name, and so - it hands the file_t to the interpreter process and passes /dev/fds/3 as - script name - < teythoon> afaics that breaks e.g. python - < youpi> yes - < youpi> pinotree's exec_file_name is supposed to fix that, but for some - reason it doesn't work here - -[[glibc#execve_relative_paths]]. - - < pinotree> it was pochu's, not mine - < youpi> ah, right - < teythoon> ah I see, I was wondering about that - < pochu> it was working for a long time, wasn't it? - < pochu> and only stopped working recently - < youpi> did it completely stop? - < youpi> I have indeed seen odd issues - < youpi> I haven't actually checked whether it has completely stopped - working - < youpi> probably worth looking there first - < pinotree> gtk+3.0 fails, but other stuff like glib2.0 and gtester-using - stuff works - < teythoon> huh? I created tests like "#!/bin/sh\necho $0" and that says - /dev/fd..., and a python script doing the same doesn't even run, so how - can it work for a package build? - < youpi> it works for me in plain bash - < youpi> #!/bin/sh - < youpi> echo $0 - < youpi> € $PWD/test.sh - < youpi> /home/samy/test.sh - < teythoon> it does !? - < youpi> yes - < youpi> not in fakeroot-hurd however, as we said - < teythoon> well, obviously it works when not being run under - fakeroot-hurd, yes - < youpi> ok, so we weren't talking about the same thing - < youpi> a mere shell script doesn't work in fakeroot-hurd indeed - < youpi> that's why we still use fakeroot-sysv - < teythoon> right - < youpi> err, -tcp - - -## IRC, freenode, #hurd, 2013-11-18 - - <teythoon> I believe I figured out the argv[0] issue with fakeroot-hurd - <teythoon> but I'm not sure how to fix this - <teythoon> first of all, Emilios file_exec_file_name patch set works fine - -[[glibc#execve_relative_paths]]. - - <teythoon> but not with fakeroot - <teythoon> - http://git.sceen.net/hurd/hurd.git/blob/HEAD:/exec/hashexec.c#l300 - <teythoon> check_hashexec tries to locate the script file using a heuristic - <teythoon> Emilios patch improves the situation with just providing this - information - <teythoon> but then, the identity port of the "discovered" file is compared - with the id port of the script file - <teythoon> to verify if the heuristic found the right file - <teythoon> but when using fakeroot-hurd, /hurd/fakeroot proxies all - requests - <teythoon> but the exec server is outside of the /hurd/fakeroot - environment, so it gets the id port from the real filesystem - <teythoon> we could skip that test if the script name is explicitly - provided though - <teythoon> that test was meant to see whether a search through $PATH turned - up the right file - <braunr> teythoon: nice - <teythoon> braunr: thanks :) - <teythoon> unfortunately, dpkg-buildpackaging hurd with it still fails for - some reason - <teythoon> but it is faster than fakeroot-tcp :) - <braunr> even chown ? - <braunr> or chmod ? - <teythoon> dunno in detail, but the whole build is faster - <braunr> if you can try it, i'm interested - <braunr> because chown/chmod is also slow on linux with fakeroot-tcp - <teythoon> i can try... - <braunr> so it's probably not a hurd bug - <teythoon> braunr: yes, it really is - <braunr> no i mean - <braunr> chown/chmod being slow with fakeroot-tcp is probably not a hurd - bug - <braunr> but a fakeroot-tcp bug - <teythoon> chowning all files in /usr/bin takes 5.930s with fakeroot-hurd - (6.09 with startup overhead) vs 26.42s (26.59s) with fakeroot-tcp - <braunr> but try it on linux (fakeroot-tcp i mean) - <braunr> although you may want to do it on something you don't care much - about :p) - - -## IRC, freenode, #hurd, 2013-12-03 - - * teythoon is gonna hunt a fakeroot bug ... - <teythoon> % fakeroot-hurd /bin/sh -c ":> /tmp/some_file" - <teythoon> /bin/sh: 1: cannot create /tmp/some_file: Is a directory - <braunr> ah fakeroot-hurd - <teythoon> prevents installing stuff with /bin/install - <teythoon> sure fakeroot-hurd, why would i work on the slow one ? - <braunr> i don't know - <braunr> because it makes chmod/chown/maybe others horrenddously slow - <braunr> ? - <teythoon> yes, fixing this involves fixing fakeroot-hurd - <braunr> are you sure ? - <braunr> i prefer repeating just in case: i saw that problem on linux as - well - <braunr> with fakeroot-sysv - <teythoon> so ? - <braunr> i'm almost certain it's a pure fakeroot bug, not a hurd bug - <braunr> so - <teythoon> even if this is fixed, it still has to pay the socket - communication overhead - <braunr> fixing fakeroot-hurd so that i can be used instead of fakeroot-tcp - is a very good thing to do, obviously - <braunr> it* - <braunr> but it won't solve the chown/chmod speed - <braunr> (or, probably won't) - <teythoon> huh, why not ? - <braunr> 15:53 < braunr> i'm almost certain it's a pure fakeroot bug, not a - hurd bug - <braunr> when i say it's slow, i should be more precise - <braunr> it doesn't show up in top - <teythoon> yes, but why would fakeroot-hurd suffer from the same issue ? - <braunr> the cpu is almost idle - <braunr> oh right, it's a completely different tool - <braunr> my bad - <braunr> right, right, the proper way to implement fakeroot actually :) - <teythoon> yes - <teythoon> this will bring near-native speed - - -## IRC, freenode, #hurd, 2013-12-05 - - <teythoon> fakeroot-hurd just successfully built mig :) - <teythoon> hangs in dh_gencontrol when building gnumach or hurd though - <teythoon> i believe it hangs waiting for a lock - <teythoon> lock like in file lock that is - <teythoon> braunr: no more room for vm_map_find_entry in 80220a40 - <teythoon> 80220a40 <- is that a task ? - <braunr> or a vm_map, not sure - <braunr> probably a vm_map - - -## IRC, freenode, #hurd, 2013-12-06 - - <teythoon> well, aren't threads a source of endless entertainment ... ? - <teythoon> well, I found three more bugs in fakeroot-hurd - <teythoon> one of them requires fixing the locking used in fakeroot - <braunr> ouch - <teythoon> the current code does some lock cycling to aquire a lock out of - order - <braunr> cycling ? - <teythoon> in the netfs_node_norefs function - <teythoon> release and reaquire - <braunr> i see - <teythoon> which imho should be better solved with a weak reference - <teythoon> working on it, it no longer deadlocks but i broke something else - ... - <teythoon> endless fun ;) - <braunr> such things could have been done right in the beginning - <braunr> ... - <teythoon> yes, I wonder - <teythoon> libports has weak references - <teythoon> but pflocal is the only user - <braunr> hm - <teythoon> none of the lib*fs support that - <braunr> didn't i add one in libdiskfs too ? - <braunr> anyway, irrelevant - <braunr> weak references are a nice feature - <braunr> teythoon: i don't see the cycling you mentioned - <braunr> only netfs_node_refcnt_lock being dropped temporarily - <teythoon> yep, that one - <teythoon> line 145 - <teythoon> note that due to another bug this code is currently never run - <braunr> how surprising .. - <braunr> the note about some leak actually gave a hint about that - <teythoon> yeah, that leak - <teythoon> I think i'm actually very close - <teythoon> it's just so frustrating, i thought i got it last night - <braunr> good luck then - <teythoon> thanks :) - - -## IRC, freenode, #hurd, 2013-12-09 - - <teythoon> sweet, i fixed fakeroot-hurd :) - <braunr> /clap - <braunr> what was the problem ? - <teythoon> lots - <braunr> i see - <teythoon> it's amazing it actually run as well as it did - <braunr> mess strikes again - <braunr> i hate messy code .. - * teythoon is building half a hurd package using this ... stay tuned ;) - <azeem> teythoon: is this going to make building faster as well? - <teythoon> most likely, yes - <teythoon> fakeroot-tcp is known to be slow, even on linux - <braunr> teythoon: are you sure about the transparent retry patch ? - <teythoon> pretty sure, why ? - <braunr> it's about a more general issue that we didn't fix yet - <braunr> our last discussions about it lead us to agree that clients should - check the identity of a server before interacting with it - <teythoon> braunr: i don't understand, what's the problem here ? - <braunr> teythoon: fakeroot does the lookup itself, doesn't it ? - <teythoon> yes - <braunr> teythoon: but was that also the case before your patch ? - <teythoon> braunr: yes - <braunr> teythoon: then ok - <braunr> teythoon: i guess fakeroot handles requests only for a specific - set of calls right ? - <braunr> and for others, requests are directly relayed - <teythoon> braunr: yes - <braunr> and that still is the case, right ? - <teythoon> yes - <braunr> ok - <braunr> looks right since it only affects lookups - <braunr> ok then - <teythoon> well, fakeroot-hurd built half a hurd package in less than 70 - minutes - <teythoon> a new record for my box - <braunr> compared to how much before ? - <braunr> (and why half of it ?) - <teythoon> unfortunately it hung after signing the packages... some perl - process with a /usr/bin/tee child - <teythoon> killing tee made it succeed though - <teythoon> braunr: i don't build the udeb package - <braunr> oh ok - <teythoon> braunr: compared with ~75 with fakeroot-tcp and my demuxer - rework, ~80 before - <braunr> teythoon: nice - - -## IRC, freenode, #hurd, 2013-12-18 - - <teythoon> there, i fixed the last fakeroot-hurd bug - <teythoon> *whee* :) - <teythoon> i thought so many times that i got the last fakeroot bug ... - <teythoon> last as in it's in a good enough shape to compile the hurd - package that is - <teythoon> but now it is - <braunr> :) - <braunr> this will make glibc and others so much faster to build - - -## IRC, freenode, #hurd, 2013-12-19 - - <braunr> teythoon_: hum, you should make the behaviour of fakeroot-hurd on - the last client exiting optional - <teythoon_> y? - <teythoon_> fakeroot-tcp does the very same thing - <braunr> fakeroot-hurd is different - <braunr> it's part of the file system - <teythoon_> yes - <braunr> users may want it to stay around - <braunr> and reuse it without checking it's actually there - <teythoon_> but once the last client is gone, who is ever getting another - port to it ? - <teythoon_> no - <teythoon_> that cannot happen - <braunr> really ? - <teythoon_> yes - <braunr> i thought it was like remap - <braunr> since remap is based on it - <teythoon_> the same thing applies to remap - <teythoon_> only settrans has the control port - <braunr> hum - <teythoon_> and uses it once to get a protid for the working dir of the - initial process started inside the chrooted environment - <braunr> you may not want to chroot inside - <teythoon_> so ? - <teythoon_> then, you get another protid - <braunr> i'll make an example - <braunr> i create a myroot directory implemented by fakeroot - <braunr> populate it - <braunr> leave and do something else, - <braunr> i might want to return to it later - <teythoon_> ah - <teythoon_> ok, so you are not using settrans --chroot - -[[hurd/settrans/discussion#chroot]]. - - <braunr> or maybe i'm confusing the fakeroot translator and fakeroot-hurd - <braunr> 10:48 < braunr> you may not want to chroot inside - <braunr> yes - <teythoon_> hm - <teythoon_> ok, so the patch could be changed to check whether the last - control port is gone too - <braunr> i have no idea of any practical use, but i don't see a valid - reason to make a translator go away just because it has no client - <braunr> except for resource usage - <braunr> and if it's installed as a passive translator - <braunr> although that would make fakeroot loose its state - <braunr> though remap state is on the command line so it would be fine for - it - <braunr> see what i mean ? - <teythoon_> yes i do - <braunr> fakeroot state could be saved in some db one day so it may apply, - if anyone feels the need - <teythoon_> so what about checking for control ports too ? - <braunr> i'm not too familiar with those - <braunr> who has the control port of a passive translator ? the parent ? - <teythoon_> that should cover the use case you described - <teythoon_> for the parent translator - <teythoon_> for fsys_getroot requests it has to keep it around - <teythoon_> and for more fsys stuff too - <braunr> and if active ? settrans ? who just looses it ? - <teythoon_> if settrans is used to start an active translator, the parent - fs still gets a right to the control port - <braunr> ok - <braunr> i don't have a clear view of what this implies for fakeroot-hurd - <braunr> we'd want fakeroot-hurd to clean all resources including the - fakeroot translator on exit - <teythoon_> for fakeroot-hurd (or any child translator) this means that a - port from the control port class will still exists - <teythoon_> so we do not exit - <teythoon_> oh, you're speaking of fakeroot.sh ? the wrapper script ? - <braunr> probably - <braunr> for me, fakeroot-hurd is the command line too, similar to - fakeroot-sysv and fakeroot-tcp - <braunr> and fakeroot is the translator - <teythoon_> yes, agreed - <teythoon_> fakeroot-hurd could use settrans --force --chroot ... to force - fakeroot to exit if the main chrooted process dies - <teythoon_> but that'd kill anything that outlives that process - <teythoon_> that might be legitimate, say a process daemonized - <teythoon_> so detecting that noone uses fakeroot is the much cleaner - solution - <braunr> ok - <teythoon_> also, that's what fakeroot-tcp does - <braunr> which is why i suggested an option for that - <teythoon_> why add an option if we can do the right thing without - troubling the user ? - <braunr> ah, if we can, good - <teythoon_> i think we can - <teythoon_> I'll rework the patch, thanks for the hint - <braunr> so - <braunr> just to be clear - <braunr> the way you intend it to work is - <braunr> wait for all clients and the control port to drop before shutting - down - <braunr> the control port is dropped when dettaching the translator, right - ? - <teythoon_> yes - <braunr> but hm - <braunr> what if clients spawn other processes ? - <braunr> they won't find the translator any more - <teythoon_> then, that client get's a port to fakeroot at least for it's - working dir - <teythoon_> so another protid is created - <braunr> ah yes, it's usually choorted for such uses - <braunr> chrooted - <teythoon_> so fakeroot will stick around - <braunr> but clients, even from fakeroot, might simply use absolute paths - <teythoon_> so ? - <braunr> in which case they won't find fakeroot - <teythoon_> it will hit fakeroots dir_lookup - <teythoon_> sure - <braunr> how so ? - <teythoon_> if the path is absolute, it will trigger a magic retry of some - kind - <teythoon_> so the client uses it's root dir port - <braunr> i thought the lookup would be done straight from the root fs port - .. - <teythoon_> which points to fakeroot of course - <braunr> ah, chrooted again - <teythoon_> that's the whole point - <braunr> so this implies clients are chrooted - <teythoon_> they are - <teythoon_> even if you do another chroot - <braunr> what i mean is - <teythoon_> that root port also points to a fakeroot port - <braunr> if we detach the translator, and clients outside the chroot spawn - processes, say shell scripts, they won't find the fakeroot tree - <braunr> now, i wonder if we want to actually handle that - <braunr> i'm just uncomfortable with a translator silently shutting down - because it has no client - <teythoon_> if fakeroot is detached, how are clients outside the chroot - ever supposed to get a handle to files inside the fakerooted env ? - <braunr> it makes sense for fakeroot, so the expected behaviours here aer - conflicting - <braunr> they had those before fakeroot being detached - <teythoon_> then fakeroot wouldn't go away - <braunr> right - <braunr> unless there is a race but i don't think there is - <teythoon_> there isn't - <teythoon_> i call netfs_shutdown - <braunr> clients get the rights before the parent has a chance to terminate - <teythoon_> and only shutdown if it doesn't return ebusy - <braunr> makes sense - <braunr> ok go ahead :) - <teythoon_> cool, thanks for the walk-through ;) - <braunr> on the other hand .. - <braunr> that's a complicated topic left unfinished by the original authors - <teythoon_> one of many - <braunr> having translators automatically go away when there is no client - may be a good feature - <braunr> but it only makes sense for passive translators - <braunr> and this should be automated - <braunr> the lib*fs libraries should be able to handle it - <teythoon_> or, we could go for proper persistence instead - <braunr> stay around if active, leave after a while when no more clients if - passive - <braunr> why ? - <teythoon_> clean solution - <braunr> persistence looks much more expensive to me - <teythoon_> other benefits - <braunr> i mean - <braunr> persistence looks so expensive it doesn't make sense in a general - purpose system - <teythoon_> sure, we could make our *fs libs handle this smarter at a much - lower cost - <teythoon_> don't we get a handle to the underlying file ? - <braunr> i think we do yes - <teythoon_> if that's actually a file and not a directory, we could store - data into it - <braunr> many translators are read-only - <teythoon_> so ? - <braunr> well, when we can write, we can use passive translators instead - <braunr> normally - <teythoon_> yes - <braunr> depends on the fs type actually but you're right, we could use - regular files - <braunr> or a special type of file, i don't know - <antrik> braunr: BTW, I agree that active translators should only go away - when no ports are open anymore, while passive ones can exit when control - ports are still open but no protids - <teythoon> antrik: you mean as a general rule ? - <teythoon> that leaves the question how the translator distinguishes - between having a passive translator record and not having one - <antrik> I believe I already arrived at that conclusion in some design - discussion, probaly regarding namespace-based translator selection - <antrik> teythoon: yeah, as a general rule - <teythoon> interesting - <antrik> currently there are command line arguments controling timeouts, - but they don't consider control ports IIRC - <teythoon> i thought there are problems with shutting down translators in - general - <antrik> (also, command line arguments seem inconvenient to distinguish the - passive vs. active case...) - <teythoon> yeah, but we disregard the timeouts in the debian flavor of hurd - <antrik> teythoon: err... no we don't. at least not last time I knew. are - you confusing this with thread timeouts? - <antrik> simple test: do ls -l on /dev, wait a few minutes, compare - <teythoon> what do you expect will happen ? - <antrik> the unused translators should go away - <teythoon> no - <antrik> that must be new then - <teythoon> might be, yes - <teythoon> - http://darnassus.sceen.net/gitweb/teythoon/packaging/hurd.git/blame/HEAD:/debian/patches/libports_stability.patch - <braunr> antrik: debian currently disables both the global and thread - timeouts in libports - <braunr> my work on thread destruction consists in part in reenabling - thread timeouts, and my binary packages do that well so far :) - <antrik> braunr: any idea why the global timeouts were disabled? - - -## IRC, freenode, #hurd, 2013-12-20 - - <braunr> antrik: not sure - <braunr> but i suspect there could be races - <braunr> if a message arrives while the server is going away, i'm not sure - the client can determine this and retry transparently - <antrik> good point... not sure how that is supposed to work exactly - - -## IRC, freenode, #hurd, 2013-12-31 - - <braunr> btw, we should remove the libports_stability patch and directly - change the upstream code - <braunr> if you agree, i can force the global timeout to 0 (because we're - still not sure what can go wrong when a translator goes away while a - message is being delivered to it) - <braunr> i didn't experience any slowdown with thread destruction however - <braunr> so i'm tempted to set that to an actual reasonable timeout value - of 30-300 seconds - <teythoon> braunr: if you do, please introduce a macro for the default - value so it can be changed easily - <braunr> teythoon: yes - <braunr> i don't understand why these are left as parameters tbh - <teythoon> true - <braunr> 30 seconds seems to be plenty enough - - -## IRC, freenode, #hurd, 2014-01-17 - - <braunr> time to give fakeroot-hurd a shot - <braunr> http://darnassus.sceen.net/~rbraun/darnassus_fakeroot_hurd_assert - <teythoon> braunr: (wrt fakeroot-hurd) well in my book that shouldn't - happen - <teythoon> that's why i put the assertion there ;) - <braunr> i assumed so :) - <teythoon> then again, /me does not agree with "threads" as concurrency - model >,<, and that feeling seems to be mutual :p - <braunr> ? - <teythoon> well, obviously, the threads do not agree with me wrt to that - assertion - <braunr> the threads ? - <teythoon> well, fakeroot is a multithreaded server - <braunr> teythoon: i'm not sure i get the point, are you saying you're not - comfortable with threads ? - <teythoon> that's exactly what i'm saying - <braunr> ok - <braunr> coroutines/functional i guess ? - <teythoon> csp - <teythoon> functional not so much - - -## IRC, freenode, #hurd, 2014-01-20 - -[[open_issues/libpthread]], -[[open_issues/libpthread/t/fix_have_kernel_resources]]. - - <braunr> teythoon: it's perfectly possible that the bug i had with - fakeroot-hurd have been caused by my own glibc thread related patches - <braunr> has* - <teythoon> ok - <teythoon> *phew* :p - <braunr> :) - <teythoon> i wonder if youpi could reproduce his issue on his machine - <braunr> what issue ? - <braunr> i must have missed something - <teythoon> some package failed - <teythoon> but he didn't gave any details - <teythoon> he wanted to try it on his vm first - <braunr> ok - - -## IRC, freenode, #hurd, 2014-01-21 - - <braunr> teythoon: i still get the same assertion failure with - fakeroot-hurd - <braunr> will take a look at that sometimes too - <teythoon> braunr: hrm :/ - <braunr> teythoon: don't worry, i'm sure it's nothing big - <braunr> in the mean time, there are updated hurd and glibc packages on my - repository with fixed tls and thread destruction - <teythoon> cool :) - - -## IRC, freenode, #hurd, 2014-01-23 - - <braunr> teythoon: can you briefly explain this fake reference thing in - fakeroot when you have some time please ? - <teythoon> braunr: fakeroot creates ports to hand out to clients - <teythoon> every port represents a node and references a real node - <teythoon> fakeroot allows one to set attributes, e.g. file permissions on - any node as if the client was root - <teythoon> those faked attributes are stored in the node objects - <braunr> let's focus on fake_reference please - <teythoon> once some attribute is faked, that node has to be kept alive - <teythoon> otherwise, that faked information is lost - <teythoon> so if the last peropen object is closed and some information is - faked, a fake reference is kept - <teythoon> as indicated by a flag - <braunr> hm - <teythoon> in dir lookup, if a node is looked-up that has a fake reference, - it is recycled, i.e. the flag cleared and the referecne count is not - incremented - <teythoon> so every time fakeroot_netfs_release_protid is called b/c, the - node in question should not have the fake reference flag set - <braunr> what's the relation between the number of hard links and this fake - reference ? - <teythoon> i don' - <teythoon> i don't think fakeroot has a notion of 'hard links' - <braunr> it does - <braunr> the fake reference is added on nodes with a hard link count - greater than 0 - <braunr> but i guess that just means the underlying node still exists - <teythoon> ah yes - <teythoon> right - <teythoon> currently, if the real node is deleted, the fake node is still - kept around - <braunr> let's say it's ok for now - <teythoon> that's what the comment is talking about, the one that indicates - that garbage collection could help here - <teythoon> yes - <teythoon> properly fixing this is difficult - <braunr> agreed - <braunr> it would require something like inotify anyway - <teythoon> b/c of the way file deletion works - <braunr> let's just ignore the issue, that's not what i'm hunting - <teythoon> agreed - <braunr> the assertion i have is telling us that we're dropping a fake - reference - <braunr> are we certain this isn't possible ? - <teythoon> that function is called if a client dereferences a port - <teythoon> in order to have a port in the first place, it has to get it - from a dir_lookup - <teythoon> the dir lookup turns a fake reference into a real one - <teythoon> so i'm certain of that (barring a race condition somewhere) - <braunr> ok - <braunr> netfs_S_dir_lookup grabs idport_ihash_lock (line 354) but doesn't - release it if nn == NULL (lines 388-392) - <teythoon> hm, my file numbers are slightly different o_O - <braunr> i have printfs around - <braunr> sorry :) - <teythoon> ok - <teythoon> new node unlocks it - <teythoon> new_node - <braunr> oh - <braunr> how unintuitive .. - <teythoon> yes, don't blame me ;) that's how it was - <braunr> :) - <braunr> worse, the description says "if successful" .. - <braunr> ah no, the node lock - <braunr> ok - <teythoon> yes, badly worded description - <braunr> i strongly doubt it's a race - <teythoon> how do you trigger that assertion failure ? - <braunr> dpkg-buildpackage -rfakeroot-hurd -uc -us - <braunr> for the hurd package - <braunr> very similar to one of your test cases i think - <teythoon> umm :-/ - <braunr> one thing that i find confusing is that fake_reference seems to - apply to nodes, whereas release_protid is about, well, protids - <braunr> is there a 1:1 relationship ? - <braunr> since there is a peropen in the protid, i assume not - <braunr> it may be a race actually - <braunr> np->references must be accessed with netfs_node_refcnt_lock locked - <braunr> hm no, that's not it - <teythoon> no, it's not a 1:1 relationship - <teythoon> note that the lock idport_ihash_lock serializes most operations, - despite it's name indicating that it's only for the hash table - <teythoon> the "interesting" operations being dir_lookup and release_protid - <braunr> yes - <braunr> again, that's another issue - <teythoon> why ? that's a pretty strong guarantee already - <braunr> ah yes, i was referring to scalability - <teythoon> sure - <braunr> the assertion is triggered from ports_port_deref in - ports_manage_port_operations_multithread - <teythoon> but i found it hard to reason about fakeroot, there are multiple - locks involved, two kinds of reference counting across different libs - <braunr> yes - <teythoon> yes, that's to be expected - <braunr> teythoon: do we agree that the fake reference is reused by a - protid ? - <teythoon> braunr: yes - <braunr> why is there a ref counter for the protid as well as the peropen - then ? :/ - <teythoon> funny... i thought there was no refcnt for the peropen objects, - but there is - <teythoon> but for fakeroot-hurd that shouldn't matter, right ? - <braunr> i don't know - <teythoon> here, one protid object is associated with one peropen object - <braunr> yes - <teythoon> and the other way around, i.e. it's 1:1 - <teythoon> so the refcount for those should be identical - <braunr> but i get a case where protid has a refcnt of 0 while the peropen - has 2 .. - <teythoon> umm, that doesn't sound right - <braunr> teythoon: ok, it does look like a race on np->references - <braunr> node references are protected by a global lock in lib*fs libs - <teythoon> yes - <braunr> you check it without holding it - <braunr> which means another protid can be closed at the same time, setting - the flag on the underlying node - <braunr> i'll make a proper patch soon - <teythoon> they cannot both hold the hash lock - <braunr> hm - <braunr> teythoon: actually, i don't see why that's relevant - <braunr> one thread closes its protid, sets the fakeref flag - <braunr> the other does the same, chokes on the assertion - <braunr> serially - <teythoon> i'm always a little fuzzy when exactly the references get - decremented - <teythoon> but shouldn't only the second thread set the fakeref flag ? - <braunr> well, that's not what i see - <braunr> i'll check what happens to this ref counter - <teythoon> see how my release_protid function calls netfs_release_protid - just after the out label - <teythoon> *while holding the big hash lock - <teythoon> so, any refcounting should happen while the lock is being held, - no ? - <braunr> perhaps - <braunr> now, my logs show something new - <braunr> a case where the protid being released was never printed before - <braunr> i.e. not obtained from dir_lookup - <braunr> or at least, not fakeroot dir_lookup - <teythoon> huh, where did it came from then ? - <braunr> no idea - <teythoon> only dir_lookup hands out those - <braunr> check_openmodes calls dir_lookup too - <teythoon> yes, but that's not our dir_lookup - <braunr> that's what i mean - <braunr> it bypasses fakeroot's custom dir_lookup - <braunr> but i guess the reference already exists at this point - <teythoon> bypass ? i wouldn't call it that - <braunr> you're right, wrong wording - <teythoon> it accesses files on other translators - <braunr> yes - <braunr> the netnode is already present - <teythoon> yes - <braunr> could it be the root node ? - <teythoon> i do not believe so - <teythoon> the root node is always faked - <teythoon> and is handed out to the first process in the fakeroot env for - it's current directory port - <teythoon> so you could try something that chdirs away to test that - hypothesis - <braunr> the assertion looks triggered by a chdir - <teythoon> how do you know that ? - <braunr> dh_auto_install: error: unable to chdir to build-deb - <teythoon> ah - <teythoon> well, or that is just the operation after fakeroot died and - completely unrelated - <braunr> maybe - <teythoon> can you trigger this reliably ? - <braunr> yes - <braunr> i'm trying to write a shell script for that - <teythoon> so for you, fakeroot-hurd never succeeded in building a hurd - package ? - <braunr> no - <teythoon> on darnassus ? - <braunr> yes - <teythoon> b/c i stopped working on fakeroot-hurd when it was in a - good-enough shape to build the hurd package - <teythoon> >,< - <teythoon> maybe my system is not fast enough to hit this race (if it turns - out to be one) - <braunr> some calls seems to decrease the refcount of the root node - <braunr> call* - <teythoon> have you confirmed that it's the root node ? - <braunr> almost - <braunr> i could say yes - <braunr> teythoon: actually no, it's not .. - <braunr> could be .. - <braunr> teythoon: on what node does fakeroot-hurd install the fakeroot - translator when used to build debian packages ? - <braunr> hum - <braunr> could it simply be that the check on np->references should be - moved above the assertion ? - <teythoon> braunr: it is not bound to any node, check settrans --chroot - -[[hurd/settrans/discussion#chroot]]. - - <braunr> oh right - <braunr> teythoon: ok i mean - <braunr> does it shadow / ? - <braunr> looks very likely, otherwise the chroot wouldn't work - <teythoon> i'm not sure what you mean by shadow - <braunr> settrans --chroot cmd -- / /hurd/fakeroot ? - <teythoon> but yes, for any process in the chroot-like env every real node - is replaced, including / - <braunr> makes sense - <braunr> teythoon: moving the assertion seems to fix the issue - <braunr> intuitively, it seems reasonable to assume the fakeref flag can - only be set when there is only one reference, namely the fake reference - <braunr> (well, the fake ref, recycled by the last open) - <teythoon> no, i don't follow - <teythoon> i'd still say, that if ...release_protid is called, then there - is no way for the fake flag to be set in the first place - <teythoon> that's why i put the assertion in ;) - <braunr> on the other hand, you check the refcnt precisely because other - threads may have reacquired the node - <teythoon> but why would moving the assertion change anything ? - <teythoon> if we would do that, we'd "lose" all threads that see - np->reference being >1 - <teythoon> but for those objects the fake_reference flag should never be - set anyways - <teythoon> i cannot see why this would help - <teythoon> (does it help ?) - <teythoon> (and if it does, it points to a serious problem imho) - <braunr> i'm recreating the traces that made me think that - <braunr> to get a clearer view of what's happening - <braunr> the problem i have with the current code is this - <braunr> there can be multiple protid referring to the same node occurring - at the same time - <braunr> they are serialized by the hash table lock, ok - <braunr> but there apparently are cases where the first (of two) protids - being closed sets the fakeref flag - <braunr> and the following chokes because the flag is set - <braunr> i assume you put this refcount check because you assumed only the - last protid being closed can set the flag, right ? - <braunr> but then, why > 1 ? why not > 0 ? - <teythoon> yes, that's what i was trying to assert - <teythoon> b/c the 1 is our reference - <braunr> which one exactly ? - <teythoon> >1 is anyone *beside* us - <teythoon> ? - <braunr> hm - <braunr> you mean the reference held by the protid being destroyed - <teythoon> yes - <braunr> isn't that reference already dropped before calling the cleanup - function ? - <braunr> ah no, it's the node ref - <teythoon> yes - <braunr> released by netfs_release_protid - <teythoon> exactly - <braunr> which is called without the hash table lock held - <braunr> hm no - <braunr> it's locked - <braunr> damn my brain is slow today - <teythoon> i actually think that it's the combination of manual reference - counting and the primitive concurrency model that makes it hard to reason - about this - <braunr> well - <braunr> the model is quite simple too - <braunr> accesses to refcounters must be protected by the appropriate lock - <braunr> this isn't done here, on the assumption that all referencing - operations are protected by another global lock all the time - <teythoon> even if a model is simple, this does not mean that it is a good - model for human beings to comprehend and reason about - <braunr> i don't know - <braunr> note that netfs_drop_node is designed to be called with - netfs_node_refcnt_lock locked - <braunr> implying the refcount must remain stable between checking it and - dropping the node - <braunr> netfs_make_peropen is called without the hash table lock held in - dir_lookup - <braunr> and this increases the refcount - <braunr> although the problem is rather that something decreases it without - the lock held - <teythoon> we should port libtsan and just ask gcc -fsanitize=thread - <braunr> what about the netfs_nput call at the end of dir_lookup ? - <braunr> the fake ref should be set by the norefs function - <teythoon> that should not decrease the count to 0 b/c the caller holds a - reference too - <braunr> yes that's ugly - <braunr> ugh - <braunr> i'm unable to think clearly right now - <teythoon> as mentioned in the commit message, you cannot do something like - this in the norefs function - <teythoon> bbl ;) - <braunr> bye teythoon - <braunr> thanks for your time - <braunr> for when you come back : - <braunr> instead of maintaining this "fake" reference, why not assumeing - the hash table holds a reference, and simply count it - <braunr> the same way a cache does - <braunr> and drop that reference when removing a node, either to reflect - the current state of the underlying node, or because the translator is - being shut down ? - <braunr> why not assume* - <braunr> bbl too - <teythoon> sure, refactoring is definitively an option - - -## IRC, freenode, #hurd, 2014-01-24 - - <braunr> teythoon: ok, i'll take care of fakeroot - <teythoon> braunr: thanks. tbh i was a little fed up with that little - bugger >,< - <braunr> i can imagine - <braunr> considering the number of patches you've sent already - - <braunr> teythoon: are you sure about your call to fshelp_lock_init ? - <teythoon> yes, why do you ask ? - <teythoon> (the test case is given in the commit message) - <braunr> it doesn't look right to me to call "init" while the node is - potentially locked - <braunr> i noticed libdiskfs peropen release function takes care of - releasing locks - <braunr> it looks better to me - <teythoon> it's not about releasing the lock - <teythoon> it's about faking the file being closed which implicitly - releases the lock - <braunr> the file is being close - <braunr> closed - <braunr> since it's in the cleanup function - <teythoon> yes, but we keep it b/c the file has faked attributes - <teythoon> did you look at the problem description in the commit message ? - <braunr> we keep the node - <braunr> not the peropen - <teythoon> so ? - <teythoon> the lock is in the node - <braunr> why would libdiskfs do it in the peropen release then ? - <braunr> there is an inconsistency somwhere - <braunr> actually, the lock looks to be per open - <braunr> or rather, the lock is per node, but its status is recorded per - open - <braunr> allowing the implementation to track if a file descriptor was used - to install a lock and release it when that file descriptor goes away - <teythoon> why would the node be locked ? - <teythoon> locked in what way, file-locking locked ? - <braunr> yes - <braunr> posix explicitely says that file locks must be implicitely removed - when closing the file descriptor used to install them, so that makes - sense - <teythoon> isn't hat exactly what i'm doing ? - <braunr> no - <braunr> you're initializing the file lock - <braunr> init != unlock - <braunr> and it's specific to fakeroot, while it looks like libnetfs should - be doing it - <teythoon> libnetfs would do it - <teythoon> but we prevent that by keeping the node alive - <braunr> again, it's a per open thing - <braunr> and no, libnetfs doesn't release locks implicitely in the current - version - <teythoon> didn't we agree that for fakeroot one peropen object is - associated with one protid object ? - <braunr> yes - <braunr> and don't keep those alive - <braunr> so let them die peacefully, and fix libnetfs so it releases the - lock as it's supposed to - <braunr> and we* don't - <teythoon> we don't keep those alive - <teythoon> why would we ? - <braunr> yes that's what i wanted to say - <braunr> what i mean is - <braunr> since letting peropens die is already what is being done - <braunr> there is no need for a special handling of locks in fakeroot - <teythoon> oh - <braunr> on the other hand, libnetfs must be fixed - <teythoon> ok, that might very well be true - <teythoon> (we need to bring libnetfs and diskfs closer so that they can be - diff'ed easily) - <braunr> i just wanted to check your reason for using lock_init in the - first place - <braunr> yes .. - <braunr> teythoon: also, i think we actually do have what's necessary to - deal with garbage collection - <braunr> namely, dead-name notifications - <braunr> i'll see if i can cook something simple enough - <braunr> otherwise, merely keeping every node around is also acceptable - considering the use cases - <teythoon> dead-name notifications won't help if the real node disappears, - no ? - <braunr> teythoon: dead name notifications on the real node port :) - <braunr> teythoon: at least i can reliably build the hurd package using - fakeroot-hurd now - <braunr> let's try glibc :) - -## IRC, freenode, #hurd, 2014-01-25 - - <teythoon> braunr: awesome :) - <braunr> teythoon: hm not sure :/ - <braunr> darnassus got oom - <braunr> teythoon: could be unrelated though - <braunr> teythoon: something has apprently made /home unresponsive :( - <braunr> teythoon: i suspect bots hitting apache and in particular the git - repositories to have increased memory usage - - -## IRC, freenode, #hurd, 2014-01-26 - - <braunr> teythoon: btw, fakeroot interacts very very badly with other netfs - file systems - <braunr> e.g., listing /proc through it creates lots of nodes - <braunr> i'm not yet sure how to fix that - <braunr> using a dead name notification doesn't seem appropriate (at least - not directly) because fakeroot holds a true reference that prevents the - deallocation of the target node - - -## IRC, freenode, #hurd, 2014-01-27 - - <braunr> teythoon: good news (more or less): fakeroot is actually leaking a - lot when crossing file systems - <braunr> which means if i fix that, there is a good chance we can use it to - build all packages with it - <braunr> -with it - <teythoon> what do you mean exactly ? - <braunr> if target nodes are from /, there is no such leak - <braunr> as soon as the target nodes are from another file system, ports - rights are leaked - <braunr> that's what fills the kernel allocator actually - <teythoon> oh, so dir_lookup leaks ports when crossing translator - boundaries ? - <braunr> seems so - <teythoon> yeah, that might very well be it - <teythoon> the dir_lookup logic in lib*fs is quite involved :/ - <braunr> yes, my simple attempts were unsuccessful - <braunr> but i'm confident i can fix it soon - <teythoon> that sounds good :) - <braunr> i also remove the fake_ref flag and replace it with "accounting - the reference in the hash table" as soon as a node is faked - <teythoon> fine with me - <braunr> these will be the expected leak - <braunr> but they're far less in numbers than what i observe - <braunr> and garbage collection can be implemented later - <braunr> although i would prefer notifications a lot more - <braunr> end of the news, bbl :) - <braunr> found it :> - <teythoon> braunr: -v ;) - <braunr> err = dir_lookup (...); - <braunr> if (dir != dnp->nn->file) mach_port_deallocate (mach_task_self (), - dir); - <braunr> in other words, deallocate ports for intermediate file system root - directories .. :) - <braunr> teythoon: currently building hurd and glibc packages - <braunr> but i intend to improve some more with the addition of a default - faked state - <braunr> so that only nodes with modified faked states are retained - <teythoon> how do you mark nodes as having the default faked state ? - <braunr> i don't - <teythoon> ok, right, makes sense :) - <teythoon> this sounds awesome, thanks for following up on this - <braunr> i'm quite busy with other stuff so, with proper testing, it should - take me the week to get merged - <braunr> teythoon: well thanks for all the fixes you've done - <braunr> fakeroot was completely unusable before that - <teythoon> if you push your changes somewhere i'll integrate them into my - packages and test them - <braunr> ok - <braunr> implementing fakeroot -u could also be a good thing - <braunr> and this should work easily with that default faked state strategy - - -## IRC, freenode, #hurd, 2014-01-28 - - <braunr> teythoon: i should be able to test fakeroot-hurd with the default - faked attributes strategy today on glibc - <teythoon> braunr: very nice :) - <braunr> azeem_: do you happen to know if fakeroot -u is used by debian ? - <braunr> i mean when building packages - <teythoon> braunr: how does fakeroot-hurd perform on darnassus ? - <teythoon> i mean, does it yield a noticeable improvement over fakeroot-tcp - just like on my slow box ? - <braunr> i'm not measuring that :/ - <teythoon> ok, no problem - <braunr> and since nodes are removed from the hash table, performance might - decrease slightly - <braunr> but the number of rights is kept very low, as expected - <teythoon> that's good - <braunr> i keep seeing leaks though - <braunr> when switching cwd between file systems - <teythoon> humm - <braunr> so i assume something is wrong with the identity of . or .. - <braunr> it's so insignificant compared to the previous problems that i - won't waste time on that - <braunr> teythoon: the problem with measuring on darnassus is that it's a - public machine - <teythoon> right - <braunr> often scanned by ssh worms or http bots - -[[cannot_create__dev_null__interrupted_system_call]]. - - <braunr> but it makes complete sense to get better performance with - fakeroot-hurd - <braunr> that's actually one of the reasons i'm working on it - <braunr> if not the main one - <teythoon> :) - <teythoon> that was my motivation too - <braunr> it shows how you can get an interchangeable unix tool that - directly plugs well with the low level system - <braunr> and make it work better - <teythoon> nicely put :) - - <braunr> teythoon: i still can't manage to build glibc with fakeroot-hurd - <braunr> but i'm not sure why :/ - <braunr> there was no kernel memory exhaustion this time - <teythoon> :/ - <braunr> cp: cannot create regular file `debian/libc-bin.dirs': Permission - denied - <braunr> hum - <braunr> youpi: do you know if building debian packages requires fakeroot - -u option ? - <youpi> I don't know - <gg0> braunr: man dpkg-buildpackage says it just runs "fakeroot - debian/rules <target>" - <gg0> sources confirm that - http://sources.debian.net/src/dpkg/1.17.6/scripts/dpkg-buildpackage.pl#L465 - <braunr> gg0: ok - - -## IRC, freenode, #hurd, 2014-01-29 - - <braunr> it seems that something sets the permissions of this - debian/libc-bin.dirs file to 000 ... - <teythoon> i've seen this too - <braunr> oh - <braunr> do you think it's a fakeroot-hurd bug ? - <teythoon> have i mentioned something like this in a commit message ? - <teythoon> yes - <teythoon> it is - <braunr> ok - <braunr> i didn't see any mention of it - <braunr> but i could have missed it - <teythoon> hm, i cannot recall it either - <teythoon> but i've seen this issue with fakeroot-hurd - <braunr> ok - <braunr> it's probably the last issue to fix to get it to work for our - packages - <braunr> teythoon: i think i have a solution for that last mode bug - <braunr> fakeroot doesn't relay chmod requests, unless they change an - executable bit - <braunr> i don't see the point, and simply removed that condition to relay - any chmod request - <teythoon> braunr: did it work ? - <braunr> no - <braunr> fakeroot still consumes too many ports - <braunr> and for each file, there are at least two ports, the faked one, - and the real one - <braunr> it should be completely reworked - <braunr> but i don't have time to do that - <braunr> i'll see if it works when building from scratch - <braunr> actually, it's not even a quantity problem but a fragmentation - problem - <braunr> the function that fails is kmem_realloc .. - <braunr> ipc spaces are arrays in kernel space .... - <teythoon> it's more like three ports per file, you forgot the identity - port - <braunr> ah yes - - -## IRC, freenode, #hurd, 2014-02-03 - - <braunr> teythoon: i'll commit my changes on fakeroot tonight - <braunr> they do improve the tool, but not enough to build glibc with it - <teythoon> braunr: cool :), so how do we make it fully usable ? - <braunr> teythoon: i don't know .. - <braunr> i'll try re adding detection of nodes with no hard links for one - <braunr> but imho, it needs a rework based on what the real fakeroot does - <braunr> i won't work on it though - - <braunr> teythoon: also, it looks like i've tested building glibc with a - wrong test binary of my fakeroot version :/ - <braunr> so consider all test results irrelevant so far - - -## IRC, freenode, #hurd, 2014-02-04 - - <braunr> fakeroot-hurd might turn out to be easily usable for our debian - packages with the fixed binary :) - - <braunr> teythoon: hum, can you explain - 672005782e57e049c7c8f4d6d0b2a80c0df512b4 (trans: fix locking issue in - fakeroot) when you have time please ? - <braunr> it looks like it introduces a deadlock by calling new_node (which - acquires the hash table lock) while dir is locked, violating the hash - table -> node locking order - - <teythoon> braunr: awesome, then there still is hope for fakeroot-hurd :) - - <braunr> teythoon: i've been able to build glibc packages several times - this night - <braunr> so except for this deadlock i've seen once, it looks good - <teythoon> right - <teythoon> that deadlock - <teythoon> right, it does indeed violate the partial order of the locks :-/ - - <braunr> teythoon: can you explain why you moved the lock in attempt_mkfile - please ? - - <braunr> teythoon: i've just tested a fakeroot binary without the patch - introducing the deadlock, and glibc built without any problem - <teythoon> braunr: well, this is very good news :) - <braunr> teythoon: but i still wonder why you made this patch in the first - place, i don't want to revert it blindly and reintroduce a potential - regression - <teythoon> braunr: i thought i was fixing the order in which locks were - taken. if the commit message does not specify that it fixes an issue, - then i was probably just wrong and you can revert it - <braunr> oh ok - <braunr> good - - <braunr> teythoon: another successful build :) - <braunr> i'll commit my changes - <teythoon> awesome :) - <braunr> there might still be concurrency issues but it's much better - <teythoon> i'm curious what you did :) - <braunr> so little :) - <braunr> i was sick all week heh - <braunr> you'll se - <braunr> see - <teythoon> well, that's good actually ;) - <braunr> yes - - <braunr> teythoon: actually there was another debugging line left over, and - again, my test results are irrelevant @#! - - -## IRC, freenode, #hurd, 2014-02-05 - - <braunr> teythoon: i got an assertion about nn->np->nn not being equal to - nn atfer the hash table lookup is dir_lookup - <braunr> +failure - <teythoon> that's bad - <braunr> not over yet - <teythoon> i had a couple of those too - <teythoon> i guess it's a use after free - <braunr> yes - <teythoon> i used to poison the pointers and comment out the frees to track - them down iirc - <braunr> teythoon: one of your patches stores netnodes instead of nodes in - the hash table, citing some overwriting issue - <braunr> teythoon: i don't understand why using netnodes fixes this - <teythoon> braunr: libihash has this cookie for fast deletes - <teythoon> that has to be stored somewhere - <teythoon> the node structure has no room for it - <braunr> uh - <teythoon> yes - <teythoon> it was that bad - <braunr> ... - <teythoon> hence the uglyish back pointers - <braunr> i see - <teythoon> looking back i cannot even say why it worked at all - <braunr> well, it didn't - <teythoon> i believe libihash must have destroyed a linked list in the node - struct - <braunr> possibly - <teythoon> no, it did not >,<, but for simple tests it kind of did - <braunr> yes fakeroot sometimes corrupts memory badly .... - <braunr> and yes, turns out the assertion is triggered on nodes with 0 refs - .. - <braunr> teythoon: it looks like even the current version makes wrong usage - of the ihash interface - <braunr> locp_offset is defined as "The offset of the location pointer from - the hash value" - <braunr> and indeed, it's an intptr_t - <braunr> teythoon: hm no, it looks ok actually, forget what i said :) - <teythoon> *phew - <teythoon> :p - - <braunr> hmm, still occasional double frees in fakeroot, but it looks in - good shape for single threaded tasks like package building - - <braunr> teythoon: i've just sent my fakeroot patches - <teythoon> braunr: sweet, i'll have a closer look tomorrow :) - <braunr> teythoon: i couldn't debug the double frees though :/ - - -## IRC, freenode, #hurd, 2014-02-06 - - <braunr> btw, i'm able to successfully use fakeroot-hurd to build glibc - packages, but is there a way to make sure the resulting archives contain - the right privileges and ownerships ? - <youpi> I don't remember whether debdiff checks permissions - - <youpi> braunr: I've just got fakeroot-hurd debian/rules clean - <youpi> dh_clean - <youpi> fakeroot: ../../trans/fakeroot.c:161: netfs_node_norefs: Assertion - `np->nn->np == np' failed. - <youpi> while building eglibc - <teythoon> youpi: yes, that lockup is most annoying... :/ - <braunr> youpi: with the new version ? - <youpi> yes - <braunr> hum - <braunr> i only had rare double frees, not that any more :/ - <braunr> youpi: ok i got the error too - <braunr> still not good enough - <youpi> ok - - -## IRC, freenode, #hurd, 2014-02-07 - - <braunr> youpi: debdiff seems to handle permissions - <braunr> i've found the cause of the assertions - <youpi> braunr: groovie :) - - -## IRC, freenode, #hurd, 2014-02-08 - - <teythoon> braunr: nice :) - <braunr> http://darnassus.sceen.net/~rbraun/debdiff_report - - -## IRC, freenode, #hurd, 2014-02-10 - - <braunr> and, on a completely different topic, here is a crash i can - reproduce when using fakeroot: - http://darnassus.sceen.net/~rbraun/fakeroot_hurd_rpctrace_o_var_tmp_out_rm_rf_dir.png - - -## IRC, freenode, #hurd, 2014-02-11 - - <braunr> still working on fakeroot - <braunr> there are still races (not disturbing for package building but - still ..) - <braunr> there may be wrong right handling - <teythoon> i believe i have witnessed a fakeroot deadlock :/ - <braunr> aw - <teythoon> not sure though, buildbot killed the build process before i - could investigate - <braunr> teythoon: was it a big package ? - <teythoon> half of the hurd package - <braunr> that's not a port right overflow then - - -## IRC, freenode, #hurd, 2014-03-05 - - <teythoon> youpi: what about the exec_filename patch series? even though - fakeroot still has some issues (has it?), i consider it worthy for - inclusion - -[[glibc#execve_relative_paths]]. - - <youpi> Roland was disagreeing with it - <youpi> iirc the fakeroot issue was solved - <teythoon> braunr: ^ - <braunr> fakeroot goot a lot more robust than it used to be - <braunr> but i haven't checked that it actually behaves exactly like the - library for corner cases - <braunr> there are minor differences - <braunr> also, it seems to trigger concurrency bugs in ext2fs - <braunr> e.g. git reporting that files either "already exist" or "can't be - found" - <braunr> it happens (rarely) when directly using ext2 - <braunr> and more often through fakeroot - <braunr> i didn't take the time to investigate - -## youpi - - the daily script of debian-installer uses the -s / -i options of fakeroot. How could we manage to implement them? diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/file_systems.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/file_systems.mdwn deleted file mode 100644 index a12ea10d..00000000 --- a/service_solahart_jakarta_selatan__082122541663/virtualization/file_systems.mdwn +++ /dev/null @@ -1,24 +0,0 @@ -[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_hurd]] - -Of course, it is possible to use commodity file systems on [[virtualized -systems|virtualization]], like [[hurd/translator/ext2fs]] or -[[hurd/translator/nfs]], but there are also other possibilities which ought to -be explored. - - * [[network_file_system_by_just_forwarding_RPCs]] - - * Linux saw a patch for [*generic name to handle and open by handle - syscalls*](http://thread.gmane.org/gmane.linux.file-systems/46648) posted, - which in turn can be beneficial for a [[QEMU]] emulation of a 9P file - system. LWN's Jonathan Corbet covered this [*open by - handle*](http://lwn.net/Articles/375888/) functionality on 2010-02-23. diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/networking.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/networking.mdwn deleted file mode 100644 index 122f21ab..00000000 --- a/service_solahart_jakarta_selatan__082122541663/virtualization/networking.mdwn +++ /dev/null @@ -1,88 +0,0 @@ -[[!meta copyright="Copyright © 2010, 2013 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_hurd]] - -Collection about stuff that is relevant for *virtualization* and *networking*. - - * [[Virtual_Square_View-OS]] - - * [*Virtual Networks*](http://virtualsquare.org/vn.html) - - * [User Level Networking](http://uln.sourceforge.net/) - - * [Virtual Distributed Ethernet](http://vde.sourceforge.net/) - - * [Application Level - Environment4Networking](http://sourceforge.net/projects/ale4net/) - - *Ale4NET used dyn library call diversion to define networking at process - level.* -- what we're doing with our approach for overriding the default - [[hurd/translator/pfinet]] by setting environment variables. - - Project is now part of [[Virtual_Square_View-OS]]. - - -# OpenVPN - -[[community/meetings/GHM2013]]. - -OpenVPN can use pfinet's tun as it is, and be configured completely as joe user, -as shown below. Note that the tun0 node name has to begin with "tun", so pfinet -knows it's a tun. - - $ mkdir -p $HOME/servers/socket - $ touch $HOME/servers/tun0 - $ settrans -ca $HOME/servers/socket/2 /hurd/pfinet $HOME/servers/tun0 -a 10.0.0.1 -p 10.0.0.2 - $ cat vpn.conf - client - dev tun - dev-node /home/samy/servers/tun0 - ... - $ openvpn --config vpn.conf --verb 5 - ... - -So we let openvpn running, and now we can make applications use the pfinet -TCP/IP stack started above: the remap command below starts a new shell, where -it redirects /servers/socket/2 (where the system TCP/IP stack resides) onto -$HOME/servers/socket/2 (where the new TCP/IP stack resides). - - $ remap /servers/socket/2 $HOME/servers/socket/2 - $ wget www.gnu.org - -Ideally openvpn would be made to setup pfinet itself, but at least for now it -can be configured by hand like that. - -It would probably be possible to make pfinet able to produce a tap too, would -need some code. - -## IRC, freenode, #hurd, 2013-09-07 - - <d3f> anyone here knows how /dev/net is handled on the hurd? Programs using - it say it's not a directory. I tried creating one and setting a netdde - translator for a tun device in it, but this may be wrong as it doesn't - work - <teythoon> d3f: what does /dev/net do? - <teythoon> ah, its tun/tap stuff... - <d3f> on my gnu/linux it includes a tun device - <teythoon> right - <d3f> I am still reading about the Hurd and try to understand /hurd/netdde - and devnode but by now I am quite sure I will need those to set a tun - networktranslator on /dev/net/tun? - <teythoon> hm, I don't think netdde or devnode will be of any help - <teythoon> afaiui devnode makes mach devices available in the hurdish way, - i.e. available for lookup in the filesystem - <teythoon> d3f: ping youpi if he shows up, he hacked up openvpn to work on - the hurd - <d3f> yeah I know, I talked to him as I am tring to get tinc working on the - Hurd (tinc builds by now). I will give him a shot about creating the - "tun" device - -tun has indeed nothing to do with devnode and netdde, it's pfinet which creates it, as documented above. diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/remap_root_translator.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/remap_root_translator.mdwn deleted file mode 100644 index 8f8668fe..00000000 --- a/service_solahart_jakarta_selatan__082122541663/virtualization/remap_root_translator.mdwn +++ /dev/null @@ -1,157 +0,0 @@ -[[!meta copyright="Copyright © 2013, 2014 Free Software Foundation, Inc."]] - -[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable -id="license" text="Permission is granted to copy, distribute and/or modify this -document under the terms of the GNU Free Documentation License, Version 1.2 or -any later version published by the Free Software Foundation; with no Invariant -Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license -is included in the section entitled [[GNU Free Documentation -License|/fdl]]."]]"""]] - -[[!tag open_issue_hurd]] - -/!\ [[!tag open_issue_documentation]] Does this completely resolve -[[community/gsoc/project_ideas/server_overriding]]? - - -# IRC, freenode, #hurd, 2013-01-05 - - <youpi> so we have a "remap" root translator? - <youpi> I mean this: - <youpi> I'd run my shell in a subhurd whose only difference is that the - root is not the system's root, but my own - <youpi> which catches accesses to /servers/socket/2 for instance - <youpi> but leaves the rest flow through the system's root - <braunr> there is just boot, i don't think it can do that - <youpi> it'd be useful to have that - <youpi> it'd be a very useful feature - <youpi> to use another tcp/ip stack etc. - <braunr> what happens when translators need to locate other translators - used by the client ? - <youpi> can't it tell the client to ask the real system's root? - <youpi> (with the same path) - <youpi> I don't remember the exact reply name - <braunr> hum, it's getting too fuzzy for my head :p - <youpi> well, I mean it's just like translator entries in an ext2fs - <youpi> ext2fs replies "not me, this one" - <braunr> but what if e.g. a user has its own pflocal, and when calling - another translator, that one wants to contact the pflocal used by the - client ? - <youpi> ah, that won't work of course - <braunr> do we actually have such cases btw ? - <braunr> procfs perhaps - <youpi> I don't think we'd want it actually - <braunr> but isn't that required sometimes ? - <youpi> inside a shell script, yes - <braunr> for example, a storeio translator could ask about the priority - properties of the client to proc - <youpi> but I don't remember a case where an external translator would need - the access - <youpi> well, that's actually what we want - <youpi> we don't want to fool the storeio with user-provided data :) - <braunr> yes - <youpi> unless the user starts the storeio himself, in which case he will - have to re-root it - <braunr> so it has to locate the right translator, despite not using the - remap root translator - <youpi> err, it will already - <youpi> by just using the system's path - <braunr> ? - <youpi> maybe you need to say exactly what "it" and "right" are :) - <braunr> ok, let's imagine your previous example with a subhurd and pfinet - <braunr> the remap translator would imply that users from the subhurd - *directly* access all services from the main hurd, except when routed - otherwise by the remap translator to pfinet - <youpi> by "directly", I mean asking the remap translator, which gives as - answer "not me, the root" - <braunr> now, what if a translator in the main hurd wants e.g. network - stats from pfinet, it will ask the main one, not the one obtained through - remap - <braunr> yes - <youpi> that's completely fine - <braunr> ah - <braunr> that's fine if the results don't matter - <youpi> to get network stats from the user pfinet you'd have to be inside - the shell using the remap translator - <braunr> otherwise they're inconsistent - <braunr> yes - <youpi> I don't see why you'd want to get the pfinet stats from outside - <youpi> you mean ethernet board usage? - <braunr> service interactions - <braunr> i can't think of anything relevant with pfinet - <braunr> but imagine pflocal and credentials - <youpi> I believe that'd still be ok - <youpi> i.e. things outside the remap want to know the actual system things - <youpi> while things inside want to know the remapped things - <youpi> and you need that to avoid getting fooled by the user remapping - <braunr> for credentials, i think it works because the client provides - rights, so it would provide rights to the remapped translators in this - case - <braunr> this would need to be generalized - <youpi> I believe it's already general - <braunr> well no - <braunr> procfs for example will always talk to the "true" proc server - <youpi> sure - <youpi> that's what I want from the outside - <youpi> if the user, from the inside, wants another view, he'll have to - start another procfs - <youpi> his own one - <braunr> ok - <youpi> attached to the remapping - - -## IRC, freenode, #hurd, 2013-01-29 - - <youpi> ok, the remap translator was too easy - <youpi> just took fakeroot.c - <youpi> added if (!strcmp("bin/foo", filename)) filename = - "bin/bash"; in - <youpi> netfs_S_dir_lookup - <youpi> and it just works - -[[hurd/interface/dir_lookup]]. - - <youpi> ok, remap does indeed take my own pfinet - <youpi> good :) - <youpi> pfinet's tun seems to be working too - <youpi> it's however not really flexible, it has to show up in /dev/tunx - <youpi> I'll have a look at fixing that - <youpi> yep, works fine - - -## IRC, freenode, #hurd, 2013-02-01 - - <youpi> braunr: as I expected, simply passing FS_RETRY_REAUTH does the - remapping trick - - -# IRC, freenode, #hurd, 2013-02-12 - - <braunr> - http://darnassus.sceen.net/~hurd-web/community/gsoc/project_ideas/server_overriding/ - <braunr> youpi: isn't that your remap translator ? - <youpi> completely - <youpi> remap being (5) - - -# IRC, freenode, #hurd, 2013-02-25 - - <youpi> I'm just having an issue with getcwd getting in the sky - <youpi> I wonder whether libc might need patching to understand it's in - some sort of chroot - <youpi> or perhaps remap fixed into avoiding .. of / being odd - <youpi> erf, it's actually an explicit error - <youpi> libc just doesn't want to have a ".." / being different from CRDIR - <youpi> let me just comment out that :) - <youpi> way better :) - <youpi> yep, just works fine - - -# IRC, freenode, #hurd, 2013-03-16 - - <braunr> youpi: is the /bin/remap --help output correct ? - - -# [[hurd/fsysopts]] - -Doesn't support [[hurd/fsysopts]]. |