summaryrefslogtreecommitdiff
path: root/service_solahart_jakarta_selatan__082122541663/virtualization
diff options
context:
space:
mode:
Diffstat (limited to 'service_solahart_jakarta_selatan__082122541663/virtualization')
-rw-r--r--service_solahart_jakarta_selatan__082122541663/virtualization/capsicum.mdwn22
-rw-r--r--service_solahart_jakarta_selatan__082122541663/virtualization/fakeroot.mdwn1330
-rw-r--r--service_solahart_jakarta_selatan__082122541663/virtualization/file_systems.mdwn24
-rw-r--r--service_solahart_jakarta_selatan__082122541663/virtualization/networking.mdwn88
-rw-r--r--service_solahart_jakarta_selatan__082122541663/virtualization/remap_root_translator.mdwn157
5 files changed, 0 insertions, 1621 deletions
diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/capsicum.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/capsicum.mdwn
deleted file mode 100644
index 44503e34..00000000
--- a/service_solahart_jakarta_selatan__082122541663/virtualization/capsicum.mdwn
+++ /dev/null
@@ -1,22 +0,0 @@
-[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]]
-
-[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
-id="license" text="Permission is granted to copy, distribute and/or modify this
-document under the terms of the GNU Free Documentation License, Version 1.2 or
-any later version published by the Free Software Foundation; with no Invariant
-Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
-is included in the section entitled [[GNU Free Documentation
-License|/fdl]]."]]"""]]
-
-*Capsicum - practical capabilities for UNIX*
-
-<http://www.cl.cam.ac.uk/research/security/capsicum/>
-
-<http://www.lightbluetouchpaper.org/2010/08/12/capsicum-practical-capabilities-for-unix/>
-(server disappeared; [Google
-cache](http://webcache.googleusercontent.com/search?q=cache:cCAqjWOhhksJ:www.lightbluetouchpaper.org/2010/08/12/capsicum-practical-capabilities-for-unix/))
-
-<http://lackingrhoticity.blogspot.com/2010/10/process-descriptors-in-freebsd-capsicum.html>
-
-<http://www.cl.cam.ac.uk/research/security/capsicum/slides/20100811-usenix-capsicum.pdf>
-/ <http://www.youtube.com/watch?v=raNx9L4VH2k>
diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/fakeroot.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/fakeroot.mdwn
deleted file mode 100644
index 441d5c13..00000000
--- a/service_solahart_jakarta_selatan__082122541663/virtualization/fakeroot.mdwn
+++ /dev/null
@@ -1,1330 +0,0 @@
-[[!meta copyright="Copyright © 2010, 2013, 2014 Free Software Foundation,
-Inc."]]
-
-[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
-id="license" text="Permission is granted to copy, distribute and/or modify this
-document under the terms of the GNU Free Documentation License, Version 1.2 or
-any later version published by the Free Software Foundation; with no Invariant
-Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
-is included in the section entitled [[GNU Free Documentation
-License|/fdl]]."]]"""]]
-
-[[!tag open_issue_hurd]]
-
-
-# IRC, freenode, #hurd, 2013-02-26
-
- <youpi> btw, about fakeroot-hurd
- <youpi> the remaining issue I see is with argv[0] (yes, again...)
-
-
-## IRC, freenode, #hurd, 2013-04-03
-
- <youpi> btw, I believe our fakeroot-hurd is close to working actually
- <youpi> it's just a argv[0] issue supposed to be fixed by exec_file_name
- but apparently not fixed in that case, for some reason
-
-[[glibc#execve_relative_paths]].
-
-
-## IRC, freenode, #hurd, 2013-08-26
-
- < teythoon> also I looked into the fakeroot issue, aiui the problem is that
- scripts are not handled correctly, right?
- < teythoon> the exec server fails to locate the scripts file name, and so
- it hands the file_t to the interpreter process and passes /dev/fds/3 as
- script name
- < teythoon> afaics that breaks e.g. python
- < youpi> yes
- < youpi> pinotree's exec_file_name is supposed to fix that, but for some
- reason it doesn't work here
-
-[[glibc#execve_relative_paths]].
-
- < pinotree> it was pochu's, not mine
- < youpi> ah, right
- < teythoon> ah I see, I was wondering about that
- < pochu> it was working for a long time, wasn't it?
- < pochu> and only stopped working recently
- < youpi> did it completely stop?
- < youpi> I have indeed seen odd issues
- < youpi> I haven't actually checked whether it has completely stopped
- working
- < youpi> probably worth looking there first
- < pinotree> gtk+3.0 fails, but other stuff like glib2.0 and gtester-using
- stuff works
- < teythoon> huh? I created tests like "#!/bin/sh\necho $0" and that says
- /dev/fd..., and a python script doing the same doesn't even run, so how
- can it work for a package build?
- < youpi> it works for me in plain bash
- < youpi> #!/bin/sh
- < youpi> echo $0
- < youpi> € $PWD/test.sh
- < youpi> /home/samy/test.sh
- < teythoon> it does !?
- < youpi> yes
- < youpi> not in fakeroot-hurd however, as we said
- < teythoon> well, obviously it works when not being run under
- fakeroot-hurd, yes
- < youpi> ok, so we weren't talking about the same thing
- < youpi> a mere shell script doesn't work in fakeroot-hurd indeed
- < youpi> that's why we still use fakeroot-sysv
- < teythoon> right
- < youpi> err, -tcp
-
-
-## IRC, freenode, #hurd, 2013-11-18
-
- <teythoon> I believe I figured out the argv[0] issue with fakeroot-hurd
- <teythoon> but I'm not sure how to fix this
- <teythoon> first of all, Emilios file_exec_file_name patch set works fine
-
-[[glibc#execve_relative_paths]].
-
- <teythoon> but not with fakeroot
- <teythoon>
- http://git.sceen.net/hurd/hurd.git/blob/HEAD:/exec/hashexec.c#l300
- <teythoon> check_hashexec tries to locate the script file using a heuristic
- <teythoon> Emilios patch improves the situation with just providing this
- information
- <teythoon> but then, the identity port of the "discovered" file is compared
- with the id port of the script file
- <teythoon> to verify if the heuristic found the right file
- <teythoon> but when using fakeroot-hurd, /hurd/fakeroot proxies all
- requests
- <teythoon> but the exec server is outside of the /hurd/fakeroot
- environment, so it gets the id port from the real filesystem
- <teythoon> we could skip that test if the script name is explicitly
- provided though
- <teythoon> that test was meant to see whether a search through $PATH turned
- up the right file
- <braunr> teythoon: nice
- <teythoon> braunr: thanks :)
- <teythoon> unfortunately, dpkg-buildpackaging hurd with it still fails for
- some reason
- <teythoon> but it is faster than fakeroot-tcp :)
- <braunr> even chown ?
- <braunr> or chmod ?
- <teythoon> dunno in detail, but the whole build is faster
- <braunr> if you can try it, i'm interested
- <braunr> because chown/chmod is also slow on linux with fakeroot-tcp
- <teythoon> i can try...
- <braunr> so it's probably not a hurd bug
- <teythoon> braunr: yes, it really is
- <braunr> no i mean
- <braunr> chown/chmod being slow with fakeroot-tcp is probably not a hurd
- bug
- <braunr> but a fakeroot-tcp bug
- <teythoon> chowning all files in /usr/bin takes 5.930s with fakeroot-hurd
- (6.09 with startup overhead) vs 26.42s (26.59s) with fakeroot-tcp
- <braunr> but try it on linux (fakeroot-tcp i mean)
- <braunr> although you may want to do it on something you don't care much
- about :p)
-
-
-## IRC, freenode, #hurd, 2013-12-03
-
- * teythoon is gonna hunt a fakeroot bug ...
- <teythoon> % fakeroot-hurd /bin/sh -c ":> /tmp/some_file"
- <teythoon> /bin/sh: 1: cannot create /tmp/some_file: Is a directory
- <braunr> ah fakeroot-hurd
- <teythoon> prevents installing stuff with /bin/install
- <teythoon> sure fakeroot-hurd, why would i work on the slow one ?
- <braunr> i don't know
- <braunr> because it makes chmod/chown/maybe others horrenddously slow
- <braunr> ?
- <teythoon> yes, fixing this involves fixing fakeroot-hurd
- <braunr> are you sure ?
- <braunr> i prefer repeating just in case: i saw that problem on linux as
- well
- <braunr> with fakeroot-sysv
- <teythoon> so ?
- <braunr> i'm almost certain it's a pure fakeroot bug, not a hurd bug
- <braunr> so
- <teythoon> even if this is fixed, it still has to pay the socket
- communication overhead
- <braunr> fixing fakeroot-hurd so that i can be used instead of fakeroot-tcp
- is a very good thing to do, obviously
- <braunr> it*
- <braunr> but it won't solve the chown/chmod speed
- <braunr> (or, probably won't)
- <teythoon> huh, why not ?
- <braunr> 15:53 < braunr> i'm almost certain it's a pure fakeroot bug, not a
- hurd bug
- <braunr> when i say it's slow, i should be more precise
- <braunr> it doesn't show up in top
- <teythoon> yes, but why would fakeroot-hurd suffer from the same issue ?
- <braunr> the cpu is almost idle
- <braunr> oh right, it's a completely different tool
- <braunr> my bad
- <braunr> right, right, the proper way to implement fakeroot actually :)
- <teythoon> yes
- <teythoon> this will bring near-native speed
-
-
-## IRC, freenode, #hurd, 2013-12-05
-
- <teythoon> fakeroot-hurd just successfully built mig :)
- <teythoon> hangs in dh_gencontrol when building gnumach or hurd though
- <teythoon> i believe it hangs waiting for a lock
- <teythoon> lock like in file lock that is
- <teythoon> braunr: no more room for vm_map_find_entry in 80220a40
- <teythoon> 80220a40 <- is that a task ?
- <braunr> or a vm_map, not sure
- <braunr> probably a vm_map
-
-
-## IRC, freenode, #hurd, 2013-12-06
-
- <teythoon> well, aren't threads a source of endless entertainment ... ?
- <teythoon> well, I found three more bugs in fakeroot-hurd
- <teythoon> one of them requires fixing the locking used in fakeroot
- <braunr> ouch
- <teythoon> the current code does some lock cycling to aquire a lock out of
- order
- <braunr> cycling ?
- <teythoon> in the netfs_node_norefs function
- <teythoon> release and reaquire
- <braunr> i see
- <teythoon> which imho should be better solved with a weak reference
- <teythoon> working on it, it no longer deadlocks but i broke something else
- ...
- <teythoon> endless fun ;)
- <braunr> such things could have been done right in the beginning
- <braunr> ...
- <teythoon> yes, I wonder
- <teythoon> libports has weak references
- <teythoon> but pflocal is the only user
- <braunr> hm
- <teythoon> none of the lib*fs support that
- <braunr> didn't i add one in libdiskfs too ?
- <braunr> anyway, irrelevant
- <braunr> weak references are a nice feature
- <braunr> teythoon: i don't see the cycling you mentioned
- <braunr> only netfs_node_refcnt_lock being dropped temporarily
- <teythoon> yep, that one
- <teythoon> line 145
- <teythoon> note that due to another bug this code is currently never run
- <braunr> how surprising ..
- <braunr> the note about some leak actually gave a hint about that
- <teythoon> yeah, that leak
- <teythoon> I think i'm actually very close
- <teythoon> it's just so frustrating, i thought i got it last night
- <braunr> good luck then
- <teythoon> thanks :)
-
-
-## IRC, freenode, #hurd, 2013-12-09
-
- <teythoon> sweet, i fixed fakeroot-hurd :)
- <braunr> /clap
- <braunr> what was the problem ?
- <teythoon> lots
- <braunr> i see
- <teythoon> it's amazing it actually run as well as it did
- <braunr> mess strikes again
- <braunr> i hate messy code ..
- * teythoon is building half a hurd package using this ... stay tuned ;)
- <azeem> teythoon: is this going to make building faster as well?
- <teythoon> most likely, yes
- <teythoon> fakeroot-tcp is known to be slow, even on linux
- <braunr> teythoon: are you sure about the transparent retry patch ?
- <teythoon> pretty sure, why ?
- <braunr> it's about a more general issue that we didn't fix yet
- <braunr> our last discussions about it lead us to agree that clients should
- check the identity of a server before interacting with it
- <teythoon> braunr: i don't understand, what's the problem here ?
- <braunr> teythoon: fakeroot does the lookup itself, doesn't it ?
- <teythoon> yes
- <braunr> teythoon: but was that also the case before your patch ?
- <teythoon> braunr: yes
- <braunr> teythoon: then ok
- <braunr> teythoon: i guess fakeroot handles requests only for a specific
- set of calls right ?
- <braunr> and for others, requests are directly relayed
- <teythoon> braunr: yes
- <braunr> and that still is the case, right ?
- <teythoon> yes
- <braunr> ok
- <braunr> looks right since it only affects lookups
- <braunr> ok then
- <teythoon> well, fakeroot-hurd built half a hurd package in less than 70
- minutes
- <teythoon> a new record for my box
- <braunr> compared to how much before ?
- <braunr> (and why half of it ?)
- <teythoon> unfortunately it hung after signing the packages... some perl
- process with a /usr/bin/tee child
- <teythoon> killing tee made it succeed though
- <teythoon> braunr: i don't build the udeb package
- <braunr> oh ok
- <teythoon> braunr: compared with ~75 with fakeroot-tcp and my demuxer
- rework, ~80 before
- <braunr> teythoon: nice
-
-
-## IRC, freenode, #hurd, 2013-12-18
-
- <teythoon> there, i fixed the last fakeroot-hurd bug
- <teythoon> *whee* :)
- <teythoon> i thought so many times that i got the last fakeroot bug ...
- <teythoon> last as in it's in a good enough shape to compile the hurd
- package that is
- <teythoon> but now it is
- <braunr> :)
- <braunr> this will make glibc and others so much faster to build
-
-
-## IRC, freenode, #hurd, 2013-12-19
-
- <braunr> teythoon_: hum, you should make the behaviour of fakeroot-hurd on
- the last client exiting optional
- <teythoon_> y?
- <teythoon_> fakeroot-tcp does the very same thing
- <braunr> fakeroot-hurd is different
- <braunr> it's part of the file system
- <teythoon_> yes
- <braunr> users may want it to stay around
- <braunr> and reuse it without checking it's actually there
- <teythoon_> but once the last client is gone, who is ever getting another
- port to it ?
- <teythoon_> no
- <teythoon_> that cannot happen
- <braunr> really ?
- <teythoon_> yes
- <braunr> i thought it was like remap
- <braunr> since remap is based on it
- <teythoon_> the same thing applies to remap
- <teythoon_> only settrans has the control port
- <braunr> hum
- <teythoon_> and uses it once to get a protid for the working dir of the
- initial process started inside the chrooted environment
- <braunr> you may not want to chroot inside
- <teythoon_> so ?
- <teythoon_> then, you get another protid
- <braunr> i'll make an example
- <braunr> i create a myroot directory implemented by fakeroot
- <braunr> populate it
- <braunr> leave and do something else,
- <braunr> i might want to return to it later
- <teythoon_> ah
- <teythoon_> ok, so you are not using settrans --chroot
-
-[[hurd/settrans/discussion#chroot]].
-
- <braunr> or maybe i'm confusing the fakeroot translator and fakeroot-hurd
- <braunr> 10:48 < braunr> you may not want to chroot inside
- <braunr> yes
- <teythoon_> hm
- <teythoon_> ok, so the patch could be changed to check whether the last
- control port is gone too
- <braunr> i have no idea of any practical use, but i don't see a valid
- reason to make a translator go away just because it has no client
- <braunr> except for resource usage
- <braunr> and if it's installed as a passive translator
- <braunr> although that would make fakeroot loose its state
- <braunr> though remap state is on the command line so it would be fine for
- it
- <braunr> see what i mean ?
- <teythoon_> yes i do
- <braunr> fakeroot state could be saved in some db one day so it may apply,
- if anyone feels the need
- <teythoon_> so what about checking for control ports too ?
- <braunr> i'm not too familiar with those
- <braunr> who has the control port of a passive translator ? the parent ?
- <teythoon_> that should cover the use case you described
- <teythoon_> for the parent translator
- <teythoon_> for fsys_getroot requests it has to keep it around
- <teythoon_> and for more fsys stuff too
- <braunr> and if active ? settrans ? who just looses it ?
- <teythoon_> if settrans is used to start an active translator, the parent
- fs still gets a right to the control port
- <braunr> ok
- <braunr> i don't have a clear view of what this implies for fakeroot-hurd
- <braunr> we'd want fakeroot-hurd to clean all resources including the
- fakeroot translator on exit
- <teythoon_> for fakeroot-hurd (or any child translator) this means that a
- port from the control port class will still exists
- <teythoon_> so we do not exit
- <teythoon_> oh, you're speaking of fakeroot.sh ? the wrapper script ?
- <braunr> probably
- <braunr> for me, fakeroot-hurd is the command line too, similar to
- fakeroot-sysv and fakeroot-tcp
- <braunr> and fakeroot is the translator
- <teythoon_> yes, agreed
- <teythoon_> fakeroot-hurd could use settrans --force --chroot ... to force
- fakeroot to exit if the main chrooted process dies
- <teythoon_> but that'd kill anything that outlives that process
- <teythoon_> that might be legitimate, say a process daemonized
- <teythoon_> so detecting that noone uses fakeroot is the much cleaner
- solution
- <braunr> ok
- <teythoon_> also, that's what fakeroot-tcp does
- <braunr> which is why i suggested an option for that
- <teythoon_> why add an option if we can do the right thing without
- troubling the user ?
- <braunr> ah, if we can, good
- <teythoon_> i think we can
- <teythoon_> I'll rework the patch, thanks for the hint
- <braunr> so
- <braunr> just to be clear
- <braunr> the way you intend it to work is
- <braunr> wait for all clients and the control port to drop before shutting
- down
- <braunr> the control port is dropped when dettaching the translator, right
- ?
- <teythoon_> yes
- <braunr> but hm
- <braunr> what if clients spawn other processes ?
- <braunr> they won't find the translator any more
- <teythoon_> then, that client get's a port to fakeroot at least for it's
- working dir
- <teythoon_> so another protid is created
- <braunr> ah yes, it's usually choorted for such uses
- <braunr> chrooted
- <teythoon_> so fakeroot will stick around
- <braunr> but clients, even from fakeroot, might simply use absolute paths
- <teythoon_> so ?
- <braunr> in which case they won't find fakeroot
- <teythoon_> it will hit fakeroots dir_lookup
- <teythoon_> sure
- <braunr> how so ?
- <teythoon_> if the path is absolute, it will trigger a magic retry of some
- kind
- <teythoon_> so the client uses it's root dir port
- <braunr> i thought the lookup would be done straight from the root fs port
- ..
- <teythoon_> which points to fakeroot of course
- <braunr> ah, chrooted again
- <teythoon_> that's the whole point
- <braunr> so this implies clients are chrooted
- <teythoon_> they are
- <teythoon_> even if you do another chroot
- <braunr> what i mean is
- <teythoon_> that root port also points to a fakeroot port
- <braunr> if we detach the translator, and clients outside the chroot spawn
- processes, say shell scripts, they won't find the fakeroot tree
- <braunr> now, i wonder if we want to actually handle that
- <braunr> i'm just uncomfortable with a translator silently shutting down
- because it has no client
- <teythoon_> if fakeroot is detached, how are clients outside the chroot
- ever supposed to get a handle to files inside the fakerooted env ?
- <braunr> it makes sense for fakeroot, so the expected behaviours here aer
- conflicting
- <braunr> they had those before fakeroot being detached
- <teythoon_> then fakeroot wouldn't go away
- <braunr> right
- <braunr> unless there is a race but i don't think there is
- <teythoon_> there isn't
- <teythoon_> i call netfs_shutdown
- <braunr> clients get the rights before the parent has a chance to terminate
- <teythoon_> and only shutdown if it doesn't return ebusy
- <braunr> makes sense
- <braunr> ok go ahead :)
- <teythoon_> cool, thanks for the walk-through ;)
- <braunr> on the other hand ..
- <braunr> that's a complicated topic left unfinished by the original authors
- <teythoon_> one of many
- <braunr> having translators automatically go away when there is no client
- may be a good feature
- <braunr> but it only makes sense for passive translators
- <braunr> and this should be automated
- <braunr> the lib*fs libraries should be able to handle it
- <teythoon_> or, we could go for proper persistence instead
- <braunr> stay around if active, leave after a while when no more clients if
- passive
- <braunr> why ?
- <teythoon_> clean solution
- <braunr> persistence looks much more expensive to me
- <teythoon_> other benefits
- <braunr> i mean
- <braunr> persistence looks so expensive it doesn't make sense in a general
- purpose system
- <teythoon_> sure, we could make our *fs libs handle this smarter at a much
- lower cost
- <teythoon_> don't we get a handle to the underlying file ?
- <braunr> i think we do yes
- <teythoon_> if that's actually a file and not a directory, we could store
- data into it
- <braunr> many translators are read-only
- <teythoon_> so ?
- <braunr> well, when we can write, we can use passive translators instead
- <braunr> normally
- <teythoon_> yes
- <braunr> depends on the fs type actually but you're right, we could use
- regular files
- <braunr> or a special type of file, i don't know
- <antrik> braunr: BTW, I agree that active translators should only go away
- when no ports are open anymore, while passive ones can exit when control
- ports are still open but no protids
- <teythoon> antrik: you mean as a general rule ?
- <teythoon> that leaves the question how the translator distinguishes
- between having a passive translator record and not having one
- <antrik> I believe I already arrived at that conclusion in some design
- discussion, probaly regarding namespace-based translator selection
- <antrik> teythoon: yeah, as a general rule
- <teythoon> interesting
- <antrik> currently there are command line arguments controling timeouts,
- but they don't consider control ports IIRC
- <teythoon> i thought there are problems with shutting down translators in
- general
- <antrik> (also, command line arguments seem inconvenient to distinguish the
- passive vs. active case...)
- <teythoon> yeah, but we disregard the timeouts in the debian flavor of hurd
- <antrik> teythoon: err... no we don't. at least not last time I knew. are
- you confusing this with thread timeouts?
- <antrik> simple test: do ls -l on /dev, wait a few minutes, compare
- <teythoon> what do you expect will happen ?
- <antrik> the unused translators should go away
- <teythoon> no
- <antrik> that must be new then
- <teythoon> might be, yes
- <teythoon>
- http://darnassus.sceen.net/gitweb/teythoon/packaging/hurd.git/blame/HEAD:/debian/patches/libports_stability.patch
- <braunr> antrik: debian currently disables both the global and thread
- timeouts in libports
- <braunr> my work on thread destruction consists in part in reenabling
- thread timeouts, and my binary packages do that well so far :)
- <antrik> braunr: any idea why the global timeouts were disabled?
-
-
-## IRC, freenode, #hurd, 2013-12-20
-
- <braunr> antrik: not sure
- <braunr> but i suspect there could be races
- <braunr> if a message arrives while the server is going away, i'm not sure
- the client can determine this and retry transparently
- <antrik> good point... not sure how that is supposed to work exactly
-
-
-## IRC, freenode, #hurd, 2013-12-31
-
- <braunr> btw, we should remove the libports_stability patch and directly
- change the upstream code
- <braunr> if you agree, i can force the global timeout to 0 (because we're
- still not sure what can go wrong when a translator goes away while a
- message is being delivered to it)
- <braunr> i didn't experience any slowdown with thread destruction however
- <braunr> so i'm tempted to set that to an actual reasonable timeout value
- of 30-300 seconds
- <teythoon> braunr: if you do, please introduce a macro for the default
- value so it can be changed easily
- <braunr> teythoon: yes
- <braunr> i don't understand why these are left as parameters tbh
- <teythoon> true
- <braunr> 30 seconds seems to be plenty enough
-
-
-## IRC, freenode, #hurd, 2014-01-17
-
- <braunr> time to give fakeroot-hurd a shot
- <braunr> http://darnassus.sceen.net/~rbraun/darnassus_fakeroot_hurd_assert
- <teythoon> braunr: (wrt fakeroot-hurd) well in my book that shouldn't
- happen
- <teythoon> that's why i put the assertion there ;)
- <braunr> i assumed so :)
- <teythoon> then again, /me does not agree with "threads" as concurrency
- model >,<, and that feeling seems to be mutual :p
- <braunr> ?
- <teythoon> well, obviously, the threads do not agree with me wrt to that
- assertion
- <braunr> the threads ?
- <teythoon> well, fakeroot is a multithreaded server
- <braunr> teythoon: i'm not sure i get the point, are you saying you're not
- comfortable with threads ?
- <teythoon> that's exactly what i'm saying
- <braunr> ok
- <braunr> coroutines/functional i guess ?
- <teythoon> csp
- <teythoon> functional not so much
-
-
-## IRC, freenode, #hurd, 2014-01-20
-
-[[open_issues/libpthread]],
-[[open_issues/libpthread/t/fix_have_kernel_resources]].
-
- <braunr> teythoon: it's perfectly possible that the bug i had with
- fakeroot-hurd have been caused by my own glibc thread related patches
- <braunr> has*
- <teythoon> ok
- <teythoon> *phew* :p
- <braunr> :)
- <teythoon> i wonder if youpi could reproduce his issue on his machine
- <braunr> what issue ?
- <braunr> i must have missed something
- <teythoon> some package failed
- <teythoon> but he didn't gave any details
- <teythoon> he wanted to try it on his vm first
- <braunr> ok
-
-
-## IRC, freenode, #hurd, 2014-01-21
-
- <braunr> teythoon: i still get the same assertion failure with
- fakeroot-hurd
- <braunr> will take a look at that sometimes too
- <teythoon> braunr: hrm :/
- <braunr> teythoon: don't worry, i'm sure it's nothing big
- <braunr> in the mean time, there are updated hurd and glibc packages on my
- repository with fixed tls and thread destruction
- <teythoon> cool :)
-
-
-## IRC, freenode, #hurd, 2014-01-23
-
- <braunr> teythoon: can you briefly explain this fake reference thing in
- fakeroot when you have some time please ?
- <teythoon> braunr: fakeroot creates ports to hand out to clients
- <teythoon> every port represents a node and references a real node
- <teythoon> fakeroot allows one to set attributes, e.g. file permissions on
- any node as if the client was root
- <teythoon> those faked attributes are stored in the node objects
- <braunr> let's focus on fake_reference please
- <teythoon> once some attribute is faked, that node has to be kept alive
- <teythoon> otherwise, that faked information is lost
- <teythoon> so if the last peropen object is closed and some information is
- faked, a fake reference is kept
- <teythoon> as indicated by a flag
- <braunr> hm
- <teythoon> in dir lookup, if a node is looked-up that has a fake reference,
- it is recycled, i.e. the flag cleared and the referecne count is not
- incremented
- <teythoon> so every time fakeroot_netfs_release_protid is called b/c, the
- node in question should not have the fake reference flag set
- <braunr> what's the relation between the number of hard links and this fake
- reference ?
- <teythoon> i don'
- <teythoon> i don't think fakeroot has a notion of 'hard links'
- <braunr> it does
- <braunr> the fake reference is added on nodes with a hard link count
- greater than 0
- <braunr> but i guess that just means the underlying node still exists
- <teythoon> ah yes
- <teythoon> right
- <teythoon> currently, if the real node is deleted, the fake node is still
- kept around
- <braunr> let's say it's ok for now
- <teythoon> that's what the comment is talking about, the one that indicates
- that garbage collection could help here
- <teythoon> yes
- <teythoon> properly fixing this is difficult
- <braunr> agreed
- <braunr> it would require something like inotify anyway
- <teythoon> b/c of the way file deletion works
- <braunr> let's just ignore the issue, that's not what i'm hunting
- <teythoon> agreed
- <braunr> the assertion i have is telling us that we're dropping a fake
- reference
- <braunr> are we certain this isn't possible ?
- <teythoon> that function is called if a client dereferences a port
- <teythoon> in order to have a port in the first place, it has to get it
- from a dir_lookup
- <teythoon> the dir lookup turns a fake reference into a real one
- <teythoon> so i'm certain of that (barring a race condition somewhere)
- <braunr> ok
- <braunr> netfs_S_dir_lookup grabs idport_ihash_lock (line 354) but doesn't
- release it if nn == NULL (lines 388-392)
- <teythoon> hm, my file numbers are slightly different o_O
- <braunr> i have printfs around
- <braunr> sorry :)
- <teythoon> ok
- <teythoon> new node unlocks it
- <teythoon> new_node
- <braunr> oh
- <braunr> how unintuitive ..
- <teythoon> yes, don't blame me ;) that's how it was
- <braunr> :)
- <braunr> worse, the description says "if successful" ..
- <braunr> ah no, the node lock
- <braunr> ok
- <teythoon> yes, badly worded description
- <braunr> i strongly doubt it's a race
- <teythoon> how do you trigger that assertion failure ?
- <braunr> dpkg-buildpackage -rfakeroot-hurd -uc -us
- <braunr> for the hurd package
- <braunr> very similar to one of your test cases i think
- <teythoon> umm :-/
- <braunr> one thing that i find confusing is that fake_reference seems to
- apply to nodes, whereas release_protid is about, well, protids
- <braunr> is there a 1:1 relationship ?
- <braunr> since there is a peropen in the protid, i assume not
- <braunr> it may be a race actually
- <braunr> np->references must be accessed with netfs_node_refcnt_lock locked
- <braunr> hm no, that's not it
- <teythoon> no, it's not a 1:1 relationship
- <teythoon> note that the lock idport_ihash_lock serializes most operations,
- despite it's name indicating that it's only for the hash table
- <teythoon> the "interesting" operations being dir_lookup and release_protid
- <braunr> yes
- <braunr> again, that's another issue
- <teythoon> why ? that's a pretty strong guarantee already
- <braunr> ah yes, i was referring to scalability
- <teythoon> sure
- <braunr> the assertion is triggered from ports_port_deref in
- ports_manage_port_operations_multithread
- <teythoon> but i found it hard to reason about fakeroot, there are multiple
- locks involved, two kinds of reference counting across different libs
- <braunr> yes
- <teythoon> yes, that's to be expected
- <braunr> teythoon: do we agree that the fake reference is reused by a
- protid ?
- <teythoon> braunr: yes
- <braunr> why is there a ref counter for the protid as well as the peropen
- then ? :/
- <teythoon> funny... i thought there was no refcnt for the peropen objects,
- but there is
- <teythoon> but for fakeroot-hurd that shouldn't matter, right ?
- <braunr> i don't know
- <teythoon> here, one protid object is associated with one peropen object
- <braunr> yes
- <teythoon> and the other way around, i.e. it's 1:1
- <teythoon> so the refcount for those should be identical
- <braunr> but i get a case where protid has a refcnt of 0 while the peropen
- has 2 ..
- <teythoon> umm, that doesn't sound right
- <braunr> teythoon: ok, it does look like a race on np->references
- <braunr> node references are protected by a global lock in lib*fs libs
- <teythoon> yes
- <braunr> you check it without holding it
- <braunr> which means another protid can be closed at the same time, setting
- the flag on the underlying node
- <braunr> i'll make a proper patch soon
- <teythoon> they cannot both hold the hash lock
- <braunr> hm
- <braunr> teythoon: actually, i don't see why that's relevant
- <braunr> one thread closes its protid, sets the fakeref flag
- <braunr> the other does the same, chokes on the assertion
- <braunr> serially
- <teythoon> i'm always a little fuzzy when exactly the references get
- decremented
- <teythoon> but shouldn't only the second thread set the fakeref flag ?
- <braunr> well, that's not what i see
- <braunr> i'll check what happens to this ref counter
- <teythoon> see how my release_protid function calls netfs_release_protid
- just after the out label
- <teythoon> *while holding the big hash lock
- <teythoon> so, any refcounting should happen while the lock is being held,
- no ?
- <braunr> perhaps
- <braunr> now, my logs show something new
- <braunr> a case where the protid being released was never printed before
- <braunr> i.e. not obtained from dir_lookup
- <braunr> or at least, not fakeroot dir_lookup
- <teythoon> huh, where did it came from then ?
- <braunr> no idea
- <teythoon> only dir_lookup hands out those
- <braunr> check_openmodes calls dir_lookup too
- <teythoon> yes, but that's not our dir_lookup
- <braunr> that's what i mean
- <braunr> it bypasses fakeroot's custom dir_lookup
- <braunr> but i guess the reference already exists at this point
- <teythoon> bypass ? i wouldn't call it that
- <braunr> you're right, wrong wording
- <teythoon> it accesses files on other translators
- <braunr> yes
- <braunr> the netnode is already present
- <teythoon> yes
- <braunr> could it be the root node ?
- <teythoon> i do not believe so
- <teythoon> the root node is always faked
- <teythoon> and is handed out to the first process in the fakeroot env for
- it's current directory port
- <teythoon> so you could try something that chdirs away to test that
- hypothesis
- <braunr> the assertion looks triggered by a chdir
- <teythoon> how do you know that ?
- <braunr> dh_auto_install: error: unable to chdir to build-deb
- <teythoon> ah
- <teythoon> well, or that is just the operation after fakeroot died and
- completely unrelated
- <braunr> maybe
- <teythoon> can you trigger this reliably ?
- <braunr> yes
- <braunr> i'm trying to write a shell script for that
- <teythoon> so for you, fakeroot-hurd never succeeded in building a hurd
- package ?
- <braunr> no
- <teythoon> on darnassus ?
- <braunr> yes
- <teythoon> b/c i stopped working on fakeroot-hurd when it was in a
- good-enough shape to build the hurd package
- <teythoon> >,<
- <teythoon> maybe my system is not fast enough to hit this race (if it turns
- out to be one)
- <braunr> some calls seems to decrease the refcount of the root node
- <braunr> call*
- <teythoon> have you confirmed that it's the root node ?
- <braunr> almost
- <braunr> i could say yes
- <braunr> teythoon: actually no, it's not ..
- <braunr> could be ..
- <braunr> teythoon: on what node does fakeroot-hurd install the fakeroot
- translator when used to build debian packages ?
- <braunr> hum
- <braunr> could it simply be that the check on np->references should be
- moved above the assertion ?
- <teythoon> braunr: it is not bound to any node, check settrans --chroot
-
-[[hurd/settrans/discussion#chroot]].
-
- <braunr> oh right
- <braunr> teythoon: ok i mean
- <braunr> does it shadow / ?
- <braunr> looks very likely, otherwise the chroot wouldn't work
- <teythoon> i'm not sure what you mean by shadow
- <braunr> settrans --chroot cmd -- / /hurd/fakeroot ?
- <teythoon> but yes, for any process in the chroot-like env every real node
- is replaced, including /
- <braunr> makes sense
- <braunr> teythoon: moving the assertion seems to fix the issue
- <braunr> intuitively, it seems reasonable to assume the fakeref flag can
- only be set when there is only one reference, namely the fake reference
- <braunr> (well, the fake ref, recycled by the last open)
- <teythoon> no, i don't follow
- <teythoon> i'd still say, that if ...release_protid is called, then there
- is no way for the fake flag to be set in the first place
- <teythoon> that's why i put the assertion in ;)
- <braunr> on the other hand, you check the refcnt precisely because other
- threads may have reacquired the node
- <teythoon> but why would moving the assertion change anything ?
- <teythoon> if we would do that, we'd "lose" all threads that see
- np->reference being >1
- <teythoon> but for those objects the fake_reference flag should never be
- set anyways
- <teythoon> i cannot see why this would help
- <teythoon> (does it help ?)
- <teythoon> (and if it does, it points to a serious problem imho)
- <braunr> i'm recreating the traces that made me think that
- <braunr> to get a clearer view of what's happening
- <braunr> the problem i have with the current code is this
- <braunr> there can be multiple protid referring to the same node occurring
- at the same time
- <braunr> they are serialized by the hash table lock, ok
- <braunr> but there apparently are cases where the first (of two) protids
- being closed sets the fakeref flag
- <braunr> and the following chokes because the flag is set
- <braunr> i assume you put this refcount check because you assumed only the
- last protid being closed can set the flag, right ?
- <braunr> but then, why > 1 ? why not > 0 ?
- <teythoon> yes, that's what i was trying to assert
- <teythoon> b/c the 1 is our reference
- <braunr> which one exactly ?
- <teythoon> >1 is anyone *beside* us
- <teythoon> ?
- <braunr> hm
- <braunr> you mean the reference held by the protid being destroyed
- <teythoon> yes
- <braunr> isn't that reference already dropped before calling the cleanup
- function ?
- <braunr> ah no, it's the node ref
- <teythoon> yes
- <braunr> released by netfs_release_protid
- <teythoon> exactly
- <braunr> which is called without the hash table lock held
- <braunr> hm no
- <braunr> it's locked
- <braunr> damn my brain is slow today
- <teythoon> i actually think that it's the combination of manual reference
- counting and the primitive concurrency model that makes it hard to reason
- about this
- <braunr> well
- <braunr> the model is quite simple too
- <braunr> accesses to refcounters must be protected by the appropriate lock
- <braunr> this isn't done here, on the assumption that all referencing
- operations are protected by another global lock all the time
- <teythoon> even if a model is simple, this does not mean that it is a good
- model for human beings to comprehend and reason about
- <braunr> i don't know
- <braunr> note that netfs_drop_node is designed to be called with
- netfs_node_refcnt_lock locked
- <braunr> implying the refcount must remain stable between checking it and
- dropping the node
- <braunr> netfs_make_peropen is called without the hash table lock held in
- dir_lookup
- <braunr> and this increases the refcount
- <braunr> although the problem is rather that something decreases it without
- the lock held
- <teythoon> we should port libtsan and just ask gcc -fsanitize=thread
- <braunr> what about the netfs_nput call at the end of dir_lookup ?
- <braunr> the fake ref should be set by the norefs function
- <teythoon> that should not decrease the count to 0 b/c the caller holds a
- reference too
- <braunr> yes that's ugly
- <braunr> ugh
- <braunr> i'm unable to think clearly right now
- <teythoon> as mentioned in the commit message, you cannot do something like
- this in the norefs function
- <teythoon> bbl ;)
- <braunr> bye teythoon
- <braunr> thanks for your time
- <braunr> for when you come back :
- <braunr> instead of maintaining this "fake" reference, why not assumeing
- the hash table holds a reference, and simply count it
- <braunr> the same way a cache does
- <braunr> and drop that reference when removing a node, either to reflect
- the current state of the underlying node, or because the translator is
- being shut down ?
- <braunr> why not assume*
- <braunr> bbl too
- <teythoon> sure, refactoring is definitively an option
-
-
-## IRC, freenode, #hurd, 2014-01-24
-
- <braunr> teythoon: ok, i'll take care of fakeroot
- <teythoon> braunr: thanks. tbh i was a little fed up with that little
- bugger >,<
- <braunr> i can imagine
- <braunr> considering the number of patches you've sent already
-
- <braunr> teythoon: are you sure about your call to fshelp_lock_init ?
- <teythoon> yes, why do you ask ?
- <teythoon> (the test case is given in the commit message)
- <braunr> it doesn't look right to me to call "init" while the node is
- potentially locked
- <braunr> i noticed libdiskfs peropen release function takes care of
- releasing locks
- <braunr> it looks better to me
- <teythoon> it's not about releasing the lock
- <teythoon> it's about faking the file being closed which implicitly
- releases the lock
- <braunr> the file is being close
- <braunr> closed
- <braunr> since it's in the cleanup function
- <teythoon> yes, but we keep it b/c the file has faked attributes
- <teythoon> did you look at the problem description in the commit message ?
- <braunr> we keep the node
- <braunr> not the peropen
- <teythoon> so ?
- <teythoon> the lock is in the node
- <braunr> why would libdiskfs do it in the peropen release then ?
- <braunr> there is an inconsistency somwhere
- <braunr> actually, the lock looks to be per open
- <braunr> or rather, the lock is per node, but its status is recorded per
- open
- <braunr> allowing the implementation to track if a file descriptor was used
- to install a lock and release it when that file descriptor goes away
- <teythoon> why would the node be locked ?
- <teythoon> locked in what way, file-locking locked ?
- <braunr> yes
- <braunr> posix explicitely says that file locks must be implicitely removed
- when closing the file descriptor used to install them, so that makes
- sense
- <teythoon> isn't hat exactly what i'm doing ?
- <braunr> no
- <braunr> you're initializing the file lock
- <braunr> init != unlock
- <braunr> and it's specific to fakeroot, while it looks like libnetfs should
- be doing it
- <teythoon> libnetfs would do it
- <teythoon> but we prevent that by keeping the node alive
- <braunr> again, it's a per open thing
- <braunr> and no, libnetfs doesn't release locks implicitely in the current
- version
- <teythoon> didn't we agree that for fakeroot one peropen object is
- associated with one protid object ?
- <braunr> yes
- <braunr> and don't keep those alive
- <braunr> so let them die peacefully, and fix libnetfs so it releases the
- lock as it's supposed to
- <braunr> and we* don't
- <teythoon> we don't keep those alive
- <teythoon> why would we ?
- <braunr> yes that's what i wanted to say
- <braunr> what i mean is
- <braunr> since letting peropens die is already what is being done
- <braunr> there is no need for a special handling of locks in fakeroot
- <teythoon> oh
- <braunr> on the other hand, libnetfs must be fixed
- <teythoon> ok, that might very well be true
- <teythoon> (we need to bring libnetfs and diskfs closer so that they can be
- diff'ed easily)
- <braunr> i just wanted to check your reason for using lock_init in the
- first place
- <braunr> yes ..
- <braunr> teythoon: also, i think we actually do have what's necessary to
- deal with garbage collection
- <braunr> namely, dead-name notifications
- <braunr> i'll see if i can cook something simple enough
- <braunr> otherwise, merely keeping every node around is also acceptable
- considering the use cases
- <teythoon> dead-name notifications won't help if the real node disappears,
- no ?
- <braunr> teythoon: dead name notifications on the real node port :)
- <braunr> teythoon: at least i can reliably build the hurd package using
- fakeroot-hurd now
- <braunr> let's try glibc :)
-
-## IRC, freenode, #hurd, 2014-01-25
-
- <teythoon> braunr: awesome :)
- <braunr> teythoon: hm not sure :/
- <braunr> darnassus got oom
- <braunr> teythoon: could be unrelated though
- <braunr> teythoon: something has apprently made /home unresponsive :(
- <braunr> teythoon: i suspect bots hitting apache and in particular the git
- repositories to have increased memory usage
-
-
-## IRC, freenode, #hurd, 2014-01-26
-
- <braunr> teythoon: btw, fakeroot interacts very very badly with other netfs
- file systems
- <braunr> e.g., listing /proc through it creates lots of nodes
- <braunr> i'm not yet sure how to fix that
- <braunr> using a dead name notification doesn't seem appropriate (at least
- not directly) because fakeroot holds a true reference that prevents the
- deallocation of the target node
-
-
-## IRC, freenode, #hurd, 2014-01-27
-
- <braunr> teythoon: good news (more or less): fakeroot is actually leaking a
- lot when crossing file systems
- <braunr> which means if i fix that, there is a good chance we can use it to
- build all packages with it
- <braunr> -with it
- <teythoon> what do you mean exactly ?
- <braunr> if target nodes are from /, there is no such leak
- <braunr> as soon as the target nodes are from another file system, ports
- rights are leaked
- <braunr> that's what fills the kernel allocator actually
- <teythoon> oh, so dir_lookup leaks ports when crossing translator
- boundaries ?
- <braunr> seems so
- <teythoon> yeah, that might very well be it
- <teythoon> the dir_lookup logic in lib*fs is quite involved :/
- <braunr> yes, my simple attempts were unsuccessful
- <braunr> but i'm confident i can fix it soon
- <teythoon> that sounds good :)
- <braunr> i also remove the fake_ref flag and replace it with "accounting
- the reference in the hash table" as soon as a node is faked
- <teythoon> fine with me
- <braunr> these will be the expected leak
- <braunr> but they're far less in numbers than what i observe
- <braunr> and garbage collection can be implemented later
- <braunr> although i would prefer notifications a lot more
- <braunr> end of the news, bbl :)
- <braunr> found it :>
- <teythoon> braunr: -v ;)
- <braunr> err = dir_lookup (...);
- <braunr> if (dir != dnp->nn->file) mach_port_deallocate (mach_task_self (),
- dir);
- <braunr> in other words, deallocate ports for intermediate file system root
- directories .. :)
- <braunr> teythoon: currently building hurd and glibc packages
- <braunr> but i intend to improve some more with the addition of a default
- faked state
- <braunr> so that only nodes with modified faked states are retained
- <teythoon> how do you mark nodes as having the default faked state ?
- <braunr> i don't
- <teythoon> ok, right, makes sense :)
- <teythoon> this sounds awesome, thanks for following up on this
- <braunr> i'm quite busy with other stuff so, with proper testing, it should
- take me the week to get merged
- <braunr> teythoon: well thanks for all the fixes you've done
- <braunr> fakeroot was completely unusable before that
- <teythoon> if you push your changes somewhere i'll integrate them into my
- packages and test them
- <braunr> ok
- <braunr> implementing fakeroot -u could also be a good thing
- <braunr> and this should work easily with that default faked state strategy
-
-
-## IRC, freenode, #hurd, 2014-01-28
-
- <braunr> teythoon: i should be able to test fakeroot-hurd with the default
- faked attributes strategy today on glibc
- <teythoon> braunr: very nice :)
- <braunr> azeem_: do you happen to know if fakeroot -u is used by debian ?
- <braunr> i mean when building packages
- <teythoon> braunr: how does fakeroot-hurd perform on darnassus ?
- <teythoon> i mean, does it yield a noticeable improvement over fakeroot-tcp
- just like on my slow box ?
- <braunr> i'm not measuring that :/
- <teythoon> ok, no problem
- <braunr> and since nodes are removed from the hash table, performance might
- decrease slightly
- <braunr> but the number of rights is kept very low, as expected
- <teythoon> that's good
- <braunr> i keep seeing leaks though
- <braunr> when switching cwd between file systems
- <teythoon> humm
- <braunr> so i assume something is wrong with the identity of . or ..
- <braunr> it's so insignificant compared to the previous problems that i
- won't waste time on that
- <braunr> teythoon: the problem with measuring on darnassus is that it's a
- public machine
- <teythoon> right
- <braunr> often scanned by ssh worms or http bots
-
-[[cannot_create__dev_null__interrupted_system_call]].
-
- <braunr> but it makes complete sense to get better performance with
- fakeroot-hurd
- <braunr> that's actually one of the reasons i'm working on it
- <braunr> if not the main one
- <teythoon> :)
- <teythoon> that was my motivation too
- <braunr> it shows how you can get an interchangeable unix tool that
- directly plugs well with the low level system
- <braunr> and make it work better
- <teythoon> nicely put :)
-
- <braunr> teythoon: i still can't manage to build glibc with fakeroot-hurd
- <braunr> but i'm not sure why :/
- <braunr> there was no kernel memory exhaustion this time
- <teythoon> :/
- <braunr> cp: cannot create regular file `debian/libc-bin.dirs': Permission
- denied
- <braunr> hum
- <braunr> youpi: do you know if building debian packages requires fakeroot
- -u option ?
- <youpi> I don't know
- <gg0> braunr: man dpkg-buildpackage says it just runs "fakeroot
- debian/rules <target>"
- <gg0> sources confirm that
- http://sources.debian.net/src/dpkg/1.17.6/scripts/dpkg-buildpackage.pl#L465
- <braunr> gg0: ok
-
-
-## IRC, freenode, #hurd, 2014-01-29
-
- <braunr> it seems that something sets the permissions of this
- debian/libc-bin.dirs file to 000 ...
- <teythoon> i've seen this too
- <braunr> oh
- <braunr> do you think it's a fakeroot-hurd bug ?
- <teythoon> have i mentioned something like this in a commit message ?
- <teythoon> yes
- <teythoon> it is
- <braunr> ok
- <braunr> i didn't see any mention of it
- <braunr> but i could have missed it
- <teythoon> hm, i cannot recall it either
- <teythoon> but i've seen this issue with fakeroot-hurd
- <braunr> ok
- <braunr> it's probably the last issue to fix to get it to work for our
- packages
- <braunr> teythoon: i think i have a solution for that last mode bug
- <braunr> fakeroot doesn't relay chmod requests, unless they change an
- executable bit
- <braunr> i don't see the point, and simply removed that condition to relay
- any chmod request
- <teythoon> braunr: did it work ?
- <braunr> no
- <braunr> fakeroot still consumes too many ports
- <braunr> and for each file, there are at least two ports, the faked one,
- and the real one
- <braunr> it should be completely reworked
- <braunr> but i don't have time to do that
- <braunr> i'll see if it works when building from scratch
- <braunr> actually, it's not even a quantity problem but a fragmentation
- problem
- <braunr> the function that fails is kmem_realloc ..
- <braunr> ipc spaces are arrays in kernel space ....
- <teythoon> it's more like three ports per file, you forgot the identity
- port
- <braunr> ah yes
-
-
-## IRC, freenode, #hurd, 2014-02-03
-
- <braunr> teythoon: i'll commit my changes on fakeroot tonight
- <braunr> they do improve the tool, but not enough to build glibc with it
- <teythoon> braunr: cool :), so how do we make it fully usable ?
- <braunr> teythoon: i don't know ..
- <braunr> i'll try re adding detection of nodes with no hard links for one
- <braunr> but imho, it needs a rework based on what the real fakeroot does
- <braunr> i won't work on it though
-
- <braunr> teythoon: also, it looks like i've tested building glibc with a
- wrong test binary of my fakeroot version :/
- <braunr> so consider all test results irrelevant so far
-
-
-## IRC, freenode, #hurd, 2014-02-04
-
- <braunr> fakeroot-hurd might turn out to be easily usable for our debian
- packages with the fixed binary :)
-
- <braunr> teythoon: hum, can you explain
- 672005782e57e049c7c8f4d6d0b2a80c0df512b4 (trans: fix locking issue in
- fakeroot) when you have time please ?
- <braunr> it looks like it introduces a deadlock by calling new_node (which
- acquires the hash table lock) while dir is locked, violating the hash
- table -> node locking order
-
- <teythoon> braunr: awesome, then there still is hope for fakeroot-hurd :)
-
- <braunr> teythoon: i've been able to build glibc packages several times
- this night
- <braunr> so except for this deadlock i've seen once, it looks good
- <teythoon> right
- <teythoon> that deadlock
- <teythoon> right, it does indeed violate the partial order of the locks :-/
-
- <braunr> teythoon: can you explain why you moved the lock in attempt_mkfile
- please ?
-
- <braunr> teythoon: i've just tested a fakeroot binary without the patch
- introducing the deadlock, and glibc built without any problem
- <teythoon> braunr: well, this is very good news :)
- <braunr> teythoon: but i still wonder why you made this patch in the first
- place, i don't want to revert it blindly and reintroduce a potential
- regression
- <teythoon> braunr: i thought i was fixing the order in which locks were
- taken. if the commit message does not specify that it fixes an issue,
- then i was probably just wrong and you can revert it
- <braunr> oh ok
- <braunr> good
-
- <braunr> teythoon: another successful build :)
- <braunr> i'll commit my changes
- <teythoon> awesome :)
- <braunr> there might still be concurrency issues but it's much better
- <teythoon> i'm curious what you did :)
- <braunr> so little :)
- <braunr> i was sick all week heh
- <braunr> you'll se
- <braunr> see
- <teythoon> well, that's good actually ;)
- <braunr> yes
-
- <braunr> teythoon: actually there was another debugging line left over, and
- again, my test results are irrelevant @#!
-
-
-## IRC, freenode, #hurd, 2014-02-05
-
- <braunr> teythoon: i got an assertion about nn->np->nn not being equal to
- nn atfer the hash table lookup is dir_lookup
- <braunr> +failure
- <teythoon> that's bad
- <braunr> not over yet
- <teythoon> i had a couple of those too
- <teythoon> i guess it's a use after free
- <braunr> yes
- <teythoon> i used to poison the pointers and comment out the frees to track
- them down iirc
- <braunr> teythoon: one of your patches stores netnodes instead of nodes in
- the hash table, citing some overwriting issue
- <braunr> teythoon: i don't understand why using netnodes fixes this
- <teythoon> braunr: libihash has this cookie for fast deletes
- <teythoon> that has to be stored somewhere
- <teythoon> the node structure has no room for it
- <braunr> uh
- <teythoon> yes
- <teythoon> it was that bad
- <braunr> ...
- <teythoon> hence the uglyish back pointers
- <braunr> i see
- <teythoon> looking back i cannot even say why it worked at all
- <braunr> well, it didn't
- <teythoon> i believe libihash must have destroyed a linked list in the node
- struct
- <braunr> possibly
- <teythoon> no, it did not >,<, but for simple tests it kind of did
- <braunr> yes fakeroot sometimes corrupts memory badly ....
- <braunr> and yes, turns out the assertion is triggered on nodes with 0 refs
- ..
- <braunr> teythoon: it looks like even the current version makes wrong usage
- of the ihash interface
- <braunr> locp_offset is defined as "The offset of the location pointer from
- the hash value"
- <braunr> and indeed, it's an intptr_t
- <braunr> teythoon: hm no, it looks ok actually, forget what i said :)
- <teythoon> *phew
- <teythoon> :p
-
- <braunr> hmm, still occasional double frees in fakeroot, but it looks in
- good shape for single threaded tasks like package building
-
- <braunr> teythoon: i've just sent my fakeroot patches
- <teythoon> braunr: sweet, i'll have a closer look tomorrow :)
- <braunr> teythoon: i couldn't debug the double frees though :/
-
-
-## IRC, freenode, #hurd, 2014-02-06
-
- <braunr> btw, i'm able to successfully use fakeroot-hurd to build glibc
- packages, but is there a way to make sure the resulting archives contain
- the right privileges and ownerships ?
- <youpi> I don't remember whether debdiff checks permissions
-
- <youpi> braunr: I've just got fakeroot-hurd debian/rules clean
- <youpi> dh_clean
- <youpi> fakeroot: ../../trans/fakeroot.c:161: netfs_node_norefs: Assertion
- `np->nn->np == np' failed.
- <youpi> while building eglibc
- <teythoon> youpi: yes, that lockup is most annoying... :/
- <braunr> youpi: with the new version ?
- <youpi> yes
- <braunr> hum
- <braunr> i only had rare double frees, not that any more :/
- <braunr> youpi: ok i got the error too
- <braunr> still not good enough
- <youpi> ok
-
-
-## IRC, freenode, #hurd, 2014-02-07
-
- <braunr> youpi: debdiff seems to handle permissions
- <braunr> i've found the cause of the assertions
- <youpi> braunr: groovie :)
-
-
-## IRC, freenode, #hurd, 2014-02-08
-
- <teythoon> braunr: nice :)
- <braunr> http://darnassus.sceen.net/~rbraun/debdiff_report
-
-
-## IRC, freenode, #hurd, 2014-02-10
-
- <braunr> and, on a completely different topic, here is a crash i can
- reproduce when using fakeroot:
- http://darnassus.sceen.net/~rbraun/fakeroot_hurd_rpctrace_o_var_tmp_out_rm_rf_dir.png
-
-
-## IRC, freenode, #hurd, 2014-02-11
-
- <braunr> still working on fakeroot
- <braunr> there are still races (not disturbing for package building but
- still ..)
- <braunr> there may be wrong right handling
- <teythoon> i believe i have witnessed a fakeroot deadlock :/
- <braunr> aw
- <teythoon> not sure though, buildbot killed the build process before i
- could investigate
- <braunr> teythoon: was it a big package ?
- <teythoon> half of the hurd package
- <braunr> that's not a port right overflow then
-
-
-## IRC, freenode, #hurd, 2014-03-05
-
- <teythoon> youpi: what about the exec_filename patch series? even though
- fakeroot still has some issues (has it?), i consider it worthy for
- inclusion
-
-[[glibc#execve_relative_paths]].
-
- <youpi> Roland was disagreeing with it
- <youpi> iirc the fakeroot issue was solved
- <teythoon> braunr: ^
- <braunr> fakeroot goot a lot more robust than it used to be
- <braunr> but i haven't checked that it actually behaves exactly like the
- library for corner cases
- <braunr> there are minor differences
- <braunr> also, it seems to trigger concurrency bugs in ext2fs
- <braunr> e.g. git reporting that files either "already exist" or "can't be
- found"
- <braunr> it happens (rarely) when directly using ext2
- <braunr> and more often through fakeroot
- <braunr> i didn't take the time to investigate
-
-## youpi
-
- the daily script of debian-installer uses the -s / -i options of fakeroot. How could we manage to implement them?
diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/file_systems.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/file_systems.mdwn
deleted file mode 100644
index a12ea10d..00000000
--- a/service_solahart_jakarta_selatan__082122541663/virtualization/file_systems.mdwn
+++ /dev/null
@@ -1,24 +0,0 @@
-[[!meta copyright="Copyright © 2010 Free Software Foundation, Inc."]]
-
-[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
-id="license" text="Permission is granted to copy, distribute and/or modify this
-document under the terms of the GNU Free Documentation License, Version 1.2 or
-any later version published by the Free Software Foundation; with no Invariant
-Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
-is included in the section entitled [[GNU Free Documentation
-License|/fdl]]."]]"""]]
-
-[[!tag open_issue_hurd]]
-
-Of course, it is possible to use commodity file systems on [[virtualized
-systems|virtualization]], like [[hurd/translator/ext2fs]] or
-[[hurd/translator/nfs]], but there are also other possibilities which ought to
-be explored.
-
- * [[network_file_system_by_just_forwarding_RPCs]]
-
- * Linux saw a patch for [*generic name to handle and open by handle
- syscalls*](http://thread.gmane.org/gmane.linux.file-systems/46648) posted,
- which in turn can be beneficial for a [[QEMU]] emulation of a 9P file
- system. LWN's Jonathan Corbet covered this [*open by
- handle*](http://lwn.net/Articles/375888/) functionality on 2010-02-23.
diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/networking.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/networking.mdwn
deleted file mode 100644
index 122f21ab..00000000
--- a/service_solahart_jakarta_selatan__082122541663/virtualization/networking.mdwn
+++ /dev/null
@@ -1,88 +0,0 @@
-[[!meta copyright="Copyright © 2010, 2013 Free Software Foundation, Inc."]]
-
-[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
-id="license" text="Permission is granted to copy, distribute and/or modify this
-document under the terms of the GNU Free Documentation License, Version 1.2 or
-any later version published by the Free Software Foundation; with no Invariant
-Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
-is included in the section entitled [[GNU Free Documentation
-License|/fdl]]."]]"""]]
-
-[[!tag open_issue_hurd]]
-
-Collection about stuff that is relevant for *virtualization* and *networking*.
-
- * [[Virtual_Square_View-OS]]
-
- * [*Virtual Networks*](http://virtualsquare.org/vn.html)
-
- * [User Level Networking](http://uln.sourceforge.net/)
-
- * [Virtual Distributed Ethernet](http://vde.sourceforge.net/)
-
- * [Application Level
- Environment4Networking](http://sourceforge.net/projects/ale4net/)
-
- *Ale4NET used dyn library call diversion to define networking at process
- level.* -- what we're doing with our approach for overriding the default
- [[hurd/translator/pfinet]] by setting environment variables.
-
- Project is now part of [[Virtual_Square_View-OS]].
-
-
-# OpenVPN
-
-[[community/meetings/GHM2013]].
-
-OpenVPN can use pfinet's tun as it is, and be configured completely as joe user,
-as shown below. Note that the tun0 node name has to begin with "tun", so pfinet
-knows it's a tun.
-
- $ mkdir -p $HOME/servers/socket
- $ touch $HOME/servers/tun0
- $ settrans -ca $HOME/servers/socket/2 /hurd/pfinet $HOME/servers/tun0 -a 10.0.0.1 -p 10.0.0.2
- $ cat vpn.conf
- client
- dev tun
- dev-node /home/samy/servers/tun0
- ...
- $ openvpn --config vpn.conf --verb 5
- ...
-
-So we let openvpn running, and now we can make applications use the pfinet
-TCP/IP stack started above: the remap command below starts a new shell, where
-it redirects /servers/socket/2 (where the system TCP/IP stack resides) onto
-$HOME/servers/socket/2 (where the new TCP/IP stack resides).
-
- $ remap /servers/socket/2 $HOME/servers/socket/2
- $ wget www.gnu.org
-
-Ideally openvpn would be made to setup pfinet itself, but at least for now it
-can be configured by hand like that.
-
-It would probably be possible to make pfinet able to produce a tap too, would
-need some code.
-
-## IRC, freenode, #hurd, 2013-09-07
-
- <d3f> anyone here knows how /dev/net is handled on the hurd? Programs using
- it say it's not a directory. I tried creating one and setting a netdde
- translator for a tun device in it, but this may be wrong as it doesn't
- work
- <teythoon> d3f: what does /dev/net do?
- <teythoon> ah, its tun/tap stuff...
- <d3f> on my gnu/linux it includes a tun device
- <teythoon> right
- <d3f> I am still reading about the Hurd and try to understand /hurd/netdde
- and devnode but by now I am quite sure I will need those to set a tun
- networktranslator on /dev/net/tun?
- <teythoon> hm, I don't think netdde or devnode will be of any help
- <teythoon> afaiui devnode makes mach devices available in the hurdish way,
- i.e. available for lookup in the filesystem
- <teythoon> d3f: ping youpi if he shows up, he hacked up openvpn to work on
- the hurd
- <d3f> yeah I know, I talked to him as I am tring to get tinc working on the
- Hurd (tinc builds by now). I will give him a shot about creating the
- "tun" device
-
-tun has indeed nothing to do with devnode and netdde, it's pfinet which creates it, as documented above.
diff --git a/service_solahart_jakarta_selatan__082122541663/virtualization/remap_root_translator.mdwn b/service_solahart_jakarta_selatan__082122541663/virtualization/remap_root_translator.mdwn
deleted file mode 100644
index 8f8668fe..00000000
--- a/service_solahart_jakarta_selatan__082122541663/virtualization/remap_root_translator.mdwn
+++ /dev/null
@@ -1,157 +0,0 @@
-[[!meta copyright="Copyright © 2013, 2014 Free Software Foundation, Inc."]]
-
-[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
-id="license" text="Permission is granted to copy, distribute and/or modify this
-document under the terms of the GNU Free Documentation License, Version 1.2 or
-any later version published by the Free Software Foundation; with no Invariant
-Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license
-is included in the section entitled [[GNU Free Documentation
-License|/fdl]]."]]"""]]
-
-[[!tag open_issue_hurd]]
-
-/!\ [[!tag open_issue_documentation]] Does this completely resolve
-[[community/gsoc/project_ideas/server_overriding]]?
-
-
-# IRC, freenode, #hurd, 2013-01-05
-
- <youpi> so we have a "remap" root translator?
- <youpi> I mean this:
- <youpi> I'd run my shell in a subhurd whose only difference is that the
- root is not the system's root, but my own
- <youpi> which catches accesses to /servers/socket/2 for instance
- <youpi> but leaves the rest flow through the system's root
- <braunr> there is just boot, i don't think it can do that
- <youpi> it'd be useful to have that
- <youpi> it'd be a very useful feature
- <youpi> to use another tcp/ip stack etc.
- <braunr> what happens when translators need to locate other translators
- used by the client ?
- <youpi> can't it tell the client to ask the real system's root?
- <youpi> (with the same path)
- <youpi> I don't remember the exact reply name
- <braunr> hum, it's getting too fuzzy for my head :p
- <youpi> well, I mean it's just like translator entries in an ext2fs
- <youpi> ext2fs replies "not me, this one"
- <braunr> but what if e.g. a user has its own pflocal, and when calling
- another translator, that one wants to contact the pflocal used by the
- client ?
- <youpi> ah, that won't work of course
- <braunr> do we actually have such cases btw ?
- <braunr> procfs perhaps
- <youpi> I don't think we'd want it actually
- <braunr> but isn't that required sometimes ?
- <youpi> inside a shell script, yes
- <braunr> for example, a storeio translator could ask about the priority
- properties of the client to proc
- <youpi> but I don't remember a case where an external translator would need
- the access
- <youpi> well, that's actually what we want
- <youpi> we don't want to fool the storeio with user-provided data :)
- <braunr> yes
- <youpi> unless the user starts the storeio himself, in which case he will
- have to re-root it
- <braunr> so it has to locate the right translator, despite not using the
- remap root translator
- <youpi> err, it will already
- <youpi> by just using the system's path
- <braunr> ?
- <youpi> maybe you need to say exactly what "it" and "right" are :)
- <braunr> ok, let's imagine your previous example with a subhurd and pfinet
- <braunr> the remap translator would imply that users from the subhurd
- *directly* access all services from the main hurd, except when routed
- otherwise by the remap translator to pfinet
- <youpi> by "directly", I mean asking the remap translator, which gives as
- answer "not me, the root"
- <braunr> now, what if a translator in the main hurd wants e.g. network
- stats from pfinet, it will ask the main one, not the one obtained through
- remap
- <braunr> yes
- <youpi> that's completely fine
- <braunr> ah
- <braunr> that's fine if the results don't matter
- <youpi> to get network stats from the user pfinet you'd have to be inside
- the shell using the remap translator
- <braunr> otherwise they're inconsistent
- <braunr> yes
- <youpi> I don't see why you'd want to get the pfinet stats from outside
- <youpi> you mean ethernet board usage?
- <braunr> service interactions
- <braunr> i can't think of anything relevant with pfinet
- <braunr> but imagine pflocal and credentials
- <youpi> I believe that'd still be ok
- <youpi> i.e. things outside the remap want to know the actual system things
- <youpi> while things inside want to know the remapped things
- <youpi> and you need that to avoid getting fooled by the user remapping
- <braunr> for credentials, i think it works because the client provides
- rights, so it would provide rights to the remapped translators in this
- case
- <braunr> this would need to be generalized
- <youpi> I believe it's already general
- <braunr> well no
- <braunr> procfs for example will always talk to the "true" proc server
- <youpi> sure
- <youpi> that's what I want from the outside
- <youpi> if the user, from the inside, wants another view, he'll have to
- start another procfs
- <youpi> his own one
- <braunr> ok
- <youpi> attached to the remapping
-
-
-## IRC, freenode, #hurd, 2013-01-29
-
- <youpi> ok, the remap translator was too easy
- <youpi> just took fakeroot.c
- <youpi> added if (!strcmp("bin/foo", filename)) filename =
- "bin/bash"; in
- <youpi> netfs_S_dir_lookup
- <youpi> and it just works
-
-[[hurd/interface/dir_lookup]].
-
- <youpi> ok, remap does indeed take my own pfinet
- <youpi> good :)
- <youpi> pfinet's tun seems to be working too
- <youpi> it's however not really flexible, it has to show up in /dev/tunx
- <youpi> I'll have a look at fixing that
- <youpi> yep, works fine
-
-
-## IRC, freenode, #hurd, 2013-02-01
-
- <youpi> braunr: as I expected, simply passing FS_RETRY_REAUTH does the
- remapping trick
-
-
-# IRC, freenode, #hurd, 2013-02-12
-
- <braunr>
- http://darnassus.sceen.net/~hurd-web/community/gsoc/project_ideas/server_overriding/
- <braunr> youpi: isn't that your remap translator ?
- <youpi> completely
- <youpi> remap being (5)
-
-
-# IRC, freenode, #hurd, 2013-02-25
-
- <youpi> I'm just having an issue with getcwd getting in the sky
- <youpi> I wonder whether libc might need patching to understand it's in
- some sort of chroot
- <youpi> or perhaps remap fixed into avoiding .. of / being odd
- <youpi> erf, it's actually an explicit error
- <youpi> libc just doesn't want to have a ".." / being different from CRDIR
- <youpi> let me just comment out that :)
- <youpi> way better :)
- <youpi> yep, just works fine
-
-
-# IRC, freenode, #hurd, 2013-03-16
-
- <braunr> youpi: is the /bin/remap --help output correct ?
-
-
-# [[hurd/fsysopts]]
-
-Doesn't support [[hurd/fsysopts]].