Merge branch 'master' of arnebab@flubber:~wiki/wiki

5 files changed, 274 insertions, 55 deletions

diff --git a/community/da.mdwn b/community/da.mdwn
index ad3aebc8..e15aade1 100644
--- a/community/da.mdwn
+++ b/community/da.mdwn
@@ -6,6 +6,8 @@ Project: Network virtualization for subhurds etc.
 
 The [code](http://www.assembla.com/spaces/VNetHurd/trac_subversion_tool).
 
+The [[howto]] shows the instructions of setting up the virtual network in hurd and subhurd.
+
 ---
 
 ## The design and the implementation
@@ -46,28 +48,11 @@ A filter translator is needed to enforce the policies between the interface and
 
 ---
 
-## The schedule
-From 14.06 - 18.06 Coding: 
-build the connection between pfinets in sub-hurd and network interface (by broadcasting)
-
-From 21.06 - 25.06 Coding: 
-build the connection between the multiplexer and the pfinet server in the main hurd.
-
-From 26.06 - 13.07 Coding: 
-Implement the real routing among the pfinets and the network interface
-
-From 15.07 - 10.08 Coding: 
-Implement some traffic control in the multiplexer and some policy control if necessary
-
-From 11.08 - 18.08 Testing
-
-
----
-
 ## TODO
 ### Coding
 
-a proxy of the process server
+ - make subhurds running without root privileges
+ - merge BPF rules from the filter translator and the multiplexer
 
 ---
 
@@ -75,8 +60,17 @@ a proxy of the process server
 
 ### Coding
 
-pfinet server overriding by modifying glibc. The patch is [here](http://www.assembla.com/spaces/VNetHurd/documents/aJidqKp6ur3z-Nab7jnrAJ/download/A%20patch%20of%20glibc).
-the patch of pfinet is [here](http://www.assembla.com/spaces/VNetHurd/documents/dqoQg0qUer3Asvab7jnrAJ/download/A%20patch%20of%20pfinet).
+The patch of glibc (pfinet server overriding) is [here](http://www.assembla.com/spaces/VNetHurd/documents/aJidqKp6ur3z-Nab7jnrAJ/download/A%20patch%20of%20glibc).
+
+The patch of pfinet (open the virtual network interface) is [here](http://www.assembla.com/spaces/VNetHurd/documents/aWqYwYATKr3BBOab7jnrAJ/download/patch%20of%20pfinet%201%20(to%20use%20the%20virtual%20interface)).
+
+The patch of pfinet (fix pfinet to use the proper filter rule) is [here](http://www.assembla.com/spaces/VNetHurd/documents/besb-qATKr3AIxab7jnrAJ/download/patch%20of%20pfinet%202%20(to%20add%20an%20IP%20filter)).
+
+The patch of pfinet (set the mach device in the promiscuous mode) is [here](http://www.assembla.com/spaces/VNetHurd/documents/bEovN6ATKr3B8uab7jnrAJ/download/patch%20of%20pfinet%203%20(to%20set%20the%20mach%20device%20into%20the%20promiscuous%20mode)).
+
+The patch of boot (open the virtual network interface) is [here](http://www.assembla.com/spaces/VNetHurd/documents/cWkeEixHar3AdKab7jnrAJ/download/A%20patch%20of%20boot).
+
+The patch of gnumach (set the network device into the promiscuous mode) is [here](http://www.assembla.com/spaces/VNetHurd/documents/b0eLzUxHmr3ymXab7jnrAJ/download/A%20patch%20of%20gnumach).
 
 the multiplexer:
 
@@ -89,6 +83,16 @@ the filter translator:
 - Forward the packet between the interface and the pfinet server.
 - Filter the packet.
 
+the proxy of the proc server:
+
+- Forward all requests from the process to its proc server.
+- The proxy doesn't do any real work except returning the host private port and the master device port of the proxy (shown as an example).
+
+the devnode translator:
+
+- Create a device file to help open the network device.
+
+
 ### The Code Read
 
 - boot
diff --git a/community/flavioc.mdwn b/community/flavioc.mdwn
index a0c0d036..de4741f8 100644
--- a/community/flavioc.mdwn
+++ b/community/flavioc.mdwn
@@ -16,7 +16,7 @@ Some [Hurd stuff](http://opensvn.csie.org/leic/hurd/)
 
 And code: [cl-hurd](http://freehg.org/u/flavioc/cl-hurd/)
 
-hg clone http://freehg.org/u/flavioc/cl-hurd/
+hg clone [http://freehg.org/u/flavioc/cl-hurd/](http://freehg.org/u/flavioc/cl-hurd/)
 
 ## Summer session
 
@@ -25,11 +25,18 @@ Creating an extensible translator library in lisp using the mig generated stubs.
 ### What's done
 
 - The library for writing translators is mostly written.
-- This library is intended to implement virtual filesystems. Examples are: translators were data is located in a local file (like zipfs, tarfs, rarfs, ...), single file translators (that do content filtering, output of a command, etc), network based filesystems (ftpfs, httpfs, ...)
-- Right now, what's missing is: support for symlinks, file execution, and setting other translators on our node tree.
+- This library is intended to implement virtual filesystems. Examples are: translators were data is located in a local file (like zipfs, tarfs, rarfs, ...), single file translators (that do content filtering, output of a command, etc), network based filesystems (ftpfs, httpfs, ...), proxy filesystems (like hostmux, usermux, etc..)
 - It's possible to specialize the basic translator library and implement new translator classes. This is done using CLOS.
 - There is a tree-translator class that makes the managing of a node tree very easy, doing all the work for us, through a simple directory API and implementing the directory callbacks for us.
 - There is a simple example (something like zipfs) translator that can expose the directories and file contents of a ZIP file.
+- More translator examples include:
+  - /dev/null translator.
+  - /dev/zero translator.
+  - translator that creates a symlink node.
+  - tmpfs like translator.
+  - a translator that does proxying between clients and the underlying translator returning all data in upper-case.
+  - a translator that watches for changes in a file describing the file system structure.
+  - an IRC translator.
 - Translator options (manipulated through fsysopts) have a simple and easy to use API.
 - All the Mach port manipulation API is available.
 - It's possible to send and receive messages. Simple example:
@@ -44,21 +51,33 @@ Creating an extensible translator library in lisp using the mig generated stubs.
 - New message types (like :string, :integer) can be implemented, providing a powerful extension mechanism.
 - Creation of symlinks and symlink path resolution.
 - Creation of character/block devices, fifos and sockets.
+- Patch that opens stdin + stdout to /dev/null.
+- Project has been separated into 5 ASDF installable systems:
+  - hurd-common
+  - mach
+  - hurd
+  - hurd-translator
+  - tree-translator
+- Test cases are now written.
 
 
 ### What needs to be done
 
-- Using continuations for IO blocking operations. This can be done using cl-cont.
-- Callbacks:
-  - file_lock, file_lock_stat (still not sure if they are really needed)
-  - file_reparent
-- Bind the client RPC calls.
+- Fix fsys-getroot (block happens in trivfs based translators, when they do RPC's to me when I call fsys_getroot to them) and fetch-root (for passive translators).
 - Use the socket stubs?
+- Documentation
+- Integrate upstream:
+  - Done, but still not commited.
 
 
-### Notes
+### Project dependencies
+
+- CLISP
+- [CFFI](http://common-lisp.net/project/cffi/) (apt-get installable)
+- [Flexi streams](http://www.weitz.de/flexi-streams/) (apt-get installable)
+- [Trivial garbage](http://www.cliki.net/trivial-garbage) (not in debian repositories)
+- [cl-zip](http://common-lisp.net/project/zip/) (only needed for the example translator)
 
-- File execution is complicated to do, because there is no multithreading support on CLisp and continuations won't do it. Maybe forking the clisp process?.
 
 ## To do
 
diff --git a/community/howto.mdwn b/community/howto.mdwn
new file mode 100644
index 00000000..e067f77e
--- /dev/null
+++ b/community/howto.mdwn
@@ -0,0 +1,95 @@
+[[meta copyright="Copyright © 2008 Free Software Foundation, Inc."]]
+
+[[meta license="""[[toggle id="license" text="GFDL 1.2+"]][[toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled
+[[GNU_Free_Documentation_License|/fdl]]."]]"""]]
+
+
+This document briefly introduces how to set up the virtual network and connect the subhurd with the main hurd.
+
+
+### 1. Set up the virtual network.
+
+####1.1 Patch and install GNU Hurd, GNU Mach and the GNU C library.
+
+Step 1: Get the Hurd and compile it.
+
+	cvs -z3 -d:ext:username@cvs.savannah.gnu.org:/sources/hurd co -r zhengda-soc2008-virt-branch
+
+
+Step 2: apply the [patch](http://www.assembla.com/spaces/VNetHurd/documents/b0eLzUxHmr3ymXab7jnrAJ/download/A%20patch%20of%20gnumach) on the GNU Mach.
+
+In order to connect the virtual network created in hurd with the external network, we need this patch. It enables the Hurd to set the Mach device into the promiscuous mode, so the real device can accept the packet that goes to the virtual device in hurd.
+(This step is optional if we are only interested in creating a internal virtual network.)
+
+Step 3: apply the [patch](http://www.assembla.com/spaces/VNetHurd/documents/aJidqKp6ur3z-Nab7jnrAJ/download/A%20patch%20of%20glibc) on glibc.
+
+This patch enables the user to override the default socket server by using the environment variables SOCK_SERV_DIR or SOCK_SERV_%d (%d is the domain of the socket server).
+
+
+#### 1.2 Set up the Hurd components to build the virtual network.
+
+In this section, I show how to create two virtual interfaces and run three pfinet servers. I assume that the source code of Hurd is in /root/hurd.
+
+Step 4: create the network device file in /dev with devnode.
+The network device file is used to help other translators open the device.
+
+	# settrans -acfg /dev/eth0 /root/hurd/devnode/devnode -d eth0
+
+Step 5: create the virtual network device with eth-multiplexer.
+
+eth-multiplexer is responsible to create the virtual network device and dispatch the packet to the right clients that connect to the virtual device. It also connects the virtual network and the external network.
+
+	# settrans -acfg /root/multiplexer /root/hurd/eth-multiplexer/eth-multiplexer -v 2 -i /dev/eth0
+
+Step 6: create the network device files for the virtual network device with devnode.
+
+	# settrans -acfg /dev/veth0 /root/hurd/devnode/devnode -d veth0 -M /root/multiplexer
+	# settrans -acfg /dev/veth1 /root/hurd/devnode/devnode -d veth1 -M /root/multiplexer
+
+Step 7: setup the filter translator eth-filter on the network device. This step is optional. 
+eth-filter is used to set up the traffic policy on the network device.
+
+	# settrans -acfg /dev/fveth0 /root/hurd/eth-filter/eth-filter -i /dev/veth0 -S 192.168.8.0/24 -R 192.168.8.0/24
+	# settrans -acfg /dev/fveth1 /root/hurd/eth-filter/eth-filter -i /dev/veth1 -S 192.168.8.0/24 -R 192.168.8.0/24
+
+Step 8: setup the pfinet server on the virtual network device.
+
+	# settrans -afgc /root/socket0/2 /root/hurd/pfinet/pfinet -i /dev/fveth0 -a 192.168.8.10 -g 192.168.8.1 -m 255.255.255.0
+	# settrans -afgc /root/socket1/2 /root/hurd/pfinet/pfinet -i /dev/fveth1 -a 192.168.8.11 -g 192.168.8.1 -m 255.255.255.0
+	# settrans -afgc /root/socket2/2 /root/hurd/pfinet/pfinet -i /dev/fveth1 -a 192.168.8.12 -g 192.168.8.1 -m 255.255.255.0
+
+
+#### 1.3 Run the command with the customized pfinet server.
+
+Step 9: Set environment variables to use the customized pfinet server.
+
+The environment variable of SOCK_SERV_DIR is used to override all socket servers and SOCK_SERV_%d to override a specific socket server. %d after SOCK_SERV_ is the domain of the protocol that the socket server supports. The environment variable SOCK_SERV_%d has the higher priority than SOCK_SERV_DIR. 
+
+	# export SOCK_SERV_2=/root/socket1/2
+
+If the modified glibc isn't installed as the system default C library, set LD_LIBRARY_PATH environment.
+
+	# export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/the/path/of/glibc
+
+We can run any command now, for example, ping.
+
+A SHELL script is provided to run all translators I mentioned automatically: http://www.assembla.com/spaces/VNetHurd/documents/c2W71ABser3AIxab7jnrAJ/download/runmultiplexer. To use this script, the user must specify the source of the hurd tree (the default value is /root/hurd) and the path of the servers (the default value is /root) where they should be created. This script is only used to test all translators I mentioned above and shows all steps to set up the virtual network.
+
+
+### 2. Connect the subhurd with the main hurd.
+
+In the main hurd, we still need to do Step 1-9. 
+We run subhurd, 
+
+	# /root/hurd/boot/boot -f eth0=/dev/fveth0 -f eth1=/dev/fveth1 servers.boot /dev/hd1s1
+
+In the subhurd, we do Step 1, 4, 8.
+Step 4: # settrans -acfg /dev/veth0 /root/hurd/devnode/devnode -d veth0
+Step 8: # settrans -afgc /servers/socket/2 /root/hurd/pfinet/pfinet -i /dev/veth0 -a 192.168.8.20 -g 192.168.8.1 -m 255.255.255.0
+
+Now we can communicate from the subhurd with any pfinet server of the main Hurd that runs in the virtual network.
diff --git a/community/procfs.mdwn b/community/procfs.mdwn
index 472c66c9..1139c718 100644
--- a/community/procfs.mdwn
+++ b/community/procfs.mdwn
@@ -210,8 +210,11 @@ Clone URL: [git://github.com/madhusudancs/procfs.git](git://github.com/madhusuda
 * cnswap
 > Cumulative nswap for child processes (not maintained).
 
+* flags
+> PF_* fields defined in (Not Linux compatible, but nearly says the something Linux says)
 
-####I already know the where the information is exactly available.
+* nice
+> The nice value ranges from 19 to -19.
 
 * cutime
 > The number of jiffies that this process’s waited-for children have been scheduled in user
@@ -222,12 +225,21 @@ Clone URL: [git://github.com/madhusudancs/procfs.git](git://github.com/madhusuda
 
 #####File - /proc/<PID>/statm
 
+* size
+> total program size
+
 * resident
 > resident set size
 
-#####Other Per-PID Files
+* lib
+> library
 
-#####* /proc/<PID>/cwd
+* dt
+> dirty pages
+
+####I already know the where the information is exactly available.
+
+#####Other Per-PID Files
 
 #####* /proc/<PID>/exe
 
@@ -246,9 +258,6 @@ Clone URL: [git://github.com/madhusudancs/procfs.git](git://github.com/madhusuda
 * cmajflt
 > The number of major faults that the process’s waited-for children have made.
 
-* nice
-> The nice value ranges from 19 to -19.
-
 * signal
 > The bitmap of pending signals.
 
@@ -266,12 +275,13 @@ Clone URL: [git://github.com/madhusudancs/procfs.git](git://github.com/madhusuda
 
 #####File - /proc/<PID>/statm
 
-* size
-> total program size
-
 * text
 > text (code)
 
+#####Other Per-PID Files
+
+#####* /proc/<PID>/cwd
+
 ####The information may be available, but needs to be searched to know where it will be.
 
 #####File - /proc/<PID>/stat
@@ -333,19 +343,23 @@ Clone URL: [git://github.com/madhusudancs/procfs.git](git://github.com/madhusuda
 * delayacct_blkio_ticks
 > Aggregated block I/O delays, measured in clock ticks (centiseconds).
 
-* flags
-> PF_* fields defined in
 
+###Newly added to Roadmap(but these were the original goals of the project)
 
-#### Need not be implemented.
+#### procps tools need to be ported so that they run on top of the procfs
 
-#####File - /proc/<PID>/statm
+> ##### pgrep                 -  Done
+> ##### pkill                 -  Done
+> ##### killall               -  Done
+> ##### pstree                -  Done
+> ##### top                   -  Mostly Done (except per-PID shared memory field, and non per-PID caches and buffers field)
+> ##### free                  -  Mostly Done (Ditto from above)
+> ##### htop                  -  Mostly Done (Ditto again)
+> ##### watch                 -  Done
+> ##### tload                 -  I think it is done. (Need someone to test it)
+> ##### libgtop               -  In progress
+> ##### gnome-system-moitor   -  In progress
 
-* lib
-> library (not required)
-
-* dt
-> dirty pages (not required)
 
 
 ------
diff --git a/community/scolobb.mdwn b/community/scolobb.mdwn
index b4aadde0..6c959f1e 100644
--- a/community/scolobb.mdwn
+++ b/community/scolobb.mdwn
@@ -33,10 +33,40 @@ The code is at <http://github.com/scolobb/nsmux/tree/master>.
 * The skeleton which mirrors the filesystem.
 
 ####TODO:
+<<<<<<< HEAD:community/scolobb.mdwn
 
-* Provide the looking up of files with special suffixes like 'file,,x'
+<<<<<<< HEAD:community/scolobb.mdwn
+* Provide shadow nodes (modify the standard version of netfs_S_dir_lookup).
+=======
+* Implemented the propagation of a translator on all files belonging
+  to directory 'dir' in the case of the following query: 'dir,,x'
+  (might change, see the thing with "recursive wrappers")
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
 
-* Provide RPC for accessing the untranslated node.
+<<<<<<< HEAD:community/scolobb.mdwn
+* Create the generic filter translator.
+=======
+* Any translator can access the untranslated version of the node it is
+  to be set upon by specifying the corresponding flags to the
+  <lib>_startup routine, where <lib> is the name of the library used
+  to create the translator.
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+
+* Create the translator '0' (providing the untranslated version of the
+  node).
+
+* Create the "recursive wrappers" for one-node translators.
+=======
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+
+* Provide proxy nodes (modify the standard version of netfs_S_dir_lookup).
+
+* Create the generic filtering translator.
+
+* Create the translator '0' (providing the untranslated version of the
+  node).
+
+* Create the "recursive wrappers" for one-node translators.
 
 * Create special translators for the main proxy so that its functionality
   should be complete.
@@ -44,19 +74,75 @@ The code is at <http://github.com/scolobb/nsmux/tree/master>.
 * Refine the skeleton in several places so that it should become faster
   and more reliable.
 
+* Kill bugs.
+
+<<<<<<< HEAD:community/scolobb.mdwn
+<<<<<<< HEAD:community/scolobb.mdwn
+=======
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+* Integrate nsmux upstream.
+
+<<<<<<< HEAD:community/scolobb.mdwn
+=======
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+=======
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+* Solve the libtrivfs stacking issue.
+
+<<<<<<< HEAD:community/scolobb.mdwn
+<<<<<<< HEAD:community/scolobb.mdwn
+=======
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+* Patch libnetfs (it does not support file_get_translator_cntl, for
+  instance).
+<<<<<<< HEAD:community/scolobb.mdwn
+=======
+* Patch libnetfs (?).
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+=======
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+
 ---
 
 ###Progress
 
-####2: Thu Jul 17, ~13:00 UTC - Fri Jul 18, ~21:00 UTC:
+<<<<<<< HEAD:community/scolobb.mdwn
+####6: Sun Aug 3 - ?:
+=======
+####6: Mon Aug 4 - ?
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+
+<<<<<<< HEAD:community/scolobb.mdwn
+<<<<<<< HEAD:community/scolobb.mdwn
+> Implement the shadow nodes.
+=======
+> Implemeting the proxy nodes.
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+=======
+> Implementing the proxy nodes.
+>>>>>>> web commit by scolobb:community/scolobb.mdwn
+
+####5: Thu Jul 24 - Thu Jul 24:
+
+> Created a *libnetfs*-based one-node translator, working exactly like the *libtrivfs*-based translator I had written before; the former, however, can be included in a translator stack.
+
+####4: Tue Jul 22 - Thu Jul 24:
+
+> Attempted to make a *libtrivfs*-based translator to be able to be stacked upon itself (to be able to receive a translator on top of itself, more exactly); attempted to   borrow some code from *libnetfs* but this didn't bring any results.
+
+####3: Sun Jul 20 - Tue Jul 22:
+
+> Implemented the possibility to propagate a translator on all files belonging to a directory 'dir' in the request of the type 'dir,,x/'.
+
+####2: Thu Jul 17 - Fri Jul 18:
 
 > Extended the lookup code in *nsmux* to allow for looking up nodes like 'file,,x' and added the possibility to escape the double-comma in the following way: ',,,'.
 
-####1: Mon Jul 12, ~13:00 UTC - Tue Jul 13, ~15:00 UTC:
+####1: Mon Jul 12 - Tue Jul 13:
 
 > Implemented a simple *libtrivfs*-based translator to test the lookup code for *nsmux*.
 
-####0: Sat Jul 12, ~09:00 UTC - Sat Jul 12, ~18:00 UTC:
+####0: Sat Jul 12 - Sat Jul 12:
 
 > Made small changes to the code of *filterfs* to fit the needs of *nsmux*.
 
@@ -76,4 +162,5 @@ The code is at <http://github.com/scolobb/nsmux/tree/master>.
 
 ####0: Sun Apr 27:
 
-> Followed the code of *dmesgd* (<http://www.bddebian.com/junk/dmesgd/>) kindly offered by **bddebian** and rewritten it from scratch as a tutorial.
+> Followed the code of *dmesgd* (<http://www.bddebian.com/junk/dmesgd/>) kindly offered by **bddebian** and rewrote it from scratch as a tutorial.
+