Merge branch 'master' into external_pager_mechanism

Conflicts:
	microkernel/mach/external_pager_mechanism.mdwn

5 files changed, 437 insertions, 0 deletions

diff --git a/hurd/documentation/auth.html b/hurd/documentation/auth.html
new file mode 100644
index 00000000..27f9ca2c
--- /dev/null
+++ b/hurd/documentation/auth.html
@@ -0,0 +1,168 @@
+[[!meta copyright="Copyright © 2002, 2008 Free Software Foundation, Inc."]]
+
+[[!meta license="Verbatim copying and distribution of this entire article is
+permitted in any medium, provided this notice is preserved."]]
+
+[[!meta title="The Authentication Server, the transcript of a talk about the
+details of the authentication mechanisms in the Hurd by Wolfgang Jährling"]]
+
+<H3><A NAME="contents">Table of Contents</A></H3>
+<UL>
+  <LI><A HREF="#intro" NAME="TOCintro">Introduction</A>
+  <LI><A HREF="#ids" NAME="TOCids">How IDs are represented and used</A>
+  <LI><A HREF="#posix" NAME="TOCposix">POSIX and beyond</A>
+  <LI><A HREF="#servers" NAME="TOCservers">Related servers</A>
+</UL>
+<HR>
+
+<H3><A HREF="#TOCintro" NAME="intro">Introduction</A></H3>
+<P>
+In this text, which mostly resembles the talk I gave at Libre Software
+Meeting 2002 in Bordeaux, I will describe what the auth server does,
+why it is so important and which cool things you can do with it, both
+on the programming and the user side.  I will also describe related
+programs like the password and fakeauth servers.  Note that this text
+is targeted at programmers who want to understand the auth mechanism
+in detail and are already familiar with concepts like Remote Procedure
+Calls (RPCs) as well as the way User- and Group-IDs are used in the
+POSIX world.
+
+<P>
+The auth server is a very small server, therefore it gives a useful
+example when you want to learn how a server typically looks like.  One
+reason why it is so small is that the auth interface, which it
+implements, consists of only four RPCs.  You can find the interface in
+hurd/hurd/auth.defs and the server itself in hurd/auth/.
+
+<H3><A HREF="#TOCids" NAME="ids">How IDs are represented and used</A></H3>
+<P>
+Each process holds (usually) one port to auth (an auth_t in C source,
+which actually is a mach_port_t, of course).  The purpose of auth is
+to manage User-IDs and Group-IDs, which is the reason why users often
+will have no choice but to make use of the systems main auth server,
+which does not listen on /servers/auth; instead you inherit a port to
+auth from your parent process.  Each such port is (internally in the
+auth server) associated with a set of effective User- and Group-IDs as
+well as a set of available User- and Group-IDs.  So we have four sets
+of IDs in total.  The available IDs can be turned into corresponding
+effective IDs at any time.
+
+<P>
+When you send an auth_getids RPC on the port you hold, you will get
+information about which IDs are associated with it, so you can figure
+out which permissions you have.  But how will a server know that you
+have these permissions and therefore know which actions (e.g. writing
+into file "foo") it is supposed to do on your behalf and which not?
+The establishing of a trusted connection to a server works as follows:
+
+<P><OL>
+<LI>A user wants a server to know its IDs</LI>
+<LI>The user requests a reauthentication from the server</LI>
+<LI>In this request the user will include a port</LI>
+<LI>Both will hand this port to auth</LI>
+<LI>The user uses auth_user_authenticate</LI>
+<LI>The server uses auth_server_authenticate</LI>
+<LI>The server also passes a new port to auth</LI>
+<LI>auth matches these two requests</LI>
+<LI>The user gets the new port from auth</LI>
+<LI>The server learns about the IDs of the user</LI>
+<LI>The user uses the new port for further communication</LI>
+</OL>
+
+<P>
+We have different RPCs for users and servers because what we pass and
+what we get back differs for them: Users get a port, and servers get
+the sets of IDs, and have to specify the port which the user will get.
+
+<P>
+It is interesting to note that auth can match the requests by
+comparing two integers, because when you get the same port from two
+people, you will have the same mach_port_t (which is nothing but an
+integer).
+
+<P>
+All of this of course only works if they use the same auth server,
+which is why I said often you have no choice other than to use the
+one main auth server.  But this is no serious restriction, as the auth server has
+almost no functionality one might want to replace.  In fact, there is
+one replacement for the default auth implementation, but more on that
+later.
+
+<H3><A HREF="#TOCposix" NAME="posix">POSIX and beyond</A></H3>
+<P>
+Before we examine what is possible with this design, let us take a
+short look at how the POSIX semantics are implemented on top of this
+design.  When a program that comes out of POSIX-land asks for its own
+effective User- or Group-ID, we will tell it about the first of the
+effective IDs.  In the same sense, the POSIX real User- or Group-ID is
+the first available ID and the POSIX saved User- or Group-ID is the
+second available ID, which is why you have the same ID two times in
+the available IDs when you log into your GNU/Hurd machine (you can
+figure out which IDs you have with the program "ids", that basically
+just does an auth_getauth RPC).  When you lack one of those IDs (for
+example when you have no effective Group-ID), a POSIX program asking
+for this particular information will get "-1" as the ID.
+
+<P>
+But as you can imagine, we can do more than what POSIX specifies.  Fox
+example, we can modify our permissions.  This is always done with the
+auth_makeauth RPC.  In this RPC, you specify the IDs that should be
+associated with the new port.  All of these IDs must be associated
+with either the port where the RPC is sent to or one of the additional
+ports you can specify; an exception is the superuser root, which is
+allowed to creat ports that are associated with arbitrary IDs.
+Hereby you can convert available into effective IDs.
+
+<P>
+This opens the door to a bunch of nice features.  For example, we have
+the addauth program in the Hurd, which makes it possible to add an ID
+to either a single process or a group of processes if you hold the ID or know the
+appropriate password, and there is a corresponding rmauth program that
+removes an ID.  So when you are working on your computer with GNU
+Emacs and want to edit a system configuration file, you switch to
+Emacs' shell-mode, do an "addauth root", enter the password, edit the
+file, and when you are done switch back to shell-mode and do "rmauth
+root".  These programs have some interesting options, and there are
+various other programs, for setting the complete list of IDs (setauth)
+and so on.
+
+<H3><A HREF="#TOCservers" NAME="servers">Related servers</A></H3>
+<P>
+Finally, I want to explain two servers which are related to auth.  The
+first is the password server, which listens on /servers/password.  If
+you pass to it a User- or Group-ID and the correct password for it, it
+will return a port to auth to you which is associated with the ID you
+passed to it.  It can create such a port because it is running as
+root.  So let us assume you are an FTP server process.  You will start
+as root, because you want to use port 21 (in this case, "port" does
+not refer to a mach_port_t, of course).  But then, you can drop all
+your permissions so that you run without any ID.  This makes it far
+less dangerous to communicate with yet unknown users over the
+network.  But when someone now hands a username and password to you,
+you can ask the password server for a new auth port.  The password
+server will check the data you pass to it, for example by looking into
+/etc/shadow, and if it is valid, it will ask the auth server for a new
+port.  It receives this port from auth and then passes it on to you.
+So you have raised your permissions. (And for the very curious: Yes,
+we are well aware of the differences between this concept and
+capabilities; and we also do have some kinds of capabilities in
+various parts of the Hurd.)
+
+<P>
+My second example is the fakeauth server.  It also implements the auth
+protocol.  It is the part of the fakeroot implementation that gives a
+process the impression that it runs as root, even if it doesn't.  So
+when the process asks fakeauth about its own IDs, fakeauth will tell
+the process that it runs as root.  But when the process wants to make
+use of the authentication protocol described earlier in this text,
+fakeauth will forward the request to its own auth server, which will
+usually be the systems main auth server, which will then be able to
+match the auth_*_authenticate requests.  So what fakeauth does is
+acting as a proxy auth server that gives someone the impression to run
+as root, while not modifying what that one is allowed to do.
+
+<P>
+At this point, I have said at least most of what can be said about the
+auth server and the protocol it implements, so I will finish by saying
+that it might be an interesting task (for you) to modify some existing
+software to take advantage of the features I described here.
diff --git a/hurd/documentation/hurd-and-linux.mdwn b/hurd/documentation/hurd-and-linux.mdwn
new file mode 100644
index 00000000..678ea8da
--- /dev/null
+++ b/hurd/documentation/hurd-and-linux.mdwn
@@ -0,0 +1,11 @@
+[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled
+[[GNU Free Documentation License|/fdl]]."]]"""]]
+
+[[!meta redir=/hurd-and-linux]]
diff --git a/hurd/documentation/hurd-paper.mdwn b/hurd/documentation/hurd-paper.mdwn
new file mode 100644
index 00000000..06c23662
--- /dev/null
+++ b/hurd/documentation/hurd-paper.mdwn
@@ -0,0 +1,11 @@
+[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled
+[[GNU Free Documentation License|/fdl]]."]]"""]]
+
+[[!meta redir=/hurd-paper]]
diff --git a/hurd/documentation/hurd-talk.mdwn b/hurd/documentation/hurd-talk.mdwn
new file mode 100644
index 00000000..83dcaf74
--- /dev/null
+++ b/hurd/documentation/hurd-talk.mdwn
@@ -0,0 +1,11 @@
+[[!meta copyright="Copyright © 2009 Free Software Foundation, Inc."]]
+
+[[!meta license="""[[!toggle id="license" text="GFDL 1.2+"]][[!toggleable
+id="license" text="Permission is granted to copy, distribute and/or modify this
+document under the terms of the GNU Free Documentation License, Version 1.2 or
+any later version published by the Free Software Foundation; with no Invariant
+Sections, no Front-Cover Texts, and no Back-Cover Texts.  A copy of the license
+is included in the section entitled
+[[GNU Free Documentation License|/fdl]]."]]"""]]
+
+[[!meta redir=/hurd-talk]]
diff --git a/hurd/documentation/translators.html b/hurd/documentation/translators.html
new file mode 100644
index 00000000..8ae2c180
--- /dev/null
+++ b/hurd/documentation/translators.html
@@ -0,0 +1,236 @@
+[[!meta copyright="Copyright © 1998, 1999, 2007, 2008 Free Software Foundation,
+Inc."]]
+
+[[!meta license="Verbatim copying and distribution of this entire article is
+permitted in any medium, provided this notice is preserved."]]
+
+[[!meta title="Translators"]]
+
+By Marcus Brinkmann.
+
+<ul>
+<li><a href="#concept" name="TOC_concept">Concept</a></li>
+<li><a href="#examples" name="TOC_examples">Examples</a></li>
+<li><a href="#actpas" name="TOC_actpas">Passive Translators, Active Translators</a></li>
+<li><a href="#manage" name="TOC_manage">Managing Translators</a></li>
+</ul>
+<h3><a href="#TOC_concept" name="concept">Concept</a></h3>
+<p>
+Before we take a closer look at translators, let us consider regular
+filesystems. A filesystem is store for a hierarchical tree of directories
+and files. You access directories and files by a special character string,
+the path. Furthermore, there are symbolic links to refer to one file at
+several places in the tree, there are hard links to give one and the same
+file several names. There are also special device files for communication
+with the hardware device drivers of the kernel, and there are mount points
+to include other stores in the directory tree. Then there are obscure
+objects like fifos and hard links.</p>
+<p>
+Although these objects are very different, they share some common
+properties, for example, they have all an owner and a group associated with
+them as well as access rights (permissions). This information is written in
+inodes. This is a actually a further commonality: Every object has exactly
+one inode associated with it (hard links are somewhat special as they share
+one and the same inode). Sometimes, the inode has further information
+stored in it. For example, the inode can contain the target of a symbolic
+link.</p>
+<p>
+However, these commonalities are usually not exploited in the
+implementations, despite the common programming interface to them. All
+inodes can be accessed through the standard POSIX calls, for example
+<code>read()</code> and <code>write()</code>. For example, to add a new
+object type (for example a new link type) to a common monolithic unix
+kernel, you would need to modify the code for each filesystem
+seperately.</p>
+<p>
+In the Hurd, things work differently. Although in the Hurd a special
+filesystem server can exploit special properties of standard object types
+like links (in the ext2 filesystem with fast links, for example), it has a
+general interface to add such features without modifying existing code.</p>
+<p>
+The trick is to allow a program to be inserted between the actual content of
+a file and the user accessing this file. Such a program is called a
+translator, because it is able to process the incoming requests in many
+different ways. In other words, a translator is a Hurd server which provides
+the basic filesystem interface.</p>
+<p>
+Translators have very interesting properties. From the kernels point of
+view, they are just another user process. This means, translators can be run
+by any user. You don't need root priviligies to install or modify a
+translator, you only need the access rights for the underlying inode the
+translator is attached to. Many translators don't require an actual file to
+operate, they can provide information by their own means. This is why
+the information about translators is stored in the inode.</p>
+<p>
+Translators are responsible to serve all file system operations that involve
+the inode they are attached to. Because they are not restricted to the usual
+set of objects (device file, link etc), they are free to return anything
+that makes sense to the programmer. One could imagine a translator that
+behaves like a directory when accessed by <code>cd</code> or
+<code>ls</code> and at the same time behaves like a file when accessed by
+<code>cat</code>.</p>
+<h3><a href="#TOC_examples" name="examples">Examples</a></h3>
+<h4>Mount Points</h4>
+<p>
+A mount point can be seen as an inode that has a special translator attached
+to it. Its purpose would be to translate filesystem operations on the mount
+point in filesystem operations on another store, let's say, another
+partition.</p>
+<p>
+Indeed, this is how filesystems are implemented under the Hurd. A
+filesystem is a translator. This translator takes a store as its argument,
+and is able to serve all filesystem operations transparently.</p>
+<h4>Device Files</h4>
+<p>
+There are many different device files, and in systems with a monolithical
+kernel, they are all provided by the kernel itself. In the Hurd, all device
+files are provided by translators. One translator can provide support for
+many similar device files, for example all hard disk partitions. This way,
+the number of actual translators needed is quite small. However, note that
+for each device file accessed, a seperate translator task is started.
+Because the Hurd is heavily multi threaded, this is very cheap.</p>
+<p>
+When hardware is involved, a translator usually starts to communicate with
+the kernel to get the data from the hardware. However, if no hardware access
+is necessary, the kernel does not need to be involved. For example,
+<code>/dev/zero</code> does not require hardware access, and can therefore
+be implemented completely in user space.</p>
+<h4>Symbolic Links</h4>
+<p>
+A symbolic link can be seen as a translator. Accesing the symbolic link
+would start up the translator, which would forward the request to the
+filesystem that contains the file the link points to.</p>
+<p>
+However, for better performance, filesystems that have native support
+for symbolic links can take advantage of this feature and implement
+symbolic links differently. Internally, accessing a symbolic link would not
+start a new translator process. However, to the user, it would still look
+as if a passive translator is involved (see below for an explanation what a
+passsive translator is).</p>
+<p>
+Because the Hurd ships with a symlink translator, any filesystem server that
+provides support for translators automatically has support for symlinks (and
+firmlinks, and device files etc)! This means, you can get a working
+filesystem very fast, and add native support for symlinks and other features
+later.</p>
+<h3><a href="#TOC_actpas" name="actpas">Passive Translators, Active Translators</a></h3>
+<p>
+There are two types of translators, passive and active. They are really
+completely different things, so don't mix them up, but they have a close
+relation to each other.</p>
+<h4>Active Translators</h4>
+<p>
+An active translator is a running translator process, as introduced above.
+You can set and remove active translators using the
+<code>settrans -a</code></a>
+command. The <code>-a</code> option is necessary to tell
+<code>settrans</code> that you want to modify the active translator.</p>
+<p>
+The <code>settrans</code> command takes three kind of arguments. First, you
+can set options for the <code>settrans</code> command itself, like
+<code>-a</code> to modify the active translator. Then you set the inode you
+want to modify. Remember that a translator is always associated with an
+inode in the directory hierarchy. You can only modify one inode at a time.
+If you do not specify any more arguments, <code>settrans</code> will try to
+remove an existing translator. How hard it tries depends on the force
+options you specify (if the translator is in use by any process, you will
+get "device or resource busy" error message unless you force it to go away).</p>
+<p>
+But if you specify further arguments, it will be interpreted as a command
+line to run the translator. This means, the next argument is the filename of
+the translator executable. Further arguments are options to the translator,
+and not to the <code>settrans</code> command.</p>
+<p>
+For example, to mount an ext2fs partition, you can run
+<code>settrans -a -c /mnt /hurd/ext2fs /dev/hd2s5</code>. The
+<code>-c</code> option will create the mount point for you if it doesn't
+exist already. This does not need to be a directory, by the way. To unmount,
+you would try <code>settrans -a /mnt</code>.</p>
+<h4>Passive Translators</h4>
+<p>
+A passive translator is set and modified with the same syntax as the active
+translator (just leave away the <code>-a</code>, so everything said above is
+true for passive translators, too. However, there is a difference: passive
+translators are not yet started.</p>
+<p>
+This makes sense, because this is what you usually want. You don't want the
+partition mounted unless you really access files on this partition. You
+don't want to bring up the network unless there is some traffic and so
+on.</p>
+<p>
+Instead, the first time the passive translator is accessed, it is
+automatically read out of the inode and an active translator is started on
+top of it using the command line that was stored in the inode. This is
+similar to the Linux automounter functionality. However, it does not come as
+an additional bonus that you have to set up manually, but an integral part of
+the system. So, setting passive translators defers starting the translator
+task until you really need it. By the way, if the active translator dies for
+some reason, the next time the inode is accessed the translator is
+restarted.</p>
+<p>
+There is a further difference: active translators can die or get lost. As
+soon as the active translator process is killed (for example, because you
+reboot the machine) it is lost forever. Passive translators are not transient
+and stay in the inode during reboots until you modify them with the
+<code>settrans</code> program or delete the inodes they are attached to.
+This means, you don't need to maintain a configuration file with your mount
+points.</p>
+<p>
+One last point: Even if you have set a passive translator, you can still
+set a different active translator. Only if the translator is automatically
+started because there was no active translator the time the inode was
+accessed the passive translator is considered.</p>
+<h3><a href="#TOC_manage" name="manage">Managing Translators</a></h3>
+<p>
+As mentioned above, you can use
+<code>settrans</code></a>
+to set and alter passive and active translators. There are a lot of options
+to change the behaviour of <code>settrans</code> in case something goes
+wrong, and to conditionalize its action. Here are some common usages:</p>
+<ul><li><code>settrans -c /mnt /hurd/ext2fs /dev/hd2s5</code> mounts a
+partition, the translator will stay across reboots.</li>
+<li><code>settrans -a /mnt /hurd/ext2fs ~/dummy.fs</code> mounts a
+filesystem inside a data file, the translator will go away if it dies.</li>
+<li><code>settrans -fg /nfs-data</code> forces a translator to go away.</li>
+</ul>
+<p>
+You can use the <code>showtrans</code></a>
+command to see if a translator is attached to an inode. This will only show
+you the passive translator though.</p>
+<p>
+You can change the options of an active (filesystem) translator with
+<code>fsysopts</code> without actually restarting it. This is very
+convenient. For example, you can do what is called "remounting a
+partition read-only" under Linux simply by running <code>fsysopts
+/mntpoint --readonly</code>. The running active translator
+will change its behaviour according to your request if possible.
+<code>fsysopts /mntpoint</code> without a parameter shows you the current
+settings.</p>
+<h4>Examples</h4>
+<p>
+I recommend that you start by reading the <code>/bin/mount</code> command,
+it is only a small script. Because setting filesystem translators is
+similar to mounting partitions, you can easily grasp the concept this way.
+Make a file system image with <code>dd if=/dev/zero of=dummy.fs bs=1024k
+count=8; mke2fs dummy.fs</code> and "mount" it with <code>settrans -c dummy
+/hurd/ext2fs `pwd`/dummy.fs</code>. Note that the translator is not started
+yet, no new <code>ext2fs</code> process is running (verify with <code>ps
+Aux</code>). Check that everything is correct using <code>showtrans</code></p>
+<p>
+Now type <code>ls dummy</code> and you will notice the short delay that
+occurs while the translator is started. After that, there will be no more
+delays accessing dummy. Under Linux, one would say that you automounted a
+loop file system. Check with <code>ps Aux</code> that there is an <code>ext2fs
+dummy</code> process up and running now. Now put some files into the new
+directory. Try to make the filesystem read-only with <code>fsysopts</code>.
+Note how further write attempts fail now. Try to kill the active translator
+with <code>settrans -g</code>.</p>
+<p>
+You should have some understanding of what is going on now. Now remember
+that this was only <em>one</em> special server, the Hurd ext2fs server.
+There are many more server in the <code>hurd</code> directory. Some of them
+are for filesystems. Some are needed for file system features like links.
+Some are needed for device files. Some are useful for networking. Imagine
+"mounting" an FTP Server with <code>settrans</code> and downloading files
+simply with the standard <code>cp</code> command. Or editing your web sites
+with <code>emacs /ftp/homepage.my.server.org/index.html</code>!</p>