How to turn a Martian NetDrive into another OpenAFS file server

A What?

The ``Martian NetDrive'' from is a trivial-to-set-up SMB fileserver that is also silent, and small, suitable for closets or obscure shelves. One model even has 802.11, so that it doesn't even need wiring - just unpack the shipping crate, plug it in, hit the on switch, and wait for it to show up under Rendezvous in your browser (or find it in your Windows Network Neighborhood and look for the setup.html... or, if you're One Of Us, find the address in the DHCP log when you first bring it up :)

It's a really cool ``instant infrastructure'' idea; if and were still going concerns, we'd probably be working with them. In the mean time, it's a pretty good ``I need a hundred or so gig in a fileserver *now*'' item... as long as you're heavily firewalled, of course!

Do what with it? gives the details, but basically, OpenAFS is the modern production version of what was once the ``Andrew File System'', a secure (kerberized) distributed network filesystem popular among educational institutions and some large-infrastructure companies (including, MIT, CMU, UMich, and Morgen Stanley.) There are clients for every OS that anyone takes at all seriously.

One of the particularly convenient features of OpenAFS is how you add more space to it. You simply add new space (install a new machine, or connect disks to an existing one) and restart the fileserver process on that machine, and your new partitions are available. This doesn't matter to the end-users, who see a unified filesystem under /afs/ regardless of where the data is; the administrator can then ``behind the scenes'' migrate data to the new server, or add replicas of existing volumes there, or simply start creating new volumes there (and ``mounting'' them in the global namespace.)

Two Great Tastes... has been an OpenAFS cell (/afs/ for years, and has three machines in it. The machines are cheap half-GHz Celerons picked up over the last few years; they work, but they're noisy and take up a lot of space. The advent of quiet drives with fluid bearings and no fans led me to realize that I could get a bunch of office space back if I upgraded the important machines to something quiet, and there have been a bunch of interesting ITX-based system designs that would work well at ``reducing the ambient clutter of machines''.

Along comes the NetDrive, which does all of this ``in a box.'' The 40G version seemed a bit small to bother with, what with 250G drives shipping from Western Digital already, but then the 120G version came out... A few questions to their support list gave me enough hints to think I could probably make this work in my environment, and that it would at least be interesting to try.

The box arrives

The box showed up promptly, with a very simple instruction pamphlet. Plugged it in, it got an address from my open wireless net, popped up a web interface, and I could set the name and do a few simple things from there. It's still clearly all in the clear, it doesn't use SSL at all, but the usual server certificate model isn't going to work here. Still, it did work as described and the risk has to do with access to the configuration of the SMB server - the operating system is locked down quite tightly, all things considered.

In my position as a long-time security specialist, this was of course not nearly enough. One of the additional reasons to use OpenAFS is that not only does it use Kerberos for authentication, the data transfers are also encrypted. So, once it was clear that it worked as advertised (which it certainly does), it was time to start poking at it.


While the instructions don't involve any more connections than power, the box has an obvious full set of rear connectors. This means the box will probably have a bunch of other uses, but for starters, it means hooking up a PS/2 keyboard and VGA monitor. Sure enough, there's a login prompt. Hitting ctl alt del performs a normal linux-style shutdown, and reboots; after an interesting netboot option menu (it appears that system uses Intel's PXE) it drops to a LILO prompt.

At this point, use martian init=/bin/sh lets the kernel start, mount the root partition, and drop us into a shell, to poke around. (Note that if you're following this in order to try it at home on your own box, you don't have to go this path - this is just the way to explore an unknown system.) During this ``poking'' process, we discover a few things:

It's a Debian system!
It appears to be 3.0 without security updates (and in fact the security line in /etc/apt/sources.list appears to be incorrect, you need to fix that before an apt-get update is useful... and it will be useful, samba has had several security updates post-3.0.)

The user data is not stored as root
This is straightforward, but important - there's a distinct guest account that files are under, which provides a good deal of privilege isolation from the rest of the system.

The guest account name is a big hint
Once you see what the account name is, you should be able to guess the root password (if you haven't been able to guess it already.) If you can't, you can always just tweak the /etc/passwd entry while you're exploring.

Continuing along the exploration path, it is prudent to save enough of the original installation out of the way, so as to be able to restore things to the original Martian configuration later on. At the same time, after checking that root's dotfiles aren't booby-trapped, it's clearly easier and safer to do this from a real login, so this is a good time to reboot and then just log in as root. A simple way to do this is to copy root's dotfiles and then just tar everything up, carefully excluding things that don't pack well and aren't worth saving:

  # mkdir .orig
  # cp -a .??* .orig/
  # tar --exclude /root --exclude /swapfile --exclude /dev \
         --exclude /proc --exclude /home/ \
         -czvf .orig/everything.tar.gz /

Next, now that you have a backup of the original /etc/passwd, change the root password. If you're going to bring this machine up on the net with any remote access features, you want to be sure someone else (or some tool) doesn't just try it.

Having done that, you may find it useful to enable sshd so that you can get in from somewhere more comfortable. Along with changing the password, you'll want to change the host keys, since they're also duplicated on other machines out there:

  # cd /etc/ssh
  # rm sshd_not_to_be_run
  # ssh-keygen -N '' -f ssh_host_rsa_key -t rsa
  # ssh-keygen -N '' -f ssh_host_dsa_key -t dsa
  # /etc/init.d/ssh restart

* find and be amused

Preparing the box for other use

Since I'm not going to run any inherently insecure services on the box, I'll start with turning off samba and thttpd. Note that these aren't bad implementations - they are simply used in insufficiently secure ways for me to be willing to expose them to the open internet.

  # /etc/init.d/samba stop
  # /etc/init.d/thttpd stop
  # update-rc.d -f remove thttpd
  # update-rc.d -f remove samba

Note that /etc/init.d/samba contains some changes to handle a Martian-specific samba.martian.wrapper; since Debian marks this as a conffile, these changes are protected from upgrades. Also, /etc/init.d/thttpd has code that stuffs the IP address of the machine into the setup.html that is visible from the SMB share so that non-Rendezvous-enabled users can still find the box.

Next, we need to fix /etc/apt/sources.list by changing the security line to

  deb stable/updates main contrib non-free

so we can take the outstanding security updates:

  # apt-get update
  # apt-get -u upgrade
  The following packages will be upgraded
    libcupsys2 libldap2 libpng2 libssl0.9.6 perl perl-base perl-modules samba samba-common wget

We want these (I'm a little surprised that Martian hadn't pushed out a samba update already, actually) and if we do need the old ones back for some reason, we still have .orig/everything.tar.gz from above.

While we're at it, there are a bunch of other packages I tossed in during the course of this effort; you'll be happier pulling them in up front instead of going back one by one.

useful for downloading

useful for certain diagnostics (strace is already present)

for moving data around

for repartitioning

in case you ever want to put memory in the second slot

not that things in hwtools are hugely useful, but that /etc/init.d/hwtools already has a place to add hdparm settings. In particular, change
  #   hdparm -q [PUT ARGS HERE]


     hdparm -u 1 -d 1 /dev/hda1

(and reboot, or run that command directly) to get signficantly improved disk performance.

Give it a real hostname

As a server you find with Rendezvous or SMB solely on a local net, you don't need a fixed address and hostname. However, as an OpenAFS server, you need to have an address that stays fixed so that the Volume Location Server can tell clients where to find the data they've requested. If you run ifconfig you should see entries for

which is the Wireless (802.11) network interface

which is the Wired (100mbit UTP) network interface

Both of these list an HWaddr entry; this is the MAC address which you need to tell your DHCP server about. Note that the machine will always try to bring up both interfaces; you may wish to adjust /etc/network/interfaces to let you switch it from one to the other manually. I found it simplest to have both interfaces listed in the DHCP server with duplicate host stanzas, as I'd only use one mode at a time - wired for initial large fast data transfers, and then wireless for when I want to put it on a shelf somewhere out of the way (the ultimate goal.) This approach could use work, but I'll need to see what my use cases work out to in practice.

Be sure to set up a DNS entry as well, and a reverse-inaddr listing so that when you later configure Kerberos it has a chance of working.

OpenAFS client

Since the OpenAFS client includes kernel modules, we need ``kernel stuff'' around to build against. Looking at dpkg -l "kernel*" shows that kernel-source-2.4.18 and kernel-source-2.2.22 are installed, which is odd because

However, the kernel in use is properly packaged, and there is a /boot/config-2.4.20-martian so we can work with it, even if they've failed to correctly walk the two obvious GPL paths (``promise in writing for 3 years'' or ``sources included with''.) So, assuming they don't have any patches (or rather, that we can build the module without caring about any patches, I haven't looked at the wireless support closely other than to note that it uses the prism2_pci module), download 2.4.20 from

  Downloaded link:
  Suggested file name: linux-2.4.20.tar.bz2

Then get the debian-packaged OpenAFS sources:

  apt-get install openafs-modules-source

Unpack the kernel source, so that we can at least configure it:

  cd /usr/src
  tar xjf linux-2.4.20.tar.bz2

Pull in the existing config (the /boot/config-* convention, all by itself, is a reason to use debian :)

  cp /boot/config-2.4.20.martian linux-2.4.20/.config

Follow the instructions in /usr/share/doc/openafs-modules-source/README.modules:

  tar xzf openafs.tar.gz
  cd linux-2.4.20
  make-kpkg configure
  make-kpkg modules_image
  dpkg -i ../openafs-modules-2.4.20_1.2.3final2-6+10.00.Custom_i386.deb

And now we do the normal client configuration.

  apt-get install -u openafs-client openafs-krb5

debconf questions and answers:


add a zero, the default is kind of small and disk is cheap


db servers
leave blank

start client at boot
no, because while having the client on the server is useful, having it as a boot dependency is bad, and since there won't be local user accounts on this box, it's sufficient to just start the client when we need it.

Finally, start the client:

  /etc/init.d/openafs-client force-start

This takes a while the first time, because it is initializing the cache (/var/cache/openafs.)

OpenAFS server

Kerberos Support

In order to load the relevant keys, it is easier to get Kerberos set up right up front. Since I also use ssh-krb5 for everything, we take care of that too.

First we install krb5-user because it has kadmin in it:

  apt-get install krb5-user

Answer the config questions the first time, but note that it turns out that this isn't enough to make kadmin work, we need to answer some of the higher priority questions; instead of changing the priority just this once, we explicitly run dpkg-reconfigure and answer the additional questions:


Does dns contain pointers
no (because even if you have them, there isn't a srvloc type for the admin server.)

Usually kerberos.your.domain

Usually the same, but it might need to be kerberos-1 or something that explicitly always gets the master server.

Before we try to use Kerberos, we need to get the clock in sync. You could set it manually (it only needs to be within about 5 minutes) but rather than leave a lurking problem, I just install ntp. Either way, you want to first run <tzconfig> to set the timezone so that you don't get confused about the resulting values.

        apt-get install ntpdate ntp-simple

If you don't know an ntp server offhand, you may be able to use time.your domain or time.your ISP. Also, unless you know otherwise (ie. you explicitly configure broadcast NTP) I'd suggest letting debconf overwrite the config file.

Once the clock is in sync, we can configure the keys we need (if you don't also run the kerberos realm, just ask the admin for a host/whatever key.) In the examples below, assume that your

DNS domain

server hostname
is martian-afs-server

Kerberos realm
is EXAMPLE.COM (case sensitive, but the convention is for it to be the all-caps duplicate of the DNS domain.)

kerberos principle that has admin privileges
is you/admin

  # kadmin -p you/admin
  kadmin:  ank -randkey host/
  WARNING: no policy specified for host/; 
     defaulting to no policy
  Principal "host/" created.
  kadmin:  xst host/
  Entry for principal host/ with kvno 3, 
     encryption type Triple DES cbc mode with HMAC/sha1 
     added to keytab WRFILE:/etc/krb5.keytab.
  Entry for principal host/ with kvno 3, 
     encryption type DES cbc mode with CRC-32 
     added to keytab WRFILE:/etc/krb5.keytab.

Now we have a key, make it useful. Edit /etc/hostname to contain the full name, also run

  # hostname

(/etc/init.d/ takes care of this on reboot.)

Now install ssh-krb5 itself, and give yourself permission to log in as root using Kerberos (you may wish to use you/root instead, depending on your site policy.)

  # apt-get install ssh-krb5
  # echo you@EXAMPLE.COM > ~root/.k5login

Finally, confirm that it works, by running

  $ ssh

from some other machine on which you have the relevant tickets. (At this point, you can also go sit somewhere comfortable and work from your laptop, instead of working at whatever keyboard and monitor you've hooked up... in fact, you shouldn't need local access at any further point.)

While you're in there, you'll want to fix the first line of /etc/hosts to only mention localhost, and not the leftover debian and martian entries - change       debian  localhost martian

to       localhost

(I noticed and fix this while looking for another problem, which turned out to be clock skew related; I don't know if this fix is actually needed, since you'll be taking care of the clock early enough to avoid that problem. But do it anyway :)


It turns out that OpenAFS really expects to own the partition it puts volumes on. It does use the filesystem there, and I suspect the restriction is mostly a carryover from the non-linux platforms that OpenAFS supports, but it does sense to keep the OS and data partitions distinct, if nothing else because it can make fsck faster, and if you end up needing to replace the OS for some reason, the data should be untouched.

You can actually do this step much earlier in the process, but you don't need it until you actually want an OpenAFS server running.

First, reboot single user (you should simply use martian single at the LILO prompt, since you now know the root password, rather than the /bin/sh trick earlier.)

Next, do the easiest step: shrink the root to something small (plenty of room for the OS and caches, but otherwise leaving most of the disk for filesystem use). Note that I didn't run lilo (to correct lilo's list of blocks for finding the kernel) when I did this - I believe it worked because none of the kernel was in the space above the reduced extent of the partition. (I have run lilo for other reasons, long after this, and it did work, so the config file present is sufficient - but I think you'd actually have to reboot here before running lilo anyhow to get back in sync with the on-disk form of the partition, as well as to remount it writable.)

   # ext2resize /dev/hda1 4G
   # reboot

(Later steps don't work any *better* in single user mode, but this one did need the root to actually be readonly and stable.) Once it comes up, we repartition. Unfortunately, the default partitioning assumes a different geometry than the kernel comes up with by default, so the simple approach leaves the machine unbootable...

I haven't yet had a chance to test this procedure on a second box so be very cautious, and prepared to hook up a CDROM drive to boot a rescue disk from (or figure out how to use PXE to netboot something; I'd welcome tested pointers for inclusion here.)

First, here's the partitioning I ended up with:

  Disk /dev/hda: 255 heads, 63 sectors, 232581 cylinders
  Units = cylinders of 16065 * 512 bytes

     Device Boot    Start       End    Blocks   Id  System
  /dev/hda1             1       536   4305388+  83  Linux
  /dev/hda2           537       667   1052257+  83  Linux
  /dev/hda3           668     14593 111860595   83  Linux

  Command (m for help): x

  Expert command (m for help): p

  Disk /dev/hda: 255 heads, 63 sectors, 232581 cylinders

  Nr AF  Hd Sec  Cyl  Hd Sec  Cyl    Start     Size ID
   1 00   1   1    0 254  63  535       63  8610777 83
   2 00   0   1  536 254  63  666  8610840  2104515 83
   3 00   0   1  667 254  63 1023 10715355223721190 83
   4 00   0   0    0   0   0    0        0        0 00

Note that F</dev/hda2> is actually a rescue install I made so that I
could fix some of the unsuccesful attempts - but for an OpenAFS
client, it might be useful to just remount it on F</var/cache/openafs>
for a 1G fixed cache.

The critical bit is that

  Disk /dev/hda: 255 heads, 63 sectors, 232581 cylinders

says 255 heads; apparently what caused the trouble involved fdisk thinking there were only 63 heads, with the ensuing problems. To be sure things work out, run fdisk /dev/hda and then

go into eXpert mode

print the partition table in the detail above with exact H/S/C values. Save this for checking the later step, in particular save the values of the first partition.

h 255
If the head count is wrong, explicitly set it to 255 and p it again; it shouldn't change the output, but if it does, make sure you understand it.

Return to the normal comand mode, and partition the disk:

d 1
delete the first (only) partition

n 1 r +4096k
create a new regular partition, 4G in size

d 1
check the number of cylinders, then delete the partition so you can specify that number directly

n 1 r 536
create the partition with 536 cylinders

n 2 r +1024k
create the 1G cache partition (``just in case'')

n 3 r RET RET
create the third partition, which will be /vicepa, with the default options that make it the entire remainder of the disk.

go back into eXpert mode

and print out the table. Now compare the first partition (the one with the root filesystem on it) with the values you saved above. In particular, check that the starting Hd, Sec, and Cyl columns have the same values (the second triplet will be different and reflect your partitioning above, namely they should be 254,63,536.) If the first triplet isn't 1,1,0 then you need to figure out what's going on, assuming that's what you saw the first time.

write it out, only if you're happy with the x p output previously.

If the start of the partition isn't correct, I recommend looking at sfdisk, which is included with fdisk and gives you complete raw control; in particular, I used a command close to

      echo 1,8610839,83 | sfdisk /dev/hda -N1

which tells sfdisk to operate on disk /dev/hda, partition Number 1, and then it reads values from standard input, namely

start sector, 1; I assume that it skips ``track'' 1 (and starts on head 1 rather than head 0 as later partitions do) automatically, otherwise it would need to be 63, but sfdisk -n -l /dev/hda does list the same value.

8610777 is the partition size that passed fdisk's sanity check; I had to do a little arithmetic and go back and forth to get this right.

83 is the standard ``Linux'' partition id.

Now that the partition table looks good, reboot. Note that whatever happens, lilo will still boot the kernel because it has explicit block numbers for it; if it all worked, you'll be able to log in as usual, if it didn't work, you'll get an obscure reiserfs-related error message, simply because it'll try all of the available partition types and only complain about the last one that failed. In that case, you're stuck with opening the box, installing a CDROM drive (the only trick here is that you'll need a power-splitter, or ``Y'' connector, because there's only one old-style fat power cable end. You can use any random IDE CDROM drive, as long as you don't want to close the case up.) I recommend just using debian media to install onto /dev/hda2 so you can look at man pages and work with sfdisk and possible debugfs and the mount option offset, since that partition will work fine, while you poke at the primary partition.

If it did succeed, now you can do the usual partition creation, and add the journal:

  # mke2fs /dev/hda3
  # tune2fs -j -c 127 -i 12m /dev/hda3
  # mkdir /vicepa

Finally, add the /etc/fstab entry,

  /dev/hda3       /vicepa         ext3    defaults      0       3

and mount /vicepa (or reboot) to make it available.

OpenAFS Fileserver install

Finally, we actually install the server. Having previously mounted /vicepa, all we need to do is copy over the server config and start it up.

  # rsync -e ssh -avu otherserver:/etc/openafs/server /etc/openafs/

Note the importance of copying this securely; /etc/openafs/server/KeyFile itself is the thing that needs protection, as it is the shared key among all of the servers.

  # apt-get install openafs-fileserver
  # bos create martian-afs-server fs \
      fs -cmd /usr/lib/openafs/fileserver \
         -cmd /usr/lib/openafs/volserver \
         -cmd /usr/lib/openafs/salvager -localauth

Confirm that it worked:

  # bos status localhost -noauth
  Instance fs, currently running normally.
      Auxiliary status is: file server running.

At this point, you can do your normal OpenAFS administration; I generally start by (using tokens with the relevant privileges, on another machine - you don't need to ever log in to the Martian box again, other than to pull in Debian security updates, at this point) adding an entry to the ``at-a-glance'' partition directory:

   $ vos create martian-afs-server a disk.martian-afs-server.a
   $ fs mkm /afs/ disk.martian-afs-server.a
   $ fs sa /afs/ you rl

And then check it:

   $ fs df /afs/ 
   Volume Name                  kbytes      used     avail %used
   disk.martian-afs-server.a 104511260  22772468  81738792   22%

(Ok, you got me, I've used it a bunch already...)




This page was produced in emacs 21 using Perl Pod format under Mac OS X. HTML produced by pod2html 1.03 and published using RCSWEB.


Copyright 2003, Mark W. Eichin <>. Republication permission available; otherwise, href is free - Link, don't Copy.