Still working on the Secure NFS thing. Been looking at a couple of kernel patches:
Firstly, epoll. This used to be known as /dev/epoll, but it's now a set of system calls and is merged into 2.5. Patches are on that site for 2.4.
This is basically a replacement for the poll system call (though it is edge-triggered, not level-triggered) and, as the results on the webpage show, works much more quickly for large fd sets. I still have some worries about some multithreading issues with this, but it looks like I'm going to use it.
Secondly, the Kernel Mode Linux patch. This runs processes in the kernel address space, making system calls much faster. Results from my computer are 286 cycles/getpid from user-land and 6 cycles/getpid from kernel-land. This would be nice to have (see below) but, unfortunately, it seems to cause random crashes in (at least) vim and xmms.
To explain why fast system calls would be really nice, consider: My current numbers for the amount of processing I'm going to be doing is 250MB/s in 300 byte packets. That's somewhat pessimistic, but that's what I'm going on. That about 830,000 packets/second. If a system call takes 400 cycles (the 286 figure is for getpid, other system calls do a little more work and with the TBL flushes, it's at least 400) that's 330 megacycles of system calls per second (for 1 system call per packet). But it's going to take more than one system call per packet even with funky vector IO so, basically, I'm looking at about 600 MHz just for system calls. Ouch.
(I reserve the right to ridicule these numbers later)