Sun gave the Apache Software Foundation a server last year, and I kept hearing, over coffee and beer, that they were running some scary-huge number of projects on it, all independently via zones; really remarkable numbers. I kept asking them to write about it, and they kept not writing. So here’s an email interview with Mads Toftum, who does a lot of sysadmin-ing around the ASF. I don’t know how typical their workload is, but I’m an old sysadmin myself and I found this pretty interesting. Mads doesn’t blow his own horn much, but this is a remarkable installation.
Tim: What was donated, and when was that?
Mads: A v40z (4 single core AMD, 16G memory) and a 3511 disk array with 5*250G IDE drives. I don’t have an exact date but it was right around the release date for sol10.
How was the installation/setup? Surprises?
We had a sun engineer doing the work for us, so no real surprises other than him using the most common root password ever. My day job includes more than enough Solaris that there weren’t many surprises other than a few missing bits in x86 Solaris compared to the SPARC based gear I’m used to. The raid array did cause a fair bit of grief, but eventually we got that under control and I moved the zones over to it during apachecon.
What are you using it for, and how is it set up?
The setup is pretty standard - the infrastructure team runs the global zone and try to provide most common software on request.
The way it generally works, a PMC can ask for a zone to run their software - then I build a zone for them, and hand over root to whomever they have as their admin. After that, we generally don’t hear much from them unless they get themselves into trouble or we have to take care of runaway processes.
There’s close to 20 zones now spread over different projects doing things like automated builds, doc generation (fairly common), TCK testing and automated tests. Spamassassin is by far the greatest CPU hog, but there are also others using a fair bit of resources to build docs, run gump and use mavens continuum. We’ve also got a couple of zones that are used to develop infrastructure services like projects.apache.org and other things that aren’t public yet.
We’re still stressing that it is an experimental service but a few of the projects are ignoring that and starting to depend on their zone (I just hope they don’t have to learn about experimental the hard way).
How heavy’s the workload? System holding up OK?
We’ve had a couple of incidents over time and I’m seeing more and more of the automated builds beginning to overlap - ideally I’d like to find another box to move the automated build activities out of the way. I’ve asked to postpone a generic build zone project because I want to make sure it doesn’t impact other projects.
Why zones?
They are the only way we can share a machine between so many projects without placing a very high load on our infra people. Trying to sort out 20 different projects within the same machine would be close to impossible because we couldn’t delegate privileges.
If you were doing it with Linux instead, how would you approach the problem of dealing out the projects? Same question for *BSD.
There’s jails for BSD and similar solutions for Linux, but at the time we were setting up the machine, none of those solutions would have given us the same level of flexibility and virtualisation without adding extra overhead. My guess is that we would have gone with a solution similar to what we have, but we might have ended up with a more restricted model.
Using SMF? Like it?
After getting over the initial disgust over all the added xml cruft, I’ve gotten as far as seeing the benefits, but SMF still needs more work as the SMF discussions on OpenSolaris.org shows.
What do the Solaris engineers need to be working on?
Better patch management (patching zones is no fun) and being able to use liveupgrade with zones would make me a lot less worried about upgrading from several thousand miles away.
resource caps / FSS integration for other things than just CPU - we need to be able to limit other resources like memory and possibly IO too.
zone migration would be good too.
dtrace in zones
Luckily all these are already in the works and I expect seeing the last 3 in a not too distant future release. Other than that, there’s still some work to be done on better hardware support in the X86 area and a few other minor details.
CPU-limited, I/O-limited, memory-limited?
All of the above although memory problems are very rare - people generally try to behave and don’t quite use as many resources as they could have gotten away with. Most projects run in a single process gobbling only one out of 4 CPUs, but recently that has been changing.
Which Apache project burns the most resources?
Spamassassin by a wide margin although others are working hard to catch up. This is of course something I can make less evident with FSS, but as mentioned earlier that doesn’t quite fix all our problems.
I think that’s about it - we’re talking a bit about moving some of our other services to zones, but that probably won’t happen as long as we’ve only got the one machine.
