May 01 11:57:28 <mbp> oh, incidentally, i thought you might be interested in this:
May 01 11:57:29 <mbp> http://samba.org/ftp/unpacked/distcc/README
May 01 11:58:54 <daniel> yes I do
May 01 11:58:55 <daniel> much
May 01 11:59:11 <mbp> it seems to be going really well
May 01 11:59:18 <mbp> i think it's quite promising
May 01 11:59:22 <daniel> in fact, I fully expect to have 3 machines or so underfoot to try that on
May 01 11:59:33 <mbp> cool
May 01 11:59:40 <daniel> it's what I want: more machines = faster compile
May 01 11:59:44 <mbp> exactly!
May 01 11:59:45 <daniel> it's not *that* hard
May 01 12:00:03 <daniel> you have to get the required files to the machine, get the compiled files back
May 01 12:00:07 <mbp> right
May 01 12:00:21 <daniel> the net traffic is not much overhead, anyway, it's pipelined
May 01 12:00:27 <mbp> the neat thing about shipping the preprocessed source is that there is only one file going in each direction
May 01 12:00:45 <mbp> http://samba.org/ftp/unpacked/distcc/src/HACKING
May 01 12:01:19 <daniel> and, because it's not secure, it will fun inserting compiler-viruses
May 01 12:01:21 <daniel> new sport
May 01 12:01:41 <mbp> the other situation where it works nicely is when you're working on a laptop but happen to have  a faster machine around
May 01 12:01:44 <mbp> um
May 01 12:01:48 <mbp> i did think about that
May 01 12:01:59 <mbp> you can imagine running it over ssh
May 01 12:02:19 <mbp> but there will be a fair cpu hit from that, possibly enough to make it impractical
May 01 12:02:39 <daniel> where did you see the part about the preprocessor?
May 01 12:02:55 <mbp> which part about the preprocessor?
May 01 12:03:08 <daniel> <mbp> the neat thing about shipping the preprocessed source
May 01 12:03:12 <mbp> oh, right
May 01 12:03:21 <mbp> so, i always run cpp on the local machine
May 01 12:03:33 <mbp> and then ship that to the remote one, which runs cc1, cc2, as, etc
May 01 12:03:49 <mbp> that avoids the need to have header files etc on the remote machine
May 01 12:03:49 <daniel> ok
May 01 12:03:50 <daniel> right
May 01 12:03:52 <daniel> headers
May 01 12:03:55 <daniel> yesss
May 01 12:03:57 <mbp> running cpp is really cheap compared to the other stages
May 01 12:04:08 <daniel> yup
May 01 12:04:16 <-- digdude has quit (Ping timeout: 272 seconds)
May 01 12:04:27 <mbp> in fact, i think that insight is the really novel part that makes it work
May 01 12:04:31 <daniel> I need this
May 01 12:04:34 <mbp> without that it's possible, but just too hard to set up
May 01 12:04:40 <daniel> yes
May 01 12:05:05 <mbp> e.g. sun have a distributed build system, but you need all the machines to (a) share an NFS directory mounted at the exact same location, and (b) have the exact same system headers
May 01 12:05:11 <daniel> nice hack, now lemessee - I should be able to get a sub-minute compile with that
May 01 12:05:28 <mbp> it's a bit rough at the moment, but it works
May 01 12:05:48 <mbp> i booted a kernel built using it, and it seemed to work
May 01 12:05:53 <mbp> (how' that for confidence :-)
May 01 12:06:24 <mbp> one thing i might do is gently gzip -2 the cpp output; it tends to be v large and v compressible;
May 01 12:06:36 <mbp> at the moment the network seems to be the limiting factor
May 01 12:06:40 <mbp> so, please try it out
May 01 12:06:43 <mbp> it's in anoncvs
May 01 12:06:52 <mbp> grab me if you have any issues
May 01 12:06:57 <mbp> basically, you just need to do 
May 01 12:07:10 <mbp> make; ./distccd&; CC=`pwd`/distcc
May 01 12:07:17 <mbp> make; ./distccd&; CC="`pwd`/distcc gcc"
May 01 12:07:19 <mbp> and off you go
May 01 12:09:26 --> sfr (~sfr@CPE-203-51-25-76.nsw.bigpond.net.au) has joined #ozlabs
May 01 12:09:50 <daniel> that would be a most interesting number
May 01 12:09:59 <mbp> ?
May 01 12:10:14 <daniel> i.e., does the compression cost more than the transmit time it saves
May 01 12:10:14 <daniel> very interesting
May 01 12:10:18 <mbp> oh, right
May 01 12:10:26 <mbp> in fact, compression increases scalability
May 01 12:11:01 <mbp> it consumes more cpu on the master machine, in return for being able to squeeze more stuff across the network and therefore do more work remotely
May 01 12:11:53 <daniel> it might really come down to how fast the network is.  Another thing, the network transmit really just turns into latency, not loss of throughput
May 01 12:11:58 <mbp> with e.g. a really slow laptop, 100mbps network, and fast remote machine, you probably wouldn't want it because the laptop might be flat out running cpp and sending the files
May 01 12:12:10 <daniel> so by that measure, it's better to leave all the cpu for compiling
May 01 12:12:38 <mbp> well, at hp we have 3 x 1700Mhz P4s, and a 100Mbps non-switched hub
May 01 12:12:51 <mbp> on that situation, the network saturates and the remote machines are ~50% idle
May 01 12:13:06 <mbp> so, possibly spending a *little* time on compression would get more work done
May 01 12:13:40 <mbp> back-of-envelope tests with "time gzip -2" vs "time cpp" show it uses relatively little cpu
May 01 12:13:59 <mbp> oh, however, we can do more pipelining here than we do at the moment, which might help the situation
May 01 12:14:06 <mbp> the network is only intermittently flooded
May 01 12:14:26 <mbp> it may be better to run cpp into a pipe, and gcc from a pipe, so that their work can be overlapped
May 01 12:14:31 <mbp> at the moment i just use tmpfiles
May 01 12:18:12 --> digdude (~digdude@dsl-g1-42.surfcity.net) has joined #ozlabs
May 01 12:18:13 --> dave0 (^dave@cabal.evilplot.org) has joined #ozlabs
May 01 12:19:50 <daniel> mbp, I didn't realize right away that this is your project
May 01 12:19:56 <mbp> oh right :-)
May 01 12:20:01 <mbp> that's why i like it so much :-)
May 01 12:20:12 <daniel> hey, I like it more now :-)
May 01 12:21:57 <daniel> well I wasn't quite precise about the pipeline thing
May 01 12:22:21 <daniel> *so long as the compile is faster than the network* you're better off using the cpu for the compiler
May 01 12:22:27 <daniel> I thinkg ;)
May 01 12:22:46 <mbp> hm
May 01 12:23:06 <daniel> erk
May 01 12:23:08 <mbp> like many parallelization things, even though it gets less efficient as you increase parallelism, you still get more work done
May 01 12:23:11 <daniel> I meant the opposite
May 01 12:23:16 <mbp> well
May 01 12:23:43 <daniel> *so long as the compile throughput is less than the network throughput* you're better off using the cpu for the compiler
May 01 12:23:55 <mbp> do you mean using the master's cpu?
May 01 12:24:22 <mbp> i think it's a qn of opportunity cost
May 01 12:24:23 <daniel> if the master has time left over, should it not be compiling too?
May 01 12:24:34 <mbp> by default (without distcc), the master's cpu does all the compilations
May 01 12:24:39 <mbp> oh, right
May 01 12:24:42 <mbp> absolutely
May 01 12:24:58 <mbp> so typically you set DISTCC_HOSTS='localhost jonquille nevada'
May 01 12:25:07 <mbp> and it distributes jobs amongst the three
May 01 12:25:22 <mbp> "localhost"  is a special case, of course, where it just execs() the compiler directly 
May 01 12:25:31 <daniel> so... what are your numbers?  How far off linear?
May 01 12:26:24 <mbp> it's early days, but i got a 40% reduction in wall clock time from adding a second machine
May 01 12:26:53 <mbp> oh, sorry, 
May 01 12:26:58 <mbp> from adding a second *faster* machine
May 01 12:27:38 <mbp> you can see some numbers in the readme
May 01 12:27:58 <mbp> running at hp threw up a few small bugs; i haven't rerun benchmarks yet
May 01 12:28:26 <daniel> it's the coolest hack I've heard about for some time
May 01 12:28:38 <mbp> wow, thanks (-:
May 01 12:28:42 <mbp> i thought that too
May 01 12:28:50 <mbp> i just woke up one morning and the idea was in my mind
May 01 12:29:17 <mbp> of course, you need to run it with make -j<something>, so that it runs local and remote jobs in parallel
May 01 12:29:54 <mbp> the thing is that while sending across the network takes a lot of time, it takes barely any CPU, and so it basically does not interfere with compiling something else locally at the same time
May 01 12:30:08 <daniel> right
May 01 12:30:45 <mbp> so, even if you get just a few jobs done remotely, you've still won
May 01 12:31:17 <mbp> (local compiles become very slightly slower, because we need to decide where to run them, but not by much)
May 01 12:31:35 <mbp> of course, if there was only a small improvement, people might not be bothered to install it
May 01 12:31:46 <daniel> I was thinking something related: when I'm on my road, rsync my source files back to my server, compile there, and send the binary back to me
May 01 12:31:51 <daniel> faster than compiling on the laptop
May 01 12:31:51 <mbp> but i reckon installation can be quite simple, and the improvement is actually quite large
May 01 12:31:54 <mbp> yep
May 01 12:31:56 <mbp> i've tried that
May 01 12:31:59 <mbp> but it's too much hassle!
May 01 12:32:09 <mbp> this way, you can compile from within emacs (or whatever)
May 01 12:32:10 <daniel> not if somebody makes it easy :)
May 01 12:32:22 <mbp> and remote error messages come up looking exactly as if they were local
May 01 12:32:35 <daniel> and I did waste a lot of time waiting for my laptop
May 01 12:32:40 <mbp> line numbers and filenames in errors exactly correspond to ones on your local disk
May 01 12:33:05 <daniel> it takes 3 minutes to do a diff of two trees on the laptop, 1.5 seconds on the server
May 01 12:33:10 <mbp> oh, the other problem with rsyncing is that it becomes bloody confusing if the remote machine's libraries are out of sync
May 01 12:33:13 <daniel> some obvious
May 01 12:33:18 <daniel> some obvious advantage there
May 01 12:33:38 <mbp> i guess rsync is a better solution if there's a really slow link between you and the fast machine
May 01 12:33:46 <daniel> I can see a lot of potetential for confusion
May 01 12:34:09 <mbp> 33k6, not 100m
May 01 12:34:47 <mbp> oh
May 01 12:34:57 <mbp> tim just pointed out that our hub says on the front "10Mbps"
May 01 12:35:02 <mbp> so it's probably not 100Mbps :-/
May 01 12:35:34 <mbp> i'm sure we asked for a 100mbps one
May 01 12:35:36 <mbp> fooey
May 01 12:38:15 <mbp> "i've always wanted to use sendfile(), but never had a reason until now"
May 01 12:38:15 <mbp> :-)
May 01 12:39:23 <dgibson> mbp: interesting
May 01 12:39:30 <dgibson> mbp: I played with distributed compiles a few years back
May 01 12:39:47 <dgibson> mbp: nothing as clever as this - just gcc invoked over ssh on an NFS share
May 01 12:39:52 <mbp> mm
May 01 12:39:53 <mbp> yep
May 01 12:40:00 <mbp> that's the standard solution
May 01 12:40:28 <dgibson> I suspect the CPU load per se of ssh would be acceptable, but the start up cost is very high so you'd need to do as much as possible over one ssh connection
May 01 12:40:33 <mbp> right
May 01 12:40:41 <mbp> that was my original idea; i wanted to hoard ssh connections
May 01 12:40:52 <mbp> tridge persuaded me to not use it by default
May 01 12:41:10 <dgibson> What are you using as the transport now?
May 01 12:41:30 <mbp> it also reminded me of ANU's bunyip, where they just use rsh and so on inside the secured network
May 01 12:41:41 <mbp> dgibson: plain TCP, with an easily parseable format:
May 01 12:41:48 <dgibson> right
May 01 12:41:56 <mbp> 4-character command, 8-character hex length, data
May 01 12:41:58 <dgibson> So you run a server which listens on the farm machines?
May 01 12:42:05 <mbp> right
May 01 12:42:12 <mbp> either from inetd, or standalone
May 01 12:42:32 <mbp> standalone is a bit better, because then it can control the number of jobs running at once
May 01 12:42:34 <tpot> distcc over jabber!
May 01 12:42:48 <dgibson> Hmmm.. you could do that anyway
May 01 12:42:53 <mbp> from inetd, yep
May 01 12:42:57 <dgibson> The other big problem you will encounter is that most Makefiles are complete crap
May 01 12:43:03 <mbp> and can't handle -j?
May 01 12:43:13 <mbp> or have gcc hardcoded?
May 01 12:43:24 <dgibson> mbp: If running from inetd running over ssh should be pretty easy, since it's already using stdin
May 01 12:43:35 <dgibson> Can't handle -j properly
May 01 12:43:56 <mbp> well, y'know, people have to make *some* effort if they want parallel compiles
May 01 12:44:02 <dgibson> The for $$i in $(SUBDIRS); do $(MAKE) -C $$i; done construct is the most common problem
May 01 12:44:37 <mbp> mind you, even if you can't parallelize, transparently shifting compilation to a faster machine (e.g. laptop -> regatta) might be worthwhile
May 01 12:44:50 <dgibson> That's true
May 01 12:45:45 <dgibson> Presumably you could plug this into the backed of ccache quite easily too
May 01 12:45:51 <dgibson> s/backed/backend/
May 01 12:46:00 <mbp> oh, right
May 01 12:46:23 <mbp> in fact it works quite nicely, because ccache runs cpp itself to work out whether the files have really changed
May 01 12:46:41 <dgibson> That's exactly what I was thinking
May 01 12:46:42 <mbp> and so the marginal cost to the master machine of runing distcc becomes v small
May 01 12:46:53 <mbp> CC='ccache distcc gcc' works well
May 01 12:49:04 <dgibson> Can you control what preference is given to each machine
May 01 12:49:16 <dgibson> e.g. for each compile on my laptop run 24 on the regatta?
May 01 12:49:23 <mbp> that part of it is v crude at the moment, but yes
May 01 12:49:37 <mbp> DISTCC_HOSTS='localhost ego ego ego ego ego ego ego ego ego ego'
May 01 12:49:44 <mbp> they're uniformly distributed
May 01 12:49:49 <mbp> it's kind of stochastic at the moment
May 01 12:49:54 --> vmarshall (~vmarshall@user-vcaur38.dsl.mindspring.com) has joined #ozlabs
May 01 12:49:54 <dgibson> right
May 01 12:50:04 <tpot> mbp has a big ego?
May 01 12:50:07 <mbp> really the client ought to keep track of how busy it thinks each slave is, but i haven't done that yet
May 01 12:50:12 <mbp> tpot: no, paulus does
May 01 12:50:15 <dgibson> tpot: No, lots of them, but they're each quite small ;-)
May 01 12:50:55 <dgibson> At least make -jn now works meaningfully in recursive makefiles
May 01 12:51:04 <dgibson> It didn't when I was playing with this which was a big problem
May 01 12:51:22 <mbp> dgibson: i've tried building rsync, samba, and 2.4 with it, with apparent success
May 01 12:51:33 <mbp> i guess the next biggest projects would be all of GNOME, or mozilla
May 01 12:51:47 <dgibson> heh. now try X, and do battle with the monstrosity that is imake
May 01 12:52:19 <mbp> is imake used instead of make, or does it generate a makefile?
May 01 12:52:34 <dgibson> It generates a makefile
May 01 12:53:33 <mbp> hm
May 01 12:53:34 <mbp> oh well
May 01 12:53:38 <mbp> i'll try it sometime
May 01 12:53:45 <dgibson> well, many makefiles actually
May 01 12:54:14 <dgibson> I'd recommend not eating right beforehand
May 01 12:54:17 <mbp> ha ha
May 01 12:54:44 <mbp> well, it's not necessary that distcc help every single project
May 01 12:54:53 <mbp> anyhow, let me know if you get a chance to try it
May 01 12:59:55 <dgibson> do you pick a target machine in the farm randomly or deterministically?
May 01 13:06:52 --> Rasmus (~rasmus@07-034.006.popsite.net) has joined #ozlabs
May 01 13:07:19 <Rasmus> Server has been working ok I take it?
May 01 13:08:34 <tpot> yup
May 01 13:09:33 <Rasmus> it's in Sifry's basement now
May 01 13:17:11 <tpot> i notice you had a major isp change
May 01 13:19:19 <-- shawnm has quit (Client Exit: Client Exiting)
May 01 13:21:37 <daniel> night everybody
May 01 13:21:48 <daniel> birds are singing out there, time for zzz's
May 01 13:24:03 <tpot> cheep!
May 01 13:25:57 --> anton (~anton@CPE-203-51-24-59.nsw.bigpond.net.au) has joined #ozlabs
May 01 13:29:20 <mbp> dgibson: at the moment it's randomly distributed across the specified hosts, 
May 01 13:29:27 <mbp> because i don't know how busy they are
May 01 13:29:48 <dgibson> mbp: ok.  round robin might be better than random and easier than keeping track of how busy they are
May 01 13:30:29 <mbp> yep
May 01 13:30:35 <mbp> that's what i thought i'd do next
May 01 13:30:59 <mbp> it just requires a reliable, simple mechanism of sharing a global counter between all processes
May 01 13:31:10 <mbp> i thought perhaps just truncating a lock file to length n
May 01 13:31:20 <mbp> since races are not too harmful
May 01 13:31:23 <mbp> other suggestions welcome
May 01 13:37:07 <tpot> use a tdb!
May 01 13:40:50 <dgibson> Hmm... if you're not so worried about races surely it would be easier just to write the counter to a file
May 01 13:41:16 <mbp> well
May 01 13:41:33 <mbp> you might have somebody running two compilations in different windows, with different _HOSTS settings
May 01 13:41:42 <mbp> in that case a simple counter would get confusing
May 01 13:41:58 <mbp> it might not be any worse than random, but it's not much better
May 01 13:42:12 <dgibson> True, but that's probably not a terribly common case
May 01 13:42:33 <-- daniel has quit (Ping timeout: 272 seconds)
May 01 13:42:47 <dgibson> the race can be avoided just by doing a fcntl() lock, read value, write value, fcntl() unlock
May 01 13:43:12 <dgibson> Use value % length as the index into _HOSTS
May 01 13:43:16 <mbp> yep
May 01 13:43:25 <mbp> that's what i had in mind
May 01 13:43:50 <mbp> randomness actually turns out to be somewhat more reasonable if each host only accept()s one or two jobs at a time
May 01 13:44:02 <mbp> because everybody else will just block waiting
May 01 13:44:29 <dgibson> ?
May 01 13:46:32 --> enigma (~eriddle@adsl-63-195-7-203.dsl.chic01.pacbell.net) has joined #ozlabs
May 01 13:47:32 <mbp> hm, nevermind
May 01 13:47:33 <enigma> good job Rasmus.  Did you set your DNS to update more frequently in anticipation or do you just keep it that way?
May 01 13:47:40 <mbp> i think my reasoning there is a bit dodgy
May 01 13:47:44 <enigma> hi mbp
