Here is the graph for one of the projects:
The build initially took about 30 minutes, then I had a period where I jumped to about 70 minutes. This lag was entirely my unit-test and after some refactorings I took is down again to about 30 minutes, which is decent.
Now take a look at another project:
The difference is that the project took all long about 40 minutes, and now I have spikes of 3-4 hours !
What's the catch: well, the first project is taking so long during unit tests since I have a lot of GUI tests, where the code has to sleep and give the interface time to repaint, etc. So, although the time is 30 minutes, it's mostly waiting for the GUI (inside an Xvnc instance) to paint.
The second project though does a massive build where I just produce JARs and don't run any unit tests. It's massively IO-bound.
So, the way I see it, in the past 2 months or so the machine I'm running my VPS on, has been getting slower doing IO request.
IO has always been a problem with my limited VPS experience. First, I got rid of this by moving from the 256mb slice to the 512mb slice since apparently I was just trashing the swap file.
But now, I'm not so certain it's a RAM issue anymore. The 2nd compilation just needs to touch the disk so it doesn't matter how much RAM do I add, after the minimal amount needed for ant and javac.
I'm starting to think I should move the build server onto an EC2 instance. This way I could use a smaller slice just to run Hudson, but do the actual building onto a bigger EC2 instance. I'm not certain it will be cheaper though.
Later edit: The discussion here is continued with my post about using EC2 instances as build servers.
3 comments:
If you do make this switch, please put up another post on the subject. I'm running into similar issues with my slice and would benefit from your experience.
Thanks!
I do plan on switching to a hybrid EC2-slicehost solution after my current iteration will be over. Right now I've just migrated to the 768 mb slice and I have about the same results as I had with the 512 slice in the beginning.
I think using EC2 will also be cheaper, depending on the number of builds you do every month. I only have a maximum of 100 builds or so per month so it will be cheaper to pay for 100 hours of EC2 (which is also a much larger machine).
Also a build server uses little bandwidth so, most likely, you're paying the included bandwidth for nothing.
See continued discussion, this time with EC2: http://emilian-bold.blogspot.com/2010/04/ec2-as-build-server.html
Post a Comment