ANT - speed improvements to existing script - deployment

Question

I've built a set of generic deployment scripts which work great for the majority of our stuff. We've just however introduced our largest project to the setup and we're now finding times are far too varied and long for our liking.
The project size as it stands is 33,226files at a size of 400Mb plus. Times are currently taking between 13mins & 55mins (the last deployments time) depending on certain decisions made by ANT ( more below ).
In terms of the steps we currently do the following on x2 servers:-
1) ANT exports the project from SVN to both servers (made up of 3 parts).
2) It begins to shutdown the Web Services on Server #1.
This was the workaround we put in to stop Windows (2003) file locking failing the deployment.
3) ANT runs a "move" task on the current version (all parts) into temporary folders & moves the exported new version into its place.
4) Customised deployment code is run - one part being to move permanant features from the temporary folder into the new (i.e system files / Web Server Admin tools).
5) Delete the temporary folder.
6) Bring the Web Services back online
... rinse and repeat for the 2nd server steps 2 -> 6.
7) Save the ANT logs.
The main issue I'm having is that the ANT move task seems to make one of two decisions. It either:
a) Very simply swaps the versions over and moves on - taking a minute or two to handling it or
b) Goes through some kind of integrity check that it moves every file and folder from one place to the other. This floods the logs and takes a fair length of time to complete. Hence the 40+ minutes extra added on.
I can't find anything online that explains what causes ANT/OS to make that decision. Option A would be the ideal full-time situation.
I've tried copy, delete separately. I've tried the sync task. All seem to have this slow performance.
So really I'm asking what others with more experienced than me do with deployments of this scale. Do you have any hints / tips on how I could improve / speed this process up? Any ideas what the move is doing and if there is maybe a better way of doing all this?
Thanks a lot,
James

Answer 1

Thanks for the input all.
Just to add an answer to this one I've made the following changes which seem to have knocked a few minutes of it.
The first one was I've changed how the swapping happens off the back of the comment I mentioned before. It seems that ANT will try and do the following :-
"If the target directory does not already exist, Ant will do a rename of the directory. But if the target directory exists, it instead does a copy into the directory and delete from the source directory instead."
I think what's happened is ANT is trying to put the new version in before the old version has been completely removed. So instead of trying to rename the old I've now moved it into a temporary folder and deleted this at the end of the build. That seems to have stabled things on that front.
A few other things I've added to make ANT a bit smart :-)
1) I've set it up so ANT will not deploy any part of the build that is the same as what currently exists. So if part 1 is selected and that's on the Test environment already then it's removed from the build and SVN exports.
2) With the service shutdown / startup I've got ANT reading the response that comes back. If a service tells it that it's already started when calls, as sometimes happens if a service relies on others Windows automatically boots them up, then I've told ANT to hang around and move onto the next one.
Little steps like that seem to have improved things by a fair bit. I'd like to still try and get more out of them but these certaintly have given them a big step.
Thanks again,
James

Answer 2

even we faced this problem of auto deployment script taking longer time in our organization. So initially we had our script running in sequential like cleaning, stopping tomcats, updating, starting tomcats and making sure that all webapps are properly deployed.
So we have done following things like:
1. parallely cleaning and stopping all tomcats
2. do svn swith
3. parallely start all tomcats
4. make sure all webapps are deployed properly using jmx
here is the piece of code :
<target name="all_clean_parallel">
<parallel>
<antcall target="x1_clean"/>
<antcall target="x2_clean"/>
<antcall target="x3_clean"/>
</parallel>
</target>
<target name="all_start_parallel">
<parallel>
<antcall target="x1_start"/>
<antcall target="x2_start"/>
<antcall target="x3_start"/>
</parallel>
</target>
And the piece of code to check whether webapp is deployed propely or not with jmx help :
<macrodef name="mStatus">
<attribute name="aModule" />
<attribute name="aHost" default="localhost"/>
<attribute name="aPort" default="9012"/>
<attribute name="aMaxWait" default="240"/>
<attribute name="aTomcat" default=""/>
<attribute name="aState" default="1"/>
<sequential>
<waitfor maxwait="#{aMaxWait}" maxwaitunit="second" timeoutproperty="#{aHost}.#{aTomcat}.#{aModule}.#{aPort}.server.timeout" >
<and>
<jmx:equals
host="#{aHost}"
port="#{aPort}"
ref="#{aHost}.#{aTomcat}.#{aModule}.#{aPort}"
name="Catalina:j2eeType=WebModule,name=//localhost/#{aModule},J2EEApplication=none,J2EEServer=none"
attribute="state"
value="#{aState}"
/>
</and>
</waitfor>
<if>
<equals arg1="${#{aHost}.#{aTomcat}.#{aModule}.#{aPort}.server.timeout}" arg2="true" />
<then>
<var name="failBuild" value="true"/>
<echo message="*************************Host.Tomcat.Module = #{aHost}.#{aTomcat}.#{aModule} is not deployed into the tomcat" />
</then>
<else>
<echo message="#{aHost}.#{aModule} is deployed into the tomcat" />
</else>
</if>
</sequential>