FYI, this is a snippet of the log that I am trying to translate, running in Automise 4.0.0.467:
Downloaded /COWATCH02646/A10EXT.ZIP to \\dutch\\Downloads\\Jordans\COWATCH02646\A10EXT.ZIP in 6139 ms Downloaded /COWATCH02646/A11EXT.ZIP to \\dutch\\Downloads\\Jordans\COWATCH02646\A11EXT.ZIP in 76 ms Downloaded /COWATCH02646/A12EXT.ZIP to \\dutch\\Downloads\\Jordans\COWATCH02646\A12EXT.ZIP in 60 ms
The idea is to extract the destination full filename
Still not having any luck with the regex filtering. I found a tool to test Delphi regex’s but I can’t make much sense of it (usually use Expresso for .NET regexe’s which works really well). Can you try the Regex and see what you get? Thanks.
Things are a lot more stable now with this action, however the evaluation of differences is still very slow when interrogating a large number of files. Admittedly this is an extreme case as there are 732,868 files in 1,184 folders to process locally! I don’t know if it is possible, but making the sorting/evaluation code multi-threaded could give a big boost as the server this is running on has 12 cores.
Keep Alive Timeout: Sending NOOP Sorted local items in 120298 ms Keep Alive Timeout: Sending NOOP Keep Alive Timeout: Sending NOOP Keep Alive Timeout: Sending NOOP Keep Alive Timeout: Sending NOOP Sorted remote items in 8205 ms Evaluating Differences… Keep Alive Timeout: Sending NOOP
Keep Alive Timeout: Sending NOOP 9430 Differences Found Evaluated Differences in 5021087 ms Sorted Differences in 512 ms Downloading files…
Further to my comment below, one glitch I have noticed is that the FTP connection sometimes drops, causing the mirror action to retry. This leads to another 10 minutes of delay whilst it goes through all comparison steps again. Would it be possible to have an inner retry loop, so that it would get straight back to resuming whichever step had failed rather than repeating the whole process?
That certainly is a great number of files being reviewed, and then mirrored. I have logged both issues you raise here, the speed of the difference check, and an internal retry. I am certain we can get the difference check down to something a lot quicker, the internal retry will require a bit more work as its a structural change to how actions run (keeping state between retries).
I will keep you informed as to how things progress.
Thanks for such a prompt reply! Another thought - the NOOPs seem to happen quite often (10s?). Perhaps the interval could be adjustable, and/or an option to not log the NOOP action. That will make log files a lot smaller.
I have added the option for turning off the NOOPs and changing their interval as another case. It should be very simple to achieve a solution for this. In other news I have found a speed improvement for the differences test. It should show a substantial improvement in speed. I will also look at the sorting methods, I believe these could be quicker.
After these have been done I will make a release. Multi-threading the action will take some more work which will mean we head on into next week before its done.
Please let me know the timings your script gets with these builds. Currently in testing the timings for 1 million files are looking as such;
Sorting Local Files… Sorted local items in 46289 ms Sorting Remote Files… Sorted remote items in 9850 ms Evaluating Differences… 773216 Differences Found Evaluated Differences in 3494 ms Sorting Differences… Sorted Differences in 35817 ms
My differences are larger and this does affect the numbers a little, but this is significantly down from the 80+ minutes you were seeing. Please let me know how this goes.
Looks like a massive improvement in performance. Thank you!
Sorting Local Files… Sorted local items in 63317 ms Sorting Remote Files… Sorted remote items in 1110 ms Evaluating Differences… 3051 Differences Found Evaluated Differences in 2306 ms Sorting Differences… Sorted Differences in 69 ms Downloading files…
Been using the fixed FTPMirror for a while but I’m now seeing the below error message. Maybe there is a memory leak, or the number of files being requested is too large? I’m thinking the former as stopping the IDE causes it to crash (memory usage is 1.3Gb)
CRC is not supported by this FTP Server. Skipped Calculation of Server Time Skew Generating File List… FTP Mirror failed. Out of memory
We have found the issue which is causing the out of memory error. I have a solution which will keep the speed and reduce the memory overhead we currently have when listing the files. It will take a little time to implement.
In the meantime I suggest splitting your FTP mirror at a certain folder level. This isn’t ideal I know, however it should mean you won’t hit the memory issue. As a ballpark to aim for in this interim solution would be to split the mirror out into approximately 300-400k file blocks to be safe. If you can manage less that would be even better.
I will post a fix as soon as I have one. Working through implementing and testing now.
Please let me know how these go for you. I have tested them quite extensively here so not expecting anything untoward to occur.NOOPs do send too often at present, I am working on a way to reduce them. It doesn’t impact performance that substantially.
I’ve installed 615 but the FTPMirror actions seems to be broken. I managed to get the IDE to send an error report before it crashed. Have reverted to 585 for the moment until this is resolved.
The error message is: Local Directory [%LocalPath%] does not exist.
I have found and fixed the errors which were occurring. These were namely a stop build error and mirror download error with dates checked. The releases are located here;