Copy workspace to server via zip file

Continua CI Feature Requests
1

I know that there is a possibility to copy and extract zip file to agent workspace or copy to server to zip file.
But I would find useful to be able to extract zip file (from agent) on server side.

Meaning I have a coverage report that generates tens of thousand files (zip ~100mb) and if I want to show the report the copy takes hours whereas copy of zip takes couple of seconds (extract takes few minutes but still faster than file by file copy).

What I mean is something like:
server_path/ < coverage.zip:
Which seem to not work right now.

2

Hi Michal,

The workspace rules are currently run by the agent, therefore they are not able to extract zip files on the server. This is however something that we could do when processing the artefact rules at the end of each stage. We’ll look into this further.

3

OK, thank you.

As a side note to this long workspace sync, it would be nice if there would be a progress indication (if possible).

4

Hi Michal,

It does look we are not including the workspace synchronisation in the estimated progress bar for each stage. We’ll look into the feasibility of including it. Note, if you enable the “Log workspace files file” on the Workspace Rules tab of the stage options, then each file will be listed in the log as it is copied. This may provide at least some indication of progress.

It’s interesting that you say that it takes hours to transfer the files. That seems to be long time - even for tens of thousands of files. Is the transfer from a remote agent? Can you check which transport type (UNC or SSH) is used to sync the workspace? You can find this in the build log under the Workspace Syncing node.

If it is using SSH to transfer files and the agent is on the same network as the server you should be able to speed thing up significantly by configuring agent access to the server data share. See Networking and Security - Continua CI - VSoft Technologies Documentation Wiki.

Note that we have now added archive extraction rules to the artifact rules in development and this will be in the next version.

5

It’s using UNC transport, but the problem is that it’s 650 mb in ~115 thousand files which I think is the worse case scenario transferring over network.
Both server and agent are on the same network, although on different machines and under different virtualizations. Nevertheless I think the problem is cause by so many small files, I don’t know of a protocol which handles such situations in a good way.

6

Hi Michal,

Archive extraction rules have now been added to the artifact rules and are available in version 1.9.2.832. See artifact rules in the documentation for the format.

7

Thank you, I will plan the update :slight_smile:

8

Finally had time to update.
For me the time improvement is 4h reduced to 2 minutes :smiley:

Thank you.

1 Like