How can I use Automise to download only the latest file from an FTP server? I don’t know what the file will be called, so I can’t use naming patterns.
If I assume that a new file is added to the FTP server every day, I suppose I could put yesterday’s date in a project variable, and then use that variable in the date/time filter… but what if there are multiple files a day, how can I tell it to only download the latest?
The way I typically achieve this is to list the directory contents with detail logging. This will typically give back the filename, and date created in a list. I then loop through this list and parse the date and filename from each line. From this I can then work out what is the latest file date and download that.
The “FTP List Directory” action will give the list of files on the FTP server. Typically in the format “filename, permissions, creation datetime”, but this does depend on how the server is setup. You will need to parse the output and split out the filename and datetime so that these can be tested against each other or sorted.
From that you will figure out what the latest file is and be able to download that.
Currently we don’t have any FTP functionality with can directly find this for you.
Ok, I need some guidance on how to parse the file listing information and find the file with the latest date.
This is an example of what the SFTP List Files action returns in my example:
-rwxrwxrwx 1 0 0 199008 Nov 21 16:10 MLT0209.EDI
-rwxrwxrwx 1 0 0 199004 Nov 22 16:50 MLT0212.EDI
-rwxrwxrwx 1 0 0 199184 Nov 25 16:06 MLT0214.EDI
So if I use a List Iterator or File Content Iterator (depending on how I store the information I pull from the FTP), how do I configure the next actions?
I’m guessing first step is Get/Find Text, and I have to use regular expressions to get the date starting at position 40 and ending at the end of the line, so I have date and file name? And I would write that to a text file, and keep appending each new line to it?
Then I would have to sort my new list of strings from the text file I created, that look like “Nov 21 16:10 MLT0209.EDI”? How can I sort that by date, I only see the Sort Text action which only does alphabetically?
Hi Katie,
I have attached a example project of how I would do this. Its not perfect, and if I had more time I would have used from JavaScript to massage the data. With this said the examples shows how output monitors can be used to gather output from one process and then massage it into what is required.
We are looking at introducing filesets of other actions which retrieve a list of files. This would make the process your trying to achieve much simpler.
FTP download only latest file.zip
Thank you so much for the example, that helps a lot.
This seems like something that should be easy… but once the list is sorted, how do you pull out the first item in the list? In this case, the list is inside a variable, but question also applies to filesets or text files as well.
Would you use an iterator? If so, how do you tell the iterator to stop after 1 iteration?
Hi Katie,
For a FileSet there is the following scripting methods available;
- Count - returns the amount (integer) of files in the FileList
- IsEmpty - returns true or false (boolean) depending if any files are in the FileSet
- File - returns the filename (string) specified by the index. The index specified is zero based (ie. first file in fileset is index 0) and must be less than the total amount of files in the FileSet.
- AllFiles - returns all the files (string) in the FileSet
Note that File requires the index to be specified directly after the word File, e.g. MyList.File5.
For getting the first item out of a list place the list into another list iterator, say %MyList%. Set the Variable for the list iterator to where you would like the first item stored, say %MyFirstItem%. Lastly use an Exit Loop action to leave the loop on the first iteration. Once the loop has completed the first item should be in the variable %MyFirstItem%.
So apparently fileset iterator puts the name (or maybe path and the name if you check that box) into the iterator variable, as opposed to a pointer to the file?
Also, apparently the iterator value goes to null once it exits the iterator… so if you want to do something with the filename you found inside the iterator, I guess you have to store it to a separate variable first before you leave the iterator?
Is there anyway I can set a fileset to that file? For example, if the iterator variable (%LatestFile%) is equal to C:\textFile.txt… within the iterator loop, can I define a new fileset that includes C:%LatestFile%? That doesn’t seem to work, I get an error.
Attachment unavailable
Oh man, now I’m having trouble uploading my project zip file, is there a trick to that?
Trying again and clicking Insert Image.
Attachment unavailable
I noticed there are actions FTP Send Command and FTPS Send Command.
I’m using SFTP, and I don’t see an action for that… but even if there was one, would that help at all? Could we use that to sort the files on the FTP server, and then download the latest?
Hi Katie,
“So apparently fileset iterator puts the name (or maybe path and the name if you check that box) into the iterator variable, as opposed to a pointer to the file?”
- This is correct, its just the filename (or directory and filename) of the current file in the FileSet.
“Also, apparently the iterator value goes to null once it exits the iterator… so if you want to do something with the filename you found inside the iterator, I guess you have to store it to a separate variable first before you leave the iterator?”
- The last iteration of the FileSet does set the iterator variable to a blank string. If the value is required outside of the iterator is best to store the value into another variable.
“Is there anyway I can set a fileset to that file? For example, if the iterator variable (%LatestFile%) is equal to C:\textFile.txt… within the iterator loop, can I define a new fileset that includes C:%LatestFile%? That doesn’t seem to work, I get an error.”"
- Another fileset could be defined with the filename as the mask. This shouldn’t be done with in an iterator however as a new FileSet will be defined for each loop which is an error as only one FileSet of that name can exist.
I suggest saving the value of the file you require to another variable name with in the iterator, then using this filename variable to perform and actions on the file you require. Storing it inside a FileSet isn’t required.
Hi Katie,
“Oh man, now I’m having trouble uploading my project zip file, is there a trick to that?”
- You will most likely need to add a link to the post to have it appear. We are seeing issues with the forums with regards to attachments at present. Something is looking into it.
Hi Katie,
“I noticed there are actions FTP Send Command and FTPS Send Command.
I’m using SFTP, and I don’t see an action for that… but even if there was one, would that help at all? Could we use that to sort the files on the FTP server, and then download the latest?”
There is a set of SFTP actions. This offer the ability to list the contents of a folder on the FTP server using the “SFTP List Files” action (note the connection will need to be defined prior to this action in the script). The action allows the log of files to be stored in a variable. This variable could then sorted as I showed in the example project I attached to this thread previously.
Jason,
Thanks for your help, I hope it didn’t seem like I was asking the same question as before. I did get the SFTP File List action to work, where I parsed it within an iterator to find the latest file name.
It just seems crazy to do that many steps to get the latest file, I was looking for an easier solution. If I was using FTP (which I might be in the future), would it be possible to connect and then use the FTP Send Command action to sort the files by date on the FTP server? And then somehow use the FTP File Download to get just the top file (the latest file)? Or would that not work?
Hi Katie,
Yes there is some work to do in this case. The solution in your current script of getting the listing, sorting the list, and then extracting the most recent file is the best way I can think of achieving what your trying to at present.
The only other real option would be to write a custom action to retrieve the most recent file from a list. This would be done through action studio. Note this is still after getting the directory listing from the FTP Directory Listing action as you wouldn’t want to re-implement the FTP action set.
The main reason for the large amount of work here is that FTP doesn’t offer some of the features you require. As always we are happy to take suggestions on how actions or the IDE could be improved. Please feel free to send these to support or even make a post on the wish list.
We use coreftp here as our ftp client, and I figured out how to download the latest file using the command line, so that’s an option too.
Attachment unavailable