I needed to be able to automate the download of specifically named files from an FTP server and then delete them upon successful download. The need arose when the old script I was using was downloading some zero byte files (likely due to poor network conditions, workstation issues or remote server issues) then deleting the source file! D’oh!!
This script checks that the file is not a zero byte file before deleting from the remote server. Not great error/consistency checking, but better than I had before. It also copies the successfully downloaded files to an IMPORT directory and then archives the originally downloaded files to an ARCHIVE directory.
Hopefully you find it useful.
Note: I updated the post to include logging.
#!/bin/bash<br /> ################################################<br /> # Set some of the variables that will be needed<br /> # during the execution of this script<br /> ################################################<br /> # set current date<br /> now=`date +%Y-%m-%d-%H%M-%S`<br /> #<br /> # set the root dir<br /> rootdir="/ftp_files"<br /> #<br /> # set the download directory<br /> dldir="$rootdir/DOWNLOADS"<br /> #<br /> # set session download dir<br /> sessdldir="$dldir/$now"<br /> #<br /> # set the import dir<br /> importdir="$rootdir/IMPORT"<br /> #<br /> # set the archive dir<br /> archivedir="$rootdir/ARCHIVE"<br /> #<br /> # set up logging<br /> LOGFILE="$rootdir/LOGS/ftplog.txt"<br /> #<br /> ################################################<br /> # Lets get to work<br /> #################################################<br /> # create a directory based on todays date<br /> mkdir $sessdldir<br /> #<br /> # change to the new directory prior to FTP connection<br /> cd $sessdldir<br /> #<br /> if [ -f $LOGFILE ]<br /> then<br /> echo "****** FTP PROCESS STARTED AT: $now ******" &gt;&gt; $LOGFILE<br /> else<br /> echo "****** NEW LOG FILE CREATED AT: $now ******" &gt; $LOGFILE<br /> echo "****** FTP PROCESS STARTING AT: $now ******" &gt;&gt; $LOGFILE<br /> fi</p> <p>## FTP Connection parameters<br /> ftpserver=server.com<br /> username=youruser<br /> password=yourpass<br /> remotedir=/directoryname</p> <p>## Connect to FTP server and download all TXT files<br /> ftp -inv $ftpserver &gt;&gt; $LOGFILE &lt;&lt;MYEND<br /> user $username $password<br /> cd $remotedir<br /> bin<br /> mget *.TXT<br /> bye<br /> MYEND</p> <p>## Test for existence of files in the session download dir<br /> ## (we cd’d here earlier)<br /> if [ `ls | wc -l` -gt 0 ]<br /> then<br /> echo "" &gt;&gt; $LOGFILE<br /> echo "&gt;&gt; Files have been downloaded" &gt;&gt; $LOGFILE<br /> dlfiles=`ls`<br /> echo "&gt;&gt;&gt;&gt; Downloaded files:" &gt;&gt; $LOGFILE<br /> echo "$dlfiles" &gt;&gt; $LOGFILE<br /> echo "" &gt;&gt; $LOGFILE</p> <p>## Create the FTP delete script in order to remove the downloaded<br /> ## files from the FTP server. We will be sure to check for empty<br /> ## files below. The FTP delete script will be created in the session<br /> ## download directory and named ftp.$PID<br /> echo "open $ftpserver<br /> user $username $password<br /> binary<br /> cd $remotedir" &gt; $sessdldir/ftp.$$<br /> echo "&gt;&gt; Creating delete script to delete non-zero files from remote server" &gt;&gt; $LOGFILE</p> <p>## Iterate through the downloaded files one by one<br /> for i in $( ls *.TXT ); do<br /> ## If the file exists and is larger than 0 bytes, THEN add<br /> ## a delete command to the FTP delete script. This will allow<br /> ## us to delete all non-zero byte files from the FTP server.<br /> ## Non-zero byte files could be a sign that the file didn’t<br /> ## download properly due to an FTP timeout or other network<br /> ## issues. By not deleting the non-zero byte file we can attempt<br /> ## to download it the next time the script is run.<br /> if [ -s $i ]<br /> then<br /> echo "del $i" &gt;&gt; $sessdldir/ftp.$$<br /> echo "&gt;&gt;&gt;&gt; $i will be added to delete script" &gt;&gt; $LOGFILE<br /> fi<br /> done</p> <p>## Finish off the FTP delete script<br /> echo "quit" &gt;&gt; $sessdldir/ftp.$$</p> <p>echo "" &gt;&gt; $LOGFILE<br /> echo "&gt;&gt; Here is the delete script"&gt;&gt; $LOGFILE<br /> echo "" &gt;&gt; $LOGFILE<br /> cat $sessdldir/ftp.$$ &gt;&gt; $LOGFILE<br /> echo "" &gt;&gt; $LOGFILE<br /> echo "&gt;&gt; Running the delete script" &gt;&gt; $LOGFILE<br /> echo "" &gt;&gt; $LOGFILE</p> <p>## Run the FTP script<br /> ftp -ivn &lt; $sessdldir/ftp.$$ &gt;&gt; $LOGFILE</p> <p>## Copy all the non-zero byte TXT files to the import folder<br /> echo "" &gt;&gt; $LOGFILE<br /> echo "&gt;&gt; Copying non-zero size files to IMPORT" &gt;&gt; $LOGFILE</p> <p>for i in $( ls *.TXT ); do<br /> if [ -s $i ]<br /> then<br /> cp $i $importdir<br /> echo "&gt;&gt;&gt;&gt; Copying $i to IMPORT directory" &gt;&gt; $LOGFILE<br /> fi<br /> done<br /> fi</p> <p>## Copy session specific download dir to the ARCHIVE folder<br /> echo "" &gt;&gt; $LOGFILE<br /> echo "&gt;&gt; Copying entire $sessdldir to $archivedir" &gt;&gt; $LOGFILE</p> <p>cp -R $sessdldir $archivedir/</p> <p>## Delete the session specific download dir<br /> rm -rf $sessdldir<br /> echo "" &gt;&gt; $LOGFILE<br /> echo "&gt;&gt; Deleting $sessdldir" &gt;&gt; $LOGFILE<br /> echo "" &gt;&gt; $LOGFILE</p> <p>## ENDING LOG FILE FOR SESSION<br /> echo "****** FTP PROCESS ENDED AT: $now ******" &gt;&gt; $LOGFILE<br /> echo "******************************************************" &gt;&gt; $LOGFILE<br /> echo "" &gt;&gt; $LOGFILE
Please consider subscribing to GeoffManning.com to receive new posts in your RSS Reader or by Email.
No related posts.
