A Linux bash script to recognize when a file is added to FTP
1. The Challenge
You have uploaded a compressed file to an FTP server containing a published web application, perhaps via Continuous Integration. You want your Linux to automatically detect it, so you can run the commands to deploy it.
All examples included in this article have been tested on Ubuntu 18.04.2 LTS.
I use the Linux command console
My user has sudo rights.
2. The approach
This approach works if you have access to your FTP server’s log. We’re going to monitor it using the tail command (more on that later) to watch for successful uploads.
vsftpd as my FTP server whose logs include lines like this:
Sat Sep 21 13:48:07 2019 [pid 6645] CONNECT: Client "::ffff:18.104.22.168" Sat Sep 21 13:48:07 2019 [pid 6644] [UserName] OK LOGIN: Client "::ffff:22.214.171.124" Sat Sep 21 13:48:10 2019 [pid 6647] [UserName] OK UPLOAD: Client "::ffff:126.96.36.199", "/files/MySite.zip", 28398250 bytes, 12906.89Kbyte/sec
UPLOAD line only gets added when
vsftpd has successfully saved the file and that’s what we need to detect.
This is a simpler, less taxing solution than monitoring the ftp directory and notifying us of changes with the
inotify API. The genius (if I may humbly say so!) of this solution is that this sort of portable event-triggered approach has less load and is more reliable and easier to debug than a polling approach with traditional cron.
3.Create the Bash script
1. Use cd to change to the directory where you want to save the new shell file.
I’ve chosen to save mine in the main user directory. You can navigate there with the tilde shortcut
$ cd ~ #this takes us to /home/UserName/
This is also the default directory terminal always opens in. You can type
pwd to confirm your location.
I keep my scripts in a folder named bin which you make using the
$ mkdir bin #this creates /home/UserName/bin
2. Add the code
cd to your new directory.
$ cd bin
We are going to use the inbuilt nano editor to create the shell script.
Type nano in the terminal to open the nano editor.
The command will open nano as follows:
Add the following code.
#!/bin/bash sudo tail -F /var/log/vsftpd.log | while read line; do if echo "$line" | grep -q 'OK UPLOAD:'; then filename=$(echo "$line" | cut -d, -f2) if [[ "$filename" == *"CreativelyCode.zip"* ]]; then echo "here" # Stop service sudo service CreativelyCode stop # Make backup sudo zip -r /var/backups/site/$(date +%F)_CreativelyCode /var/www/CreativelyCode/publish # -o to overwrite sudo unzip -o /home/joe/ftp/files/CreativelyCode.zip -d /var/www/CreativelyCode # Remove uploaded zip sudo rm /home/joe/ftp/files/CreativelyCode.zip #Restart service sudo service CreativelyCode start fi fi done
Press Ctrl-X on your keyboard to exit
nano will ask you if you want to save the modified file. Hit the y key (for "yes"), choose your file name with the .sh extension and then press
enter to confirm. I’ve called mine
Make the script executable with command
chmod +x <filename.sh>.
Let’s analyse this code step by step.
I. Watch the vsftp log for changes with tail
Syntax: tail [OPTION]... [FILE]...
tail is a command-line utility for printing the last part of one or more files.
tail prints the last 10 lines of a file.
-f option which stands for “follow” will keep the log open and watch as new lines are written to it in real time. This is one of the most common uses of the tail command.
This option will cause tail to loop until broken by the user. When new data appears, it will be printed.
II. Use grep to search
Syntax: grep [OPTION...] PATTERNS [FILE...]
We read the tailed output line by line and combine it with grep to check whether any line contains 'OK UPLOAD:'. This tells us that a new file has been uploaded.
grepis an acronym that stands for Global Regular Expression Print.
grepis used to search text in a specified file for lines containing a match to the given string or regular expression. When it finds a match, it prints the line with the result. Combining
grepcommands can be powerful when searching for text through massive log files or even thousands of files.
III. Use cut to retrieve the file name
Once we have a line that matches, we will use the cut command to retrieve just the section of text containing the file name.
Notice that the
vsftp log output uses a comma to separate its information, we refer to this as it’s delimiter.
To cut using a delimiter, use the
-d option followed by the given delimiter,
As we have three commas, the line from our log will be split into three parts or fields. We use the -f option to specify which part of the text we want to extract. We use
-f2 to specify the second of these,
What is a delimiter?
A delimiter is a sequence of one or more characters for specifying the boundary between separate, independent regions in plain text or other data streams. For example, in the case of the
vsftpdlog, information is separated by a comma.
IV. Add your script
What you want your script to do once a new files has been uploaded is down to you.
My script shuts down the service running my website, makes a backup, overwrites the existing files and then restarts the service.
3. Running as a systemd service
We want our new script to run at all times and be restarted in case of a failure or unexpected exit such as a server restart.
So, what we’ll do is use
systemd to run our shell script as a service.
systemdis a system and service manager responsible for controlling what programs run when a Linux system boots up. These days it is considered the default
initsystem for most distributions (although not without controversy!). Services are background processes and in
systemdlingo are referred to as Daemons.
To do this:
I. Make a new file in
nano (this time with sudo) (or the editor of your choice).
II. Add the following code to make a very basic daemon.
[Unit] Description=Watches vsftp log for successful uploads, \ closes CreativelyCode app, replaces files and restarts app. [Service] ExecStart=/home/UserName/bin/deploy_site_from_ftp.sh Restart=on-failure [Install] WantedBy=default.target
You’ll need to:
• Set the proper path to your script in ExecStart
By default, if the program exits for whatever reason,
systemdwill not restart your service. Usually this is not what you want for a service that must always be accessible, so we instruct
systemdto restart our service whenever there’s an unexpected exit.
III. Launch the service (and other handy system commands)
Firstly, enable the service:
$ systemctl enable /etc/systemd/user/deploy_site_from_ftp.service
You will be prompted for your administrator’s password.
Start the service:
sudo systemctl stop deploy_site_from_ftp.service
To stop the service:
sudo systemctl start deploy_site_from_ftp.service
To restart the service:
sudo systemctl restart deploy_site_from_ftp.service
To check the status of the service:
sudo systemctl status deploy_site_from_ftp.service
My machine now knows whenever a new file has been uploaded to my FTP server. For me, that means not having to worry about fiddling with anything every time my continuous integration pipeline has deployed my updated code. Hope it helps!