A Linux bash script to recognize when a file is added to FTP

1. The Challenge

You have uploaded a compressed file to an FTP server containing a published web application, perhaps via Continuous Integration. You want your Linux to automatically detect it, so you can run the commands to deploy it.

Note:

All examples included in this article have been tested on Ubuntu 18.04.2 LTS.

I use the Linux command console

My user has sudo rights.

2. The approach

This approach works if you have access to your FTP server’s log. We’re going to monitor it using the tail command (more on that later) to watch for successful uploads.

I run vsftpd as my FTP server whose logs include lines like this:

Sat Sep 21 13:48:07 2019 [pid 6645] CONNECT: Client "::ffff:37.196.82.158"
Sat Sep 21 13:48:07 2019 [pid 6644] [UserName] OK LOGIN: Client "::ffff:37.196.82.158"
Sat Sep 21 13:48:10 2019 [pid 6647] [UserName] OK UPLOAD: Client "::ffff:37.196.82.158", "/files/MySite.zip", 28398250 bytes, 12906.89Kbyte/sec

The UPLOAD line only gets added when vsftpd has successfully saved the file and that’s what we need to detect.

This is a simpler, less taxing solution than monitoring the ftp directory and notifying us of changes with the inotify API. The genius (if I may humbly say so!) of this solution is that this sort of portable event-triggered approach has less load and is more reliable and easier to debug than a polling approach with traditional cron.

3.Create the Bash script

1. Use cd to change to the directory where you want to save the new shell file.

I’ve chosen to save mine in the main user directory. You can navigate there with the tilde shortcut ~.

$ cd ~ #this takes us to /home/UserName/

This is also the default directory terminal always opens in. You can type pwd to confirm your location.

I keep my scripts in a folder named bin which you make using the mkdir command.

$ mkdir bin #this creates /home/UserName/bin

2. Add the code

Whilst in /home/joe, cd to your new directory.

$ cd bin

We are going to use the inbuilt nano editor to create the shell script.

Type nano in the terminal to open the nano editor.

$ nano

The command will open nano as follows:

Add the following code.

#!/bin/bash

sudo tail -F /var/log/vsftpd.log | while read line; do
  if echo "$line" | grep -q 'OK UPLOAD:'; then
     filename=$(echo "$line" | cut -d, -f2)
     if [[ "$filename" == *"CreativelyCode.zip"* ]]; then
     echo "here"
     # Stop service
     sudo service CreativelyCode stop
     # Make backup
     sudo zip -r /var/backups/site/$(date +%F)_CreativelyCode /var/www/CreativelyCode/publish
     # -o to overwrite
     sudo unzip -o /home/joe/ftp/files/CreativelyCode.zip -d /var/www/CreativelyCode
     # Remove uploaded zip
     sudo rm /home/joe/ftp/files/CreativelyCode.zip
     #Restart service
     sudo service CreativelyCode start
     fi
  fi
done

Press Ctrl-X on your keyboard to exit nano.

nano will ask you if you want to save the modified file. Hit the y key (for "yes"), choose your file name with the .sh extension and then press enter to confirm. I’ve called mine deploy_site_from_ftp.sh

Make the script executable with command chmod +x <filename.sh>.

Let’s analyse this code step by step.

I. Watch the vsftp log for changes with tail

Syntax: tail [OPTION]... [FILE]...

tail is a command-line utility for printing the last part of one or more files.

By default, tail prints the last 10 lines of a file.

The -f option which stands for “follow” will keep the log open and watch as new lines are written to it in real time. This is one of the most common uses of the tail command.

This option will cause tail to loop until broken by the user. When new data appears, it will be printed.

Syntax: grep [OPTION...] PATTERNS [FILE...]

We read the tailed output line by line and combine it with grep to check whether any line contains 'OK UPLOAD:'. This tells us that a new file has been uploaded.

grep grep is an acronym that stands for Global Regular Expression Print. grep is used to search text in a specified file for lines containing a match to the given string or regular expression. When it finds a match, it prints the line with the result. Combining grep commands can be powerful when searching for text through massive log files or even thousands of files.

III. Use cut to retrieve the file name

Once we have a line that matches, we will use the cut command to retrieve just the section of text containing the file name.

Notice that the vsftp log output uses a comma to separate its information, we refer to this as it’s delimiter.

To cut using a delimiter, use the -d option followed by the given delimiter, -d, .

As we have three commas, the line from our log will be split into three parts or fields. We use the -f option to specify which part of the text we want to extract. We use -f2 to specify the second of these, "/path/to/file.zip".

What is a delimiter?

A delimiter is a sequence of one or more characters for specifying the boundary between separate, independent regions in plain text or other data streams. For example, in the case of the vsftpd log, information is separated by a comma.

IV. Add your script

What you want your script to do once a new files has been uploaded is down to you.

My script shuts down the service running my website, makes a backup, overwrites the existing files and then restarts the service.

3. Running as a systemd service

We want our new script to run at all times and be restarted in case of a failure or unexpected exit such as a server restart.

So, what we’ll do is use systemd to run our shell script as a service.

What is systemd?

systemd is a system and service manager responsible for controlling what programs run when a Linux system boots up. These days it is considered the default init system for most distributions (although not without controversy!). Services are background processes and in systemd lingo are referred to as Daemons.

To do this:

I. Make a new file in /etc/systemd/user/

Again using nano (this time with sudo) (or the editor of your choice).

II. Add the following code to make a very basic daemon.
[Unit]
Description=Watches vsftp log for successful uploads,  \
closes CreativelyCode app, replaces files and restarts app.

[Service]
ExecStart=/home/UserName/bin/deploy_site_from_ftp.sh
Restart=on-failure

[Install]
WantedBy=default.target

You’ll need to:

• Set the proper path to your script in ExecStart

Restart=always

By default, if the program exits for whatever reason, systemd will not restart your service. Usually this is not what you want for a service that must always be accessible, so we instruct systemd to restart our service whenever there’s an unexpected exit.

III. Launch the service (and other handy system commands)

Firstly, enable the service:

$ systemctl enable /etc/systemd/user/deploy_site_from_ftp.service

You will be prompted for your administrator’s password.

Start the service: sudo systemctl stop deploy_site_from_ftp.service

To stop the service: sudo systemctl start deploy_site_from_ftp.service

To restart the service: sudo systemctl restart deploy_site_from_ftp.service

To check the status of the service: sudo systemctl status deploy_site_from_ftp.service

4. Conclusion

My machine now knows whenever a new file has been uploaded to my FTP server. For me, that means not having to worry about fiddling with anything every time my continuous integration pipeline has deployed my updated code. Hope it helps!