rsync copying tool
Rsync is a fast and powerful file copying tool. Rysnc can copy locally, to and from another host over any remote shell, or to and from a remote rsync daemon. Rsync has a vast set of options that allow you to control virtually every aspect of its behaviour and the way files are copied. Rsync is famous for its "delta-transfer" algorithm which reduces the amount of data transferred over a network by sending only the differences between the source files and the existing files in the destination. This makes rsync extremely useful as a backup tool.
The rsync tool locates files that need to be transferred using a "quick check" algorithm that looks for any files that have changed in size or changes in the last-modified time.
Rsync is generally run from a command line or script which makes it ideally suited for making backups. Many tools are also available with a gui (Graphical User Interface) that allow you to configure backups quickly. An example of a GUI rsync tool is "Grsync". Grsync is basically a front end to the "rsync" command.
Below are examples of some basic uses of the "rsync" command for copying files:
Retrieving files from a remote host with rsync
The following rsync command retrieves files from a remote location to a specified directory. In the following example we are going to use our local machine "Ubuntu" and our remote machine "suse":
"Ubuntu" currently has no files in the directory below:
john@john-desktop:~/ubuntu$ ls -l
total 0
"suse" has the following files in its directory:
john@linux-pd5y:~/suse> pwd
/home/john/suse
john@linux-pd5y:~/suse> ls -l
total 40
-rw-r--r-- 1 john users 2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john users 5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john users 8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john users 14330 Apr 25 09:00 file4.txt
On our local machine "Ubuntu" we are going to issue the command:
rsync -v -e ssh root@192.168.0.11:/home/john/suse/* .
This command will copy the files from the remote "suse" machine to our local directory:
john@john-desktop:~/ubuntu$ rsync -v -e ssh root@192.168.0.11:/home/john/suse/* .
Password:
file1.txt
file2.txt
file3.txt
file4.txt
sent 87 bytes received 31778 bytes 4902.31 bytes/sec
total size is 31526 speedup is 0.99
First we will be asked to supply a password for the remote machine. Once authenticated, our files should be copied to our local machine in our current directory. If we now issue the "ls -l" command on our local machine we should see that the files have been transferred successfully.
john@john-desktop:~/ubuntu$ ls -l
total 40
-rw-r--r-- 1 john john 2866 Apr 25 09:09 file1.txt
-rw-r--r-- 1 john john 5732 Apr 25 09:09 file2.txt
-rw-r--r-- 1 john john 8598 Apr 25 09:09 file3.txt
-rw-r--r-- 1 john john 14330 Apr 25 09:09 file4.txt
Notice, the files that have been copied have a different time stamp to the original files on the remote "suse" machine.
If we wanted to preserve this file stamp, then we can modify our command to include the "-t" option to preserve our time stamp information:
rsync -vt -e ssh root@192.168.0.11:/home/john/suse/* .
john@john-desktop:~/ubuntu$ rsync -vt -e ssh root@192.168.0.11:/home/john/suse/* .
Password:
file1.txt
file2.txt
file3.txt
file4.txt
sent 375 bytes received 428 bytes 146.00 bytes/sec
total size is 31526 speedup is 39.26
john@john-desktop:~/ubuntu$ ls -l
total 40
-rw-r--r-- 1 john john 2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john john 5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john john 8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john john 14330 Apr 25 09:00 file4.txt
This time the "time stamp" information has been preserved and should match that of the original file from the remote "suse" machine.
Copying a local file to a remote server with rsync
In the next example we are going to copy a local file to our remote "suse" machine.
For this example, we have added a new file called ubuntu_file.txt into our current directory:
john@john-desktop:~/ubuntu$ ls -l
total 72
-rw-r--r-- 1 john john 2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john john 5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john john 8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john john 14330 Apr 25 09:00 file4.txt
-rw-rw-r-- 1 john john 31526 Apr 25 09:18 ubuntu_file.txt
Now on our local machine (Ubuntu) we issue the following command:
rsync -avz /home/john/ubuntu/ john@192.168.0.11:/home/john/suse/
john@john-desktop:~/ubuntu$ rsync -avz /home/john/ubuntu/ john@192.168.0.11:/home/john/suse/
Password:
sending incremental file list
./
ubuntu_file.txt
sent 1066 bytes received 46 bytes 202.18 bytes/sec
total size is 63052 speedup is 56.70
Now if we take a look at the remote "suse" machine:
john@linux-pd5y:~/suse> ls -l
total 72
-rw-r--r-- 1 john 1000 2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john 1000 5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john 1000 8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john 1000 14330 Apr 25 09:00 file4.txt
-rw-rw-r-- 1 john 1000 31526 Apr 25 09:18 ubuntu_file.txt
We can see that only the new file has been transferred.
rsync with ssh
In all of the previous examples so far we have had to supply our credentials (password) each time we connect to the remote "suse" machine. It is possible to create what is known as a public key which can then be copied to the remote server. This will then allow us to login without supplying a password. To achieve this we must carry out the following actions:
Define a ssh key on our local machine (Ubuntu):
This is achieved by issuing the "ssh-keygen" command. For more details on this visit our ssh page.
In the example below we did not provide any "passphrase", we simply pressed "enter".
john@john-desktop:~/.ssh$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/john/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/john/.ssh/id_rsa.
Your public key has been saved in /home/john/.ssh/id_rsa.pub.
The key fingerprint is:
d7:f5:6e:20:9b:fe:29:51:f3:d4:2f:bb:6c:c5:9a:31 john@john-desktop
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| . .|
| . .o.o|
| S . o..=o|
| . .+Eo=|
| o. Oo|
| ...=o |
| .+=. |
+-----------------+
From the above, we can see that we have generated what is known as a "public key" called "id_rsa.pub" in the following location: /home/john/.ssh
Next we need to copy the contents of our key "id_rsa.pub" to the remote machine.
copying a public key to a remote location
We then copy the contents of our public key from our local server "/home/john/.ssh/id_rsa.pub" to the "authorized_keys" file on the "remote "suse" machine. To achieve this we can simply issue the following command. You will be prompted for your password to authorise this activity on the remote server.
cat $HOME/.ssh/id_rsa.pub | ssh 192.168.0.11 'cat >> .ssh/authorized_keys && echo "Contents Copied Successfully"'
john@john-desktop:~/ubuntu$ cat $HOME/.ssh/id_rsa.pub | ssh 192.168.0.11 'cat >> .ssh/authorized_keys && echo "Contents Copied Successfully"'
Password:
Contents Copied Successfully
Now lets confirm that the file has been copied to the remote "suse" machine:
john@linux-pd5y:~/.ssh> ls -l
total 4
-rw-r--r-- 1 john users 399 Apr 25 09:34 authorized_keys
john@linux-pd5y:~/.ssh> cat authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpqMCtzPvRZD+pfnO8gHoJrtXF+JjOji/GXTK5FeioFHyZMzvT4CTXqAQIFRIkq1fIoZkMz8LYmVqBPP/vfS7SbTL81dR3Mpri6YzngjGka1ejH6+34PVgZT47eBi65kZddGWOpfpBYj/1x+7aNwvZTRwx6TLZcq0NQ5Ot8ZTJ/yTXJfRAaYKlLRN4cu6JZ4HYqVC0kBoVSGzeBdCmA1Yoa3ecWqGuAKXskb8sseRH2es6/7G+C3Wa205TUYf42DoUyD6X3BXhs0WvvsqXKrKOx+ZvUBgq0EsH58lL4tKu2aaKamg5mR8B6WUJZzlAkksaBukAW9sJqL6RxG4zJpNx john@john-desktop
Now on the remote machine we need to modify the permissions of the newly created "authorized_keys" file with the "chmod" command so that only the owner of the file can view its content:
john@linux-pd5y:~/.ssh> chmod 600 authorized_keys
john@linux-pd5y:~/.ssh> ls -l authorized_keys
-rw------- 1 john users 399 Apr 25 09:34 authorized_keys
Testing our ssh key
To test that our key is working correctly, we can issue the command: ssh 192.168.0.11 from the local "ubuntu" machine.
john@john-desktop:~/ubuntu$ ssh 192.168.0.11
Last login: Thu Apr 25 09:39:34 2013 from 192.168.0.14
Have a lot of fun...
Success we now do not have to supply a password each time we use the "rsync" command on our remote "suse" machine.
rsync with ssh example
For our new test I have created an additional file called "landoflinux.com":
john@john-desktop:~/ubuntu$ ls -rtl
total 76
-rw-r--r-- 1 john john 2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john john 5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john john 8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john john 14330 Apr 25 09:00 file4.txt
-rw-rw-r-- 1 john john 31526 Apr 25 09:18 ubuntu_file.txt
-rw-rw-r-- 1 john john 333 Apr 25 09:42 landoflinux.com
Now lets issue our "rsync" command: rsync -avz /home/john/ubuntu/ john@192.168.0.11:/home/john/suse/
john@john-desktop:~/ubuntu$ rsync -avz /home/john/ubuntu/ john@192.168.0.11:/home/john/suse/
sending incremental file list
./
landoflinux.com
sent 333 bytes received 34 bytes 244.67 bytes/sec
total size is 63385 speedup is 172.71
This time we didn't get asked for a password! The only file that was transferred was the "landoflinux.com" file. We can confirm this by looking at the remote "suse" server:
john@john-desktop:~/ubuntu$ ssh 192.168.0.11 ls -l /home/john/suse
total 76
-rw-r--r-- 1 john 1000 2866 Apr 25 08:59 file1.txt
-rw-r--r-- 1 john 1000 5732 Apr 25 08:59 file2.txt
-rw-r--r-- 1 john 1000 8598 Apr 25 08:59 file3.txt
-rw-r--r-- 1 john 1000 14330 Apr 25 09:00 file4.txt
-rw-rw-r-- 1 john users 333 Apr 25 09:42 landoflinux.com
-rw-rw-r-- 1 john 1000 31526 Apr 25 09:18 ubuntu_file.txt
Notice now, we can remotely execute commands on the remote server directly from our local machine! In the above examples, we have seen only a very small part of what "rsync" is capable of. To get a full overview of the "rsync" command, I would recommend reading the "man pages" for rsync. An over view of some of the options that can be used with the rsync command follow. To view this option you can issue the command "rsync --help" or "man rsync"
rsync options
Usage: rsync [OPTION]... SRC [SRC]... DEST
or rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
or rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
or rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
or rsync [OPTION]... [USER@]HOST:SRC [DEST]
or rsync [OPTION]... [USER@]HOST::SRC [DEST]
or rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]
The ':' usages connect via remote shell, while '::' & 'rsync://' usages connect
to an rsync daemon, and require SRC or DEST to start with a module name.
Options
-v, --verbose increase verbosity
-q, --quiet suppress non-error messages
--no-motd suppress daemon-mode MOTD (see manpage caveat)
-c, --checksum skip based on checksum, not mod-time & size
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
-r, --recursive recurse into directories
-R, --relative use relative path names
--no-implied-dirs don't send implied dirs with --relative
-b, --backup make backups (see --suffix & --backup-dir)
--backup-dir=DIR make backups into hierarchy based in DIR
--suffix=SUFFIX set backup suffix (default ~ w/o --backup-dir)
-u, --update skip files that are newer on the receiver
--inplace update destination files in-place (SEE MAN PAGE)
--append append data onto shorter files
--append-verify like --append, but with old data in file checksum
-d, --dirs transfer directories without recursing
-l, --links copy symlinks as symlinks
-L, --copy-links transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the source tree
-k, --copy-dirlinks transform symlink to a dir into referent dir
-K, --keep-dirlinks treat symlinked dir on receiver as dir
-H, --hard-links preserve hard links
-p, --perms preserve permissions
-E, --executability preserve the file's executability
--chmod=CHMOD affect file and/or directory permissions
-A, --acls preserve ACLs (implies --perms)
-X, --xattrs preserve extended attributes
-o, --owner preserve owner (super-user only)
-g, --group preserve group
--devices preserve device files (super-user only)
--specials preserve special files
-D same as --devices --specials
-t, --times preserve modification times
-O, --omit-dir-times omit directories from --times
--super receiver attempts super-user activities
--fake-super store/recover privileged attrs using xattrs
-S, --sparse handle sparse files efficiently
-n, --dry-run perform a trial run with no changes made
-W, --whole-file copy files whole (without delta-xfer algorithm)
-x, --one-file-system don't cross filesystem boundaries
-B, --block-size=SIZE force a fixed checksum block-size
-e, --rsh=COMMAND specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on the remote machine
--existing skip creating new files on receiver
--ignore-existing skip updating files that already exist on receiver
--remove-source-files sender removes synchronized files (non-dirs)
--del an alias for --delete-during
--delete delete extraneous files from destination dirs
--delete-before receiver deletes before transfer, not during
--delete-during receiver deletes during the transfer
--delete-delay find deletions during, delete after
--delete-after receiver deletes after transfer, not during
--delete-excluded also delete excluded files from destination dirs
--ignore-errors delete even if there are I/O errors
--force force deletion of directories even if not empty
--max-delete=NUM don't delete more than NUM files
--max-size=SIZE don't transfer any file larger than SIZE
--min-size=SIZE don't transfer any file smaller than SIZE
--partial keep partially transferred files
--partial-dir=DIR put a partially transferred file into DIR
--delay-updates put all updated files into place at transfer's end
-m, --prune-empty-dirs prune empty directory chains from the file-list
--numeric-ids don't map uid/gid values by user/group name
--timeout=SECONDS set I/O timeout in seconds
--contimeout=SECONDS set daemon connection timeout in seconds
-I, --ignore-times don't skip files that match in size and mod-time
--size-only skip files that match in size
--modify-window=NUM compare mod-times with reduced accuracy
-T, --temp-dir=DIR create temporary files in directory DIR
-y, --fuzzy find similar file for basis if no dest file
--compare-dest=DIR also compare destination files relative to DIR
--copy-dest=DIR ... and include copies of unchanged files
--link-dest=DIR hardlink to files in DIR when unchanged
-z, --compress compress file data during the transfer
--compress-level=NUM explicitly set compression level
--skip-compress=LIST skip compressing files with a suffix in LIST
-C, --cvs-exclude auto-ignore files the same way CVS does
-f, --filter=RULE add a file-filtering RULE
-F same as --filter='dir-merge /.rsync-filter'
repeated: --filter='- .rsync-filter'
--exclude=PATTERN exclude files matching PATTERN
--exclude-from=FILE read exclude patterns from FILE
--include=PATTERN don't exclude files matching PATTERN
--include-from=FILE read include patterns from FILE
--files-from=FILE read list of source-file names from FILE
-0, --from0 all *-from/filter files are delimited by 0s
-s, --protect-args no space-splitting; only wildcard special-chars
--address=ADDRESS bind address for outgoing socket to daemon
--port=PORT specify double-colon alternate port number
--sockopts=OPTIONS specify custom TCP options
--blocking-io use blocking I/O for the remote shell
--stats give some file-transfer stats
-8, --8-bit-output leave high-bit chars unescaped in output
-h, --human-readable output numbers in a human-readable format
--progress show progress during transfer
-P same as --partial --progress
-i, --itemize-changes output a change-summary for all updates
--out-format=FORMAT output updates using the specified FORMAT
--log-file=FILE log what we're doing to the specified FILE
--log-file-format=FMT log updates using the specified FMT
--password-file=FILE read daemon-access password from FILE
--list-only list the files instead of copying them
--bwlimit=KBPS limit I/O bandwidth; KBytes per second
--write-batch=FILE write a batched update to FILE
--only-write-batch=FILE like --write-batch but w/o updating destination
--read-batch=FILE read a batched update from FILE
--protocol=NUM force an older protocol version to be used
--iconv=CONVERT_SPEC request charset conversion of filenames
-4, --ipv4 prefer IPv4
-6, --ipv6 prefer IPv6
--version print version number
(-h) --help show this help (-h is --help only if used alone)
No comments:
Post a Comment