16 Common SCP Commands with Examples to Copy Files and Folders

SCP (Secure Copy) is an invaluable command line tool that forms an integral part of a system administrator‘s toolkit for securely transferring files between servers. In this comprehensive 4000+ word guide, we will start with an in-depth introduction to SCP and then cover the 16 most essential SCP commands through detailed examples.

Whether you are a Linux sysadmin automating file transfers or a cybersecurity analyst gathering forensic evidence, this guide aims to make you an SCP power user. Time to finally master this open-source gem that keeps the world‘s servers humming!

An In-Depth Introduction to Secure Copy Protocol

Let‘s first understand what SCP is, what problems it solves and how it differs from other file transfer protocols.

What is Secure Copy Protocol?

SCP or Secure Copy is a command line protocol that allows secure file transfers between two remote hosts or a local and remote host.

It uses the Secure Shell (SSH) protocol underneath to transfer data. Some key aspects of SCP:

• Encryption – All data is encrypted using strong ciphers like AES, Blowfish etc. This prevents any unauthorized access to the data in transit.

• Authentication – SCP relies on SSH for public key or password based authentication. Only authorized users can establish connections.

• Data Integrity – Cryptographic checksums ensure any tampering or corruption of data is detected.

• Ease of Use – The syntax is very similar to Linux cp making it easy to use for system administrators.

In summary, SCP offers a simple and secure way to transfer files remotely for automation tasks as well as interactive use. The underlying SSH protocol provides the encryption, integrity and authentication required for secure transfers.

A Short History of SCP

The origins of SCP trace back to the old Unix rcp (remote copy) command which allowed unencrypted copying of files between hosts.

As networks evolved from local to wide area networks, the need for securing file transfers became crucial. SSH was created in 1995 by Tatu Ylönen to allow secure remote logins.

SCP was developed as an addon to SSH in 1999 by Timo Rinne and Tatu Ylönen to provide encrypted file transfer capabilities. It was readily adopted since SSH was already popular and no new infrastructure was needed.

Over the last 20+ years, SCP has become a standard tool used by system administrators, developers, IT support engineers and security analysts for managing servers and transferring files securely.

How Does Secure Copy Differ from FTP?

FTP or File Transfer Protocol is another popular protocol for file transfers. However, there are some key differences between SCP and FTP:

• Encryption – SCP transfers files over an encrypted SSH tunnel while FTP has no encryption. Data sent over FTP can be intercepted.

• Authentication – SCP uses SSH keys or passwords to authenticate users. FTP relies on plaintext usernames and passwords which can be compromised.

• Usage – SCP is command line only whereas FTP can be used from a command line as well as GUI FTP clients.

• Traffic – SCP encrypts all traffic including directory listings. FTP leaks directory structure info in plaintext.

• Security – SCP is more secure as it was designed specifically for encrypted transfers. FTP has multiple vulnerabilities if not used over SSL.

So in summary, SCP is more suited for automation tasks and secure remote file operations. FTP is better for interactive file management from a GUI tool.

SCP vs SFTP

SFTP (SSH File Transfer Protocol) also uses SSH similar to SCP. However there are some differences:

• Scope – SCP only allows file transfers. SFTP allows interactive session with features like listing directories, creating folders etc.

• Integration – SCP is integrated with SSH as a single package. SFTP is a separate protocol that uses SSH for security.

• Usage – SCP is used primarily for automated scripted file transfers. SFTP allows live interaction similar to FTP.

• Performance – SCP has lower overhead compared to SFTP and can be faster for automated bulk file copying.

So in essence, SCP is best for automated batch file operations. SFTP is better if you need an interactive remote file explorer.

Common Use Cases of SCP

Here are some popular use cases where SCP shines:

• Securely copying files between servers e.g. push configurations or upload server backups.
• Automating file distribution from a central server to multiple remote servers.
• Fetching log files or datasets from remote servers to a central location.
• Distributing software packages or patches to multiple systems.
• Transferring files for forensic analysis in information security or law enforcement.
• Implementing poor man‘s file synchronization between servers.

SCP forms an essential toolkit for system administrators, IT support professionals, developers, analysts and security researchers for managing file transfers at scale.

How Secure Copy Protocol Works

Under the hood, SCP simply relies on the SSH protocol for transferring data securely:

1. The SCP client initializes a file copy request for a source file to a destination server.

2. The client establishes an SSH connection with the server for authentication using keys or passwords.

3. An SCP process is spawned server side once authentication is successful.

4. The source file is then copied over this SSH encrypted channel to the destination directory.

5. The SSH connection terminates once the transfer is complete.

The same sequence of steps applies for transfers from server to client as well by simply switching the source and destination.

Now that we have a solid understanding of SCP, let‘s move on to the commands.

16 Essential SCP Commands and Examples

The basic syntax for SCP is:

scp [options] source destination

Where source and destination specify the local or remote file paths. There are also several options available to customize the transfer process.

Let‘s look at some common SCP command examples.

1. Copy Local File to Remote Server

To copy a file from the local system to a remote host:

scp local_file username@remotehost:/remote/folder

For example:

scp /home/john/data.txt [email protected]:/home/john/

This will copy data.txt from the local user‘s home folder to the remote user‘s home folder on server 192.168.1.10.

The key steps are:

1. Run the scp command with source as the local file path
2. Specify the destination as username@remotehost:/path/to/remote/folder
3. The remote username and IP address/hostname identifies the destination server
4. SCP will securely copy the file over SSH to the specified folder

This allows securely automating copying local files to remote servers, which is very useful for pushing out configurations, scripts or software updates.

2. Copy File from Remote Server to Local Machine

To transfer a file from a remote host to your local system:

scp username@remotehost:/remote/file /local/folder

For example:

scp [email protected]:/home/john/data.txt ~/downloads

The sequence of steps is:

1. Specify source as username@remotehost:/path/to/remote/file
2. Destination is your local folder path
3. SCP copies the file over SSH from remote to local machine

This allows securely fetching files from remote servers to your local system. Often used for retrieving logs, datasets or backups from remote hosts.

3. Recursively Copy Entire Directory from Local to Remote

To recursively copy a full directory instead of a single file:

scp -r local_directory username@remotehost:/remote/folder

For example:

scp -r /home/john/data [email protected]:/backup

This will recursively copy the entire /home/john/data directory to the /backup folder on the remote host.

The -r recursive flag ensures inner folders and files are copied as well in a nested manner. Very useful for replicating directory structures.

4. Copy Files Matching a Pattern

Instead of naming specific files, you can use wildcards to match multiple files.

For example, to copy text files:

scp *.txt username@remotehost:/remote/folder/

Or to copy all PDFs:

scp *.pdf username@remotehost:/remote/folder/

Some examples of common patterns:

• *.txt – All text files
• *.sh – All shell scripts
• docs.* – Filenames starting with docs
• *.!(png|jpg) – Exclude PNGs and JPEGs

This allows copying batches of files matching specific criteria.

5. Preserve File Attributes and Metadata

By default SCP does not maintain original file permissions, ownership and timestamp info.

To preserve the original metadata, use the -p flag:

scp -p file.txt username@remotehost:/path/to/destination

The -p option tells SCP to retain the original uid/gid, permissions and timestamp values of the source files.

Useful in scenarios where file attributes matter like replicating system files or forensic evidence collection.

6. Specify a Custom SSH Port

The default SSH port used by SCP is 22. You can customize this with the -P option:

scp -P 2222 file.txt username@remotehost:/path/to/destination  

This will connect to SSH on port 2222 instead of the standard port 22 for copying the file over the network.

7. Limit Bandwidth Usage

SCP allows restricting the bandwidth used for transfers using the -l option:

scp -l 1000 file.txt username@remotehost:/path/to/destination

The above command limits the throughput to 1000 Kbits/sec. This prevents SCP from saturating the network.

Useful when transferring large files that can choke bandwidth on slower networks or across high latency WAN links.

8. Enable Compression

To compress the data stream and improve transfer speeds over slower links, use the -C flag:

scp -C largefile.tar.gz username@remotehost:/path/to/destination

Compression reduces the filesize over the wire improving transfer speeds. Especially useful over the internet or congested networks.

9. View Verbose Logs

To see verbose logs showing the underlying SSH negotiation and diagnostics:

scp -v file.txt username@remotehost:/path/to/destination

This outputs detailed SCP execution logs. Useful for troubleshooting connection or transfer issues.

10. Specify an Alternate SSH Private Key

If you want SCP to use a specific SSH private key for authentication instead of the default id_rsa or id_dsa:

scp -i ~/.ssh/custom_key file.txt username@remotehost:/path/to/destination

This uses the SSH key at ~/.ssh/custom_key for logging into the destination server.

11. Copy Multiple Files in One Go

Instead of running SCP multiple times, you can transfer multiple files in a single SCP command:

scp file1.txt file2.txt username@remotehost:/remote/folder

This will copy over both file1.txt and file2.txt sequentially to the remote host in one go.

12. Rename Files During Copy

SCP allows renaming files during a transfer using a syntax like:

scp username@remote:/path/to/remotefile.txt /local/newname.txt

This copies remotefile.txt from the remote server and names it newname.txt locally.

13. Copy Files Between Remote Servers

SCP can directly transfer files between two remote servers without passing through your local machine:

scp username1@remotehost1:/path/to/file username2@remotehost2:/remote/folder  

The above command will securely copy files from remotehost1 to remotehost2 without routing through your machine acting as client.

14. Recursively Copy Files and Directories

To recursively copy entire folders including sub-directories while preserving metadata:

scp -rLP source username@remotehost:/remote/folder

The -r option enables recursive copy. -L follows symbolic links. And -P preserves file attributes.

15. Copy over Multiple Hops

SCP transfers can be chained over multiple hops if direct connectivity between source and destination is not available:

scp -o ProxyCommand=‘ssh server1 nc server2 22‘ /local/file username@server2:/remote/folder

This will tunnel SCP through server1 to reach server2 by combining SSH port forwarding with netcat.

16. Mount Remote Folders as Local Directory

Instead of explicitly copying files, you can directly mount a remote folder locally as a drive using SSHFS:

sshfs name@server:/remote/folder /local/mount_point

Anything you copy to /local/mount_point will get copied to the remote folder automatically over SSH.

This allows working with remote files from your local system directly without the overhead of SCP transfers.

SCP Implementation Internals

Under the hood, SCP is implemented using the SSH program itself. The scp command simply invokes ssh with the right options to securely copy files over an encrypted tunnel.

The key steps are:

1. Client initiates an SSH session to the destination server.

2. The ssh client requests the server to start an SCP process (scp -t)

3. The stdin and stdout of the scp -t process is connected to the SSH session.

4. The local SCP client sends file data to the scp -t stdin.

5. The scp -t process writes the received data to the destination file.

6. The scp -t process sends status info back to client via stdout of SSH session.

7. Once the file transfer finishes, the SSH session closes.

So in essence, the SSH connection provides the secure tunnel for piping the file data and metadata between the SCP client and server.

SCP vs Rsync

While SCP is handy for ad hoc file transfers, rsync is better suited for frequent large syncs:

SCP

• Simple to use for ad hoc file copies
• Transfers whole file each time
• Minimal overhead good for small transfers

Rsync

• Optimized for frequent bulk transfers
• Only transfers diffs, not full files
• More overhead but better for large syncs

So if you need to regularly synchronize large directories across servers, rsync is a better choice since it only transfers incremental file changes.

SCP Best Practices

Here are some tips for using SCP securely and effectively:

• Use SSH keys – Avoid password-based auth which can be brute forced. Use SSH keys instead for authentication.

• Limit access – Only allow SCP access from trusted client IPs if possible.

• Audit logs – Log SCP transactions to keep track of transfers. Monitor for anomalies.

• Prefer SFTP – For interactive transfers, use SFTP instead of SCP for more controls.

• Version control – Maintain file version history in case bad files are transferred by mistake.

• Failsafe encryption – Use client-side encryption as a secondary layer before transferring sensitive files.

• Tune performance – Enable compression and limit bandwidth on high latency networks.

• Automate transfers – Script recurring large transfers instead of manual SCP.

By following these best practices, you can securely get the most out of SCP for your infrastructure management needs.

Conclusion

I hope this detailed guide provides you a firm grasp of SCP along with the knowledge of the most common SCP commands and patterns required for securely automating and managing your file transfers.

Some key takeaways:

• SCP relies on the encryption, integrity and authentication provided by SSH for secure remote file copies.

• It is more suited for automated system administration tasks compared to interactive tools like FTP.

• You can tune SCP performance by enabling compression, limiting bandwidth etc.

• For advanced use cases, SCP allows chaining transfers via multiple hops.

• Use SFTP instead of SCP if you need more interactive remote file access.

• Automate recurring large transfers with rsync instead of SCP.

Feel free to reach out in the comments with any SCP use cases you want me to cover in the future! I hope this guide proves useful in effectively harnessing the simplicity and security of SCP.