What is a Subprocess in Python? [5 Usage Examples]

Hey there! Have you ever wanted to take your Python coding skills to the next level by tapping into the full power of your operating system? Well, subprocesses are the key that unlocks that potential!

Subprocesses allow you to leverage all the capabilities of your OS directly from Python. You can execute external programs, automate workflows, run shell commands, build pipelines, and so much more!

In this comprehensive guide, I‘ll take you on an in-depth tour of subprocesses in Python. You‘ll learn:

What exactly is a subprocess and how does it work under the hood
The ins and outs of Python‘s subprocess module
How to run external programs and capture their output
Useful examples like shell commands, Git integration, task automation, and more
Best practices for working with subprocesses safely and portably

I‘ll also share my experiences using subprocesses for things like CI/CD pipelines, data science workflows, and systems administration.

So buckle up and get ready to become a subprocess pro!

What Exactly is a Subprocess?

Let‘s start from the basics – what is a subprocess?

A subprocess is a child process spawned by another parent process. The parent can communicate with the child through stdin, stdout, and stderr streams and obtain its return code.

In Python, the subprocess module enables us to create and interact with subprocesses directly.

subprocess allows your Python program to launch new processes to:

Execute external programs
Run shell commands
Pipe data between processes
Connect to stdin, stdout, stderr
Obtain return codes

Some key benefits of using subprocesses in Python:

Launch external programs and binaries
Pass input data to child processes
Get output and return codes back from processes
Build pipelines to combine programs
Leverage all OS functionality from Python
Automate complex multi-step workflows

So in a nutshell, subprocesses give you operating system superpowers within Python!

How Subprocesses Work Internally

Under the hood, creating a subprocess in Python involves using the fork() and exec() system calls:

fork() clones the current process into a new child process
exec() replaces the child process‘ code with the external program
The parent process can then communicate with child via pipes

Here is a simple diagram showing how it works:

                  +-----------------------+
                  |       Parent          |
                  |   Python Process      |
                  +-----------------------+
                         | fork()
                         v
+-----------------------+---------------+
|                    Child                |  
|             Python Process             |
+-----------------------+---------------+
                         | exec()
                         v
+-----------------------+---------------+
|                 External                |
|                Program                  |  
+-----------------------+---------------+

When you create a subprocess in Python, here is what happens under the covers:

The fork() system call clones the parent Python process into a child copy.
The child process then uses exec() to overwrite its code with the external program you want to run.
This gives the parent Python process a way to spawn new processes and interact with them.
The parent and child processes can inter-communicate via the stdin, stdout and stderr streams.
The parent process can wait on the child and obtain its return code.

So in summary, the subprocess module harnessing fork() and exec() to provide an API for spawning child processes from Python code.

Introduction to the subprocess Module

Python‘s subprocess module contains a number of functions and classes to work with subprocesses. The key tools we have available are:

subprocess.run() – Execute a command, wait, and get a CompletedProcess instance
subprocess.Popen() – Launch a command without waiting. Returns a Popen instance.
subprocess.PIPE – Can be used to connect to stdin/stdout/stderr of a process.
subprocess.STDOUT – Redirect stderr into stdout when capturing output.
subprocess.check_output() – Run command and return stdout as a byte string.
subprocess.CalledProcessError – Exception raised when a process exits with a non-zero code.

The design of the API centers around the concept of launching and communicating with child processes.

Some examples of how to use subprocess:

# Run a process and wait until it finishes
subprocess.run("ls -l", shell=True)

# Launch a process in the background 
p = subprocess.Popen(["python", "script.py"])

# Pipe stdout of one process to another
p1 = subprocess.Popen(["grep"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["wc"], stdin=p1.stdout)

# Get stdout, stderr of a completed process
result = subprocess.run("ls non_existent", capture_output=true)
print(result.stdout, result.stderr)

# Raise error if command fails
subprocess.check_call("false", shell=True)

So in summary, the subprocess module allows launching new processes, interacting via streams, obtaining return codes, etc. This provides an API for tapping into all the capabilities of the underlying OS.

Running External Programs

One of the most common uses of subprocess is to execute external programs from Python code. This allows you to leverage any application or binary on your system directly from Python.

For example, to run an external command and get its output:

import subprocess

result = subprocess.run(["ls", "-l"], capture_output=True)
print(result.stdout)

Some key points:

Use a list to pass arguments to the program.
Set capture_output=True to get stdout/stderr.
The stdout and stderr streams are captured as bytes by default.

Here is how you would run an external binary that takes an input file, does processing, and produces an output file:

input_file = "file.txt"
output_file = "out.txt"

subprocess.run(["/usr/bin/processor", input_file, output_file])

You can even pipe data between processes using stdin and stdout:

p1 = subprocess.Popen(["cat", "file.txt"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "hello"], stdin=p1.stdout, 
                      stdout=subprocess.PIPE)

p2.communicate()

So the subprocess module makes it easy to leverage any program on your system from within Python!

Running Shell Commands

In addition to running external programs, subprocess can be used to execute shell commands directly:

subprocess.run("ls -l", shell=True)

Passing shell=True will run the command string via the shell:

Commands get interpreted by /bin/sh on Linux and cmd.exe on Windows
Allows piping, redirection, wildcards etc.
But has security and portability concerns

It is better practice to use list-based invocation without shell=True:

subprocess.run(["ls", "-l"])

This doesn‘t invoke a shell and is more secure. But you lose some shell features like pipes, wildcards etc.

To capture output from shell commands:

result = subprocess.run("ls -l", check=True,
  stdout=subprocess.PIPE, stderr=subprocess.PIPE,
  shell=True)

print(result.stdout.decode())
print(result.stderr.decode())

So feel free to use shell=True for quick stuff, but avoid it for more robust scripts.

Passing Arguments to Subprocesses

To pass command line arguments to a subprocess, provide them in a list:

subprocess.run(["convert", "img.png", "img.jpg"])

This is equivalent to running convert img.png img.jpg from the shell.

For commands like pip that accept options and arguments, add them as separate items:

subprocess.run(["pip", "install", "pandas==0.24.2"])

To pass arguments with spaces, use quotes:

subprocess.run(["pip", "install", "\"numpy==1.16.4\""])

You can also pass a string and let the shell handle tokenization:

subprocess.run("pip install \"numpy==1.16.4\"", shell=True)

But the list-based method is recommended for robustness.

Capturing Output and Errors

To capture stdout and stderr of a subprocess, pass subprocess.PIPE to stdout and stderr:

result = subprocess.run("ls nonexistent",
  stdout=subprocess.PIPE, stderr=subprocess.PIPE,
  shell=True)

print(result.stdout.decode())
print(result.stderr.decode())

This will collect stdout and stderr as byte strings that you need to explicitly decode.

To save output to a file, open the file and pass the handle:

with open(‘output.txt‘, ‘w‘) as f:
  subprocess.run(["ls"], stdout=f)

If you only care about stdout, redirect stderr to stdout:

result = subprocess.run("ls nonexistent",
  stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
  shell=True)

print(result.stdout.decode())

So subprocess gives you full control over managing output streams of subprocesses.

Checking Return Codes

The return code indicates whether the subprocess ran successfully or not. Get it using returncode:

result = subprocess.run(["false"])

if result.returncode != 0:
  print("Command failed!")

By convention, a return code of 0 means success. Anything else indicates an error.

To raise an exception on non-zero return codes, use check=True:

subprocess.run(["false"], check=True)

This will raise subprocess.CalledProcessError with details on the failure.

Using return codes allows robust handling of failures when scripting with subprocess.

Real-World Usage Examples

Now that you have a solid grasp of the subprocess module, let‘s look at some real-world examples.

Subprocesses enable tons of automation and workflow possibilities. Here are just a few ideas:

Running Git commands:

# Clone a repo
subprocess.run(["git", "clone", "https://github.com/user/repo"])

# Checkout a branch
subprocess.run(["git", "checkout", "main"]) 

# Diff between commits
result = subprocess.run(["git", "diff", "v1.0", "v2.0"], capture_output=True)
print(result.stdout)

# Add modified files, make commit, push 
subprocess.run(["git", "add", "."])
subprocess.run(["git", "commit", "-m", "Update README"])
subprocess.run(["git", "push"])

Installing Python packages:

subprocess.run([sys.executable, "-m", "pip", "install", "requests"])
subprocess.run([sys.executable, "-m", "pip", "install", "pandas==1.3.0"])

Running data science workflows:

# Process data with Pandas 
subprocess.run(["python", "preprocess.py"])

# Train model with Tensorflow
subprocess.run(["python", "train.py"]) 

# Evaluate model 
subprocess.run(["python", "evaluate.py"])

# Export model 
subprocess.run(["python", "export_model.py"])

Chaining shell commands together:

subprocess.run("cd site && ls", shell=True)

Building command pipelines:

p1 = subprocess.Popen(["dmesg"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", "memory"], stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(["wc", "-l"], stdin=p2.stdout, stdout=subprocess.PIPE)

output = p3.communicate()[0]
print(output.decode())

Running devops tasks:

# Build Docker image
subprocess.run(["docker", "build", "-t", "app:v1.3", "."])

# Push built image to registry  
subprocess.run(["docker", "push", "username/app:v1.3"])

# Deploy containers on Kubernetes
subprocess.run(["kubectl", "rollout", "restart", "deployment/app"])

So as you can see, subprocesses help automate pretty much anything you can do from the shell!

Subprocess Best Practices

While subprocesses are very powerful, here are some best practices to use them safely:

Avoid using shell=True unless absolutely required. Use list-based invocation.
Validate user input before passing to subprocess to prevent injection.
Use check=True to automatically raise errors on failures.
Capture stdout/stderr to logs if processes will run unattended.
Define resource limits like CPU time, memory, disk usage to contain processes.
Make sure error handling code exists for failures in automated processes.
Avoid buffering large outputs in memory. Stream outputs to files instead.
Set environment variables carefully and clear sensitive ones.
Use absolute paths for executables instead of relying on PATH.
Consider spawning processes in isolated environments or containers for sandboxing.

Adopting these practices will help avoid issues and limit damage in case something goes wrong.

Subprocess Use Cases from a Python Programmer

As a programmer who uses Python daily, I utilize subprocesses heavily for systems automation. Here are some examples of how I use subprocess in my own work:

Deployments: Scripts that SSH into remote servers, pull code from Git, install/upgrade packages, run migrations, restart services etc.
CI/CD pipelines: Jenkins pipelines that run builds, tests, static analysis, canary deployments.
Infrastructure management: Spinning up cloud resources, provisioning instances, configuration management.
Data pipelines: Extract-Transform-Load scripts, cron jobs to sync data, airflow tasks.
Machine learning workflows: Feature engineering, model training, evaluation, exporting pipelines.
Systems administration: Log parsing, monitoring, backups, maintenance.
Web scraping: Managing crawling, extraction, asynchronous scrapes.

Basically any task that involves chaining together external programs lends itself well to subprocess automation. I find it immensely useful for orchestrating complex workflows.

The cool thing about subprocess is it lets you harness the power and maturity of Unix-style pipelines and shell scripting – but entirely in Python! This opens up many possibilities.

When to Avoid Subprocesses

While subprocesses are powerful, they aren‘t perfect for every use case:

Python libraries exist – If native Python modules already solve your problem, use them instead of shelling out. For example, use paramiko rather than running ssh.
Interactivity required – If you need user interaction like prompts, menus etc. then shelling out won‘t work well.
Not cross-platform – Commands like grep, find differ across Linux/macOS/Windows. May not be portable.
Performs poorly – Shelling out has overhead. For performance critical tasks, keep it in Python.
Security restricted – Sandboxed environments may limit what you can spawn.
No integration needed – No need to shell out if you don‘t need external programs.

So in summary, subprocess is not a magic wand. Use it where it makes sense and provides clear benefits.

Key Takeaways

We‘ve covered a lot of ground here! Let‘s recap the key takeaways:

Subprocesses allow you to run external programs from Python code.
The subprocess module provides convenience APIs for creating child processes.
You can run shell commands, executables, pipe data between processes.
Subprocesses enable you to automate pretty much any task on your OS.
Follow best practices to use subprocesses safely and portably.
Avoid shelling out when native Python solutions exist.

I hope these tips help you become more effective with subprocesses in Python!

The possibilities are endless when it comes to automating tasks and workflows using subprocesses. It unlocks the full power of your OS from within Python.

Subprocesses do have a learning curve, but by starting simple and working your way up, you‘ll be subprocess wizard in no time!

So go forth and let me know if you have any other questions!