How to Run Bash Script/Command Using Python? An In-Depth Guide

As a programmer, being able to execute bash scripts and commands from within Python opens up many possibilities for building robust and integrated automation workflows. In this comprehensive guide, we‘ll explore the ins and outs of running bash from Python.

Why Run Bash from Python?

Here are some of the main reasons you may want to execute bash scripts/commands from Python:

Automate workflows – Run bash commands as part of an automated workflow driven by Python code. For example, you could process data in Python, then call a bash script to deploy a machine learning model.
Leverage existing scripts – Many Linux tools and scripts are already written in bash. Executing them from Python allows integrating with these while keeping Python at the core of your workflow.
Portability – Bash scripts work across Linux, macOS, and even Windows when using WSL. Calling them from Python creates portable workflows.
Security – Some system administration tasks require elevated privileges that may be dangerous to perform directly from Python. Running a vetted bash script instead can improve security.
Shell capabilities – Bash provides some shell capabilities like globbing, pipes etc. that Python does not match directly.

Overall, running bash scripts and commands allows tapping into the strengths of both languages. Python remains the driving force, with bash providing runtime system control and access.

Key Modules for Running Bash from Python

Python has some great built-in modules for running subprocesses, which we can leverage for invoking bash scripts and commands.

The two most relevant ones are:

1. subprocess – Provides tons of control for spawning processes and interacting with them via stdin, stdout and stderr pipes. Offers multiple ways to call external programs.

2. os – Contains alternative methods like os.system() for running processes at a lower level.

The subprocess module is generally recommended for running bash commands from python. It works well across all major operating systems.

Let‘s look at some code examples of using subprocess and os.system() to run bash scripts and commands from Python.

Running Bash Commands with subprocess

The subprocess module provides a flexible cross-platform way to spawn child processes and manage them from Python code.

Some of the key features we get with subprocess for running bash commands:

Get return codes and output values
Pipe data to stdin or read stdout/stderr
Full control over the child process lifecycle

Let‘s see some examples of running bash commands with subprocess.

subprocess.run()

The simplest approach is using subprocess.run(), which executes a command and returns a CompletedProcess object when finished:

import subprocess

result = subprocess.run(["ls", "-l"])
print(result.returncode) # 0

This runs ls -l, waits for completion, and returns the exit code in result.

To capture stdout and stderr:

result = subprocess.run(["ls", "fakefile"], 
                        stdout=subprocess.PIPE, 
                        stderr=subprocess.PIPE)

print(result.stdout) # b‘‘ 
print(result.stderr) # b‘ls: cannot access fakefile: No such file or directory\n‘

We can also pass input to the process‘s stdin:

result = subprocess.run(["grep", "hello"], 
                        input="hello world", text=True)
print(result.returncode) # 0

The text=True parameter is needed to pass the input as a string rather than bytes.

subprocess.Popen

For more control over the subprocess, we can use subprocess.Popen. This launches the process in the background and returns a Popen object representing it.

Some ways we can then interact with the process:

process = subprocess.Popen(["ping", "google.com"])

process.communicate() # wait for finish

process.stdin.write("hello") # send input

output = process.stdout.read() # read output  

process.terminate() # kill process

Let‘s look at a full example:

import subprocess

process = subprocess.Popen(["ping", "8.8.8.8"], stdout=subprocess.PIPE)

while True:
  line = process.stdout.readline()
  if line == b"" and process.poll() != None:
    break
  if line:
    print(line.decode().strip()) # convert bytes to string

This pings continuously until the process exits, printing each line of output.

As you can see, subprocess.Popen provides very granular control over running programs like bash commands in Python.

Other subprocess Methods

Some other subprocess methods that can be useful:

call() – Runs a command and returns exit code
check_output() – Returns stdout as a string
getoutput() – Gets output as string, but deprecated
getstatusoutput() – Returns (exitcode, output) tuple

So in summary, subprocess gives us tons of options for invoking bash commands and handling the input and output streams.

Executing Bash Scripts from Python

Bash scripts contain series of bash commands that are executed in order.

To run a bash script from Python, we just need to execute the script file path using subprocess:

import subprocess

subprocess.run(["/path/to/script.sh"])

We can also pass arguments to the script:

subprocess.run(["/path/to/script.sh", "arg1", "arg2"])

For example:

import subprocess

subprocess.run(["./count.sh", "5"]) # script counts to 5

Essentially, it works the same as running a bash command – the only difference is that we provide a script file instead.

The same subprocess methods shown above like run(), call(), Popen etc. can be used to execute bash scripts.

Running Bash with os.system()

The os module provides some more direct ways of running bash commands, including the os.system() method.

Basic usage:

import os

os.system("ls -l")

This runs the command and returns the exit code.

However, os.system() should generally be avoided for executing bash commands from Python, for several reasons:

Provides less control over the subprocess and input/output streams
Can be a security risk since it uses the shell to run commands
Not available on Windows, reducing portability
Limited error handling and reporting

So while it can work in some cases, subprocess is preferable over os.system() in most scenarios.

Best Practices for Running Bash from Python

Here are some tips for smoothly integrating bash scripts/commands into your Python code:

Use subprocess rather than os.system() for reasons outlined above
Consider security implications before running bash scripts
Avoid using bash for long-running processes – use Python threads or processes instead
Capture stdout/stderr rather than printing directly, for clean logging
Use -c flag with bash command to run short one-liners, eg: ["bash", "-c", "echo hello"]
Handle errors gracefully – catch CalledProcessError and handle exit codes

Following these best practices will ensure robust and secure execution of bash commands from Python code.

Conclusion

Python provides great facilities for executing external bash scripts and commands via the subprocess module. Here are some key takeaways:

Use subprocess for running bash from Python
- run(), Popen and other methods available
Pass script file path to execute bash scripts
Capture stdout, stderr and exit codes for robustness
Avoid os.system() – subprocess is more flexible & portable
Follow security best practices when running bash scripts

Integrating Python with the Unix shell opens up many possibilities for deploying production pipelines, DevOps automation, and more. I hope this guide has provided a solid overview of how to run bash from Python and best practices around it.

Let me know if you have any other questions!