How to Use the Pipe Command in Linux

The pipe command in Linux is one of the most frequently used tools by system administrators. If you’re not used to the command line, it’s easy to get confused. After all, it’s all text, scrolling isn’t natural without a mouse, and you can’t press “Ctrl+F” to find something. But once you’ve learned some of the tricks of the command line, and have used it for a while, you’ll see that it’s more natural for certain functions, and the pipe command is a large part of that.

On paper, the Linux pipe command sounds simple – supposedly “interprocess communication”. But such a simple description belies its utility. Specifically, for example, we can use the pipe command for the following purposes:

1. Search, filter, and highlight the output of commands
2. Monitoring processes
3. Transferring files to another server
4. Connect two commands in python

And a lot more. There are many, many applications of the pipe command that rely on an ingenious application of transferring the output of one program to another. In this article, I’ll show you a few such uses, explain the syntax of the pipe command in Linux along with some advanced techniques, and finally we’ll see a few common errors and how we can fix them.

The Syntax of the Pipe Command in Linux

Unlike other Linux commands, the pipe command doesn’t have a specific string to invoke. Instead, it uses the pipe symbol (|) as a separator between the command from which we want to pipe the output, and the command to which we want to pipe it. There are no advanced syntax parameters. The only extra information is that you can pipe together various commands in a chain. For example:

command1 | command2 | command 3

In the above code, the output from command 1 will be sent to command 2, and the output of the previous two will be sent to command 3. This allows you to have a lot of flexibility in crafting your workflows and each Linux administrator will have their own custom set of pipe commands that they use for regular tasks. Linux is all about the personalization of workflows since no two system administrators will face the same set of problems.

The pipe command is more like a shell operator in Linux, rather than a traditional command.

Stdin and Stdout and How They Relate to the Pipe Command

Programs in Linux have three “channels” through which they interact with their environment. The first is for accepting information, also known as stdin. The second is for outputting information, also known as stdout. The third is for outputting error messages.

By default, the “input” channel for a program is the keyboard and the output channel is the screen. The pipe command links the stdout of the first command to the stdin of the second command, allowing multiple commands to be chained together. The implication for this is that it doesn’t matter where the input occurs in the syntax of a command. As long as a command accepts its input from stdin, the pipe command will work.

So, for example, if we have a command like this:

$ hypotheticalcommand [input] -parameters

Then we can still pipe into it, even though the [input] isn’t located towards the end of the line. So the following command would work:

cat file.txt | hypotheticalcommand -parameters

In the above command, “cat” outputs to stdout, and “hypotheticalcommand” accepts input from stdin. So using the pipe operator between them links them together even though when it’s typed by itself, “hypotheticalcommand” doesn’t accept its input at the end of the syntax.

So the pipe operator is agnostic to the position of the input in the syntax.

Uses of the Pipe Command in Linux

In this section, I’ll show you various uses of how to use the pipe command. We’ll start with basic examples – which also happen to be the most common uses – and then show some advanced strategies as well.

Filtering Text

The most common use of the pipe command is to filter the text output of other commands. Let’s say you have a huge log file, recording each login attempt. Anyone who’s monitored a VPS knows that such log files can have thousands upon thousands of entries. If you try and view the log file on a screen, or even using an editor like vi, it can be extremely hard to isolate the events you want.

The pipe command, however, along with the “grep” command is a very efficient way to cut through the noise and isolate exactly what you want. The “grep” syntax is this:

grep [options] pattern [files]

The [files] parameter is either specified or received directly from stdin. As mentioned above, stdin is where the pipe command sends its output. So, for example, we can search a log file for a specific keyword using the following command:

cat file.txt | grep "keyword"

Here’s a screenshot of me using the above to find “Line 3” in a file I used earlier demonstrating how to squash git commits.

Grep will filter the input that it receives – in this case a file – and output the lines containing the string that you’re looking for. As you can see above, it highlights the keywords that it finds in a different color, by default. As usual, you can add various [options] to grep, such as instructing it to ignore the case, return only negatives, or display the line numbers along with the matches that it finds.

If you administer a Linux system, you’ll soon find that you’re using the pipe command with grep all the time to filter not just files, but also outputs from commands that return long streams of text.

We can even use the pipe command to send the output of constantly running processes to grep. For example, let’s say that you’re trying to see SSH login attempts in real time. Here, you can’t use a static log file, since grep will only show you the results of the activity at a given snapshot. Instead, we can use the real-time command “tail” and pipe the results to grep to see who’s trying to SSH into the server.

The command looks like this:

tail -f /var/log/syslog | grep "ssh"

And now whenever someone tries to log in via SSH, grep will show it to you immediately. And that’s the power of text filtering!

Stringing Multiple Commands Together

As mentioned above, we can use the pipe operator to process the outputs of the command sequentially and chain them together. Let’s look at a real-world example of this.

Let’s say we want to print the process numbers of a specific process. We want to achieve this with just a single line and don’t want to wade through a screen. In addition, we want the process number with no fuss, perhaps because we need to use the output somewhere else.

We can achieve the above scenario using the pipe command like this:

ps aux | grep "process_name" | awk '{print $2}

The above command lists the processes using “ps aux”, and then filters the output using grep to only search for the lines containing the process name that we want. Finally, we pipe the output again to “awk”, which prints the second column of the output, giving us the process numbers of the command we want, and nothing else.

Conclusion

The above examples illustrate the power of the pipe command in Linux. We can use it for basic text searching, as well as complex monitoring applications. As your use cases increase in complexity, your usage of the pipe operator will increase, and you’ll soon be piping commands into each other like a pro!