How to Use Pipes on Linux

Linux terminal on a laptopFatmawati Achmad Zaenuri / Shutterstock.com

Use Linux channels to choreograph how the command-line utilities work together. Simplify complex processes and increase your productivity by leveraging a collection of stand-alone commands and turning them into a committed team. We show you how.

Pipes are everywhere

Pipes are one of the most useful command-line features of Linux and Unix operating systems. Pipes are used in many ways. Look at any Linux command line item – on any web site, not just ours – and you'll see that pipes appear most often. I reviewed some How-To Geek articles on Linux, and pipes are used in all, one way or another.

Linux channels allow you to perform actions that are not immediately supported by the machine. shell. But as Linux design philosophy is to have many small utilities that perform their tasks. dedicated function very well, and with no useless functionality – the mantra "do one thing and do it right" – you can combine command strings with channels so that the result of one command becomes the input of the other. Every order you bring brings his unique talent to the team and you'll quickly see that you've built a winning team.

A simple example

Suppose we have a directory full of many types of files. We want to know how many files of a certain type are in this directory. There are other ways to do this, but the purpose of this exercise is to introduce pipes. So we will do it with pipes.

We can easily get a list of files using ls:

ls

Collection if files in a directory, in a terminal window

To separate the file type that interests you, we will use grep. We want to find files that have the word "page" in their file name or extension.

We will use the special shell character "|" to direct the output of ls in grep.

ls | grep "page"

grep prints lines that match his search pattern. This gives us a list containing only ".page" files.

list of page files in a terminal window

Even this trivial example displays the functionality of pipes. The output of ls has not been sent to the terminal window. It was sent to grep as data for the grep command to work. The output we see comes from grep, which is the last command of this chain.

Extend our chain

Let's start expanding our chain of orders. we can count the ".page" files by adding the wc command. We will use the option -l (number of lines) with wc. NOTE: We have also added the -l option (long format) to ls. We will use that shortly.

ls – | grep "page" | wc -l

grep is no longer the last command in the chain, so we do not see its output. The output of grep is introduced in the wc command. The output displayed in the terminal window is from wc. wc indicates that there are 69 ".page" files in the directory.

Let's take things a step further. We will remove the wc command from the command line and replace it with awk. The output of ls has nine columns with the -l option (long format). We will use awk for print columns five, three and nine. These are the size, the owner, and the name of the file.

ls -l | grep "page" | awk # {print $ 5 "" $ 3 "" $ 9} & # 39;

We get a list of these columns for each of the corresponding files.

List of three columns for each corresponding file in a terminal window

We will now send this output via the sort command. We'll use the -n (numeric) option to tell fate that the first column should be treated as a number.

ls -l | grep "page" | awk # {print $ 5 "" $ 3 "" $ 9} & # 39; | sort -n

The output is now sorted by file size order, with our custom selection of three columns.

Files sorted by size in a terminal window

Addition of another order

We will finish by adding the tail command. We will tell him to list the last five lines of exit only.

ls -l | grep "page" | awk # {print $ 5 "" $ 3 "" $ 9} & # 39; | sort -n | tail -5

This means that our order translates to "show me the five largest" .page "files in this directory, sorted by size." Of course, no command can accomplish this, but using tubes, we created our own. We could add this – or any other long command – as an alias or shell function to record all the typing.

Here is the exit:

Five largest .page files listed in order of size in a terminal window

We could reverse the order of sizes by adding the -r (inverse) option to the sort command and using head instead of tail to choose the lines. from the top of the exit.

This time, the five largest ".page" files are listed from largest to smallest:

Five largest .page files listed in reverse order in a terminal window

Some recent examples

Here are two interesting examples from recent geek articles.

Some commands, such as the xargscommand, are designed have an input connected to them. Here's a way to count the number of words, characters and lines in several files, by throwing ls in xargs, which then feeds the list of file names into wc as they had been passed to wc as command line parameters.

ls * .page | xargs wc

ls * .page | wc xargs in a terminal window

The total number of words, characters, and lines is listed at the bottom of the terminal window.

Number of words, charcaters and lines in a terminal window

Here is a way to get a sorted list of unique file extensions from the current directory, with a number of each type.

ls | rev | cut-off. -f1 | rev | sort | uniq -c

ls | rev | cut-off. -f1 | rev | sort | uniq -c in a terminal window

There is a lot going on here.

The output displays the list of file extensions, sorted alphabetically with a number of each unique type.

List of unique file extensions in a terminal window

Named Pipes

There is another type of hose available, called named pipes. Tubes from the previous examples are created on the fly by the shell when processing the command line. The pipes are created, used and then discarded. They are transient and leave no trace of themselves. They only exist as long as the command that uses them is running.

Named pipes appear as persistent objects in the file system, so you can see them with ls help. They are persistent because they will survive the restart of the computer, even if all unread data they contain at that time will be deleted.

Named pipes have been used extensively to allow different processes to send and receive data, but I have not seen them used in this way for a long time. There are probably people who still use them wisely, but I have not met them recently. But for the sake of completeness, or simply to satisfy your curiosity, here is how you can use them.

Named pipes are created with the mkfifo command. This order will create a named pipe called "geek-pipe" in the current directory.

mkfifo geek-pipe

mkfifo geek-pipe in a terminal window

We can see the details of the named pipe if we use the ls command with the -l option (long format):

ls -l geek-pipe

ls -l geek-pipe in a terminal window

The first character in the list is a "p", which means that it's a pipe. If it was a "d", it would mean that the object of the file system is a directory and a dash "-" would mean that it 's only one directory. is a normal file.

Using the named pipe

Let's use our pipe. The unnamed channels that we used in our previous examples immediately passed the data from the send command to the receive command. Data sent via a named channel will remain in the channel until it is read. The data is actually stored in memory, so that the size of the named channel does not vary in lists ls, whether there is data or not.

We will use two terminal windows for this example. I will use the label:

# Terminal 1

in a terminal window and

# Terminal 2

in the other, so you can differentiate them. The hash "#" tells the shell that the following is a comment and that it should ignore it.

Let's take the entirety of our previous example and redirect it into the named pipe. So we use named and unnamed channels in a single command:

ls | rev | cut-off. -f1 | rev | sort | uniq -c> geek-pipe

ls | rev | cut-off. -f1 | rev | sort | uniq -c> geek-pipe in a terminal window

Nothing will seem to happen. You may notice that you are not sent back to the command prompt, so something is happening.

In the other terminal window, run this command:

cat <geek-pipe

<img class = "full-size alignnone wp-image-439186" data-pagespeed-lazy-src = "https://www.howtogeek.com/wp-content/uploads/2019/09/x20.png.pagespeed. gp + jp + jw + pj + ws + js + rj + rp + rw + ri + cp + md.ic.eHaOfO4Dp2.png "alt =" cat

We redirect the contents of the named pipe to cat so that it displays this content in the second window of the terminal. Here is the exit:

The contents of the named piping are displayed in a terminal window

And you will see that you have been returned to the command prompt in the first terminal window.

Completed task and command prompt in a terminal window

So, what just happened.

We redirected some exits in the named pipe.
The first terminal window is not returned to the command prompt.
The data remained in the pipe until it was read into the pipe of the second terminal.
We were sent back to the command prompt in the first terminal window.

You may think that you could run the command in the first terminal window as a background task by adding an & at the end of the command. And you would be right. In this case, we would have been immediately returned to the command prompt.

The point of not using background processing was to point out that a named pipe is a blocking process. Putting something in a named pipe only opens one end of the pipe. The other end is only open when the read program retrieves the data. The kernel suspends the process in the first window of the terminal until the data is read at the other end of the channel.

The power of pipes

Nowadays, named pipes are in a way an act of novelty.

On the other hand, old Linux pipes are one of the most useful tools you can have in your Terminal Window Toolbox. The Linux command line begins to come alive for you, and you get a whole new power-up when you can orchestrate a collection of commands to produce consistent performance.

Separation Tip: It is best to write your piped commands by adding one command at a time and running this part, then piping into the next command.

Advertisements

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.