How to Use the sed Command on Linux

A terminal window on a Linux system.Fatmawati Achmad Zaenuri / Shutterstock

It may sound crazy, but the Linux sed command is an interface-free text editor. You can use it from the command line to manipulate text in files and streams. We will show you how to harness its power.

The power of sed

The sed command is a bit like chess: it takes an hour to learn the basics and a lifetime to master them (or at least a lot of practice). We will show you a selection of opening gambits in each of the main categories of sed functionality.

sed is a feed editor which works on entries or text files. However, it does not have an interactive text editing interface. Instead, you provide instructions to follow while it works in the text. All of this works in Bash and other command line shells.

With sed, you can do all of the following:

Select the text
Alternative text
Add lines to text
Delete lines from text
Edit (or keep) an original file

We have structured our examples to introduce and demonstrate concepts, not to produce the tres (and least accessible) sed commands. However, the filtering and text selection features of sed rely heavily on regular expressions (regexes). You are going to need to know them to get the most out of sed.

RELATED: How to use regular expressions (regular expressions) on Linux

A simple example

First, we will use the echo to send text to sed through a pipe, and have replaced part of the text with sed. To do this, we type the following:

echo howtogonk | sed ‘s / gonk / geek /’

The echo command sends “howtogonk” to sed, and our simple substitution rule (the “s” means substitution) is applied. sed searches the input text for an occurrence of the first string and will replace all matches with the second.

The string “gonk” is replaced by “geek” and the new string is printed in the terminal window.

Substitutions are probably the most common use of sed. Before we can delve deeper into substitutions, however, we need to know how to select and match the text.

Text selection

We will need a text file for our examples. We will use one that contains a selection of verses from Samuel Taylor Coleridge’s epic poem “The Rime of the Old Sailor”.

We type the following to take a look with less:

less coleridge.txt

To select certain lines from the file, we provide the start and end lines of the range we want to select. A single number selects this line.

To extract lines one to four, we type this command:

sed -n ‘1,4p’ coleridge.txt

Note the comma between 1 and 4. The p stands for “print the corresponding lines”. By default, sed prints all lines. We would see all the text in the file with the corresponding lines printed twice. To avoid this, we will use the -n (silent) option to remove the unmatched text.

We are modifying the line numbers so that we can select a different verse, as shown below:

sed -n ‘6,9p’ coleridge.txt

We can use the -e (expression) option to make multiple selections. With two expressions, we can select two verses, like this:

sed -n -e ‘1,4p’ -e ’31, 34p ‘coleridge.txt

If we reduce the first number in the second expression, we can insert a blank between the two verses. We type the following:

sed -n -e ‘1,4p’ -e ’30, 34p ‘coleridge.txt

We can also choose a starting line and ask sed to browse the file and print alternate lines, every five lines, or ignore any number of lines. The command is similar to the ones we used above to select a range. This time, however, we will use a tilde (~) instead of a comma to separate the numbers.

The first digit indicates the starting line. The second number tells sed which lines after the start line we want to see. The number 2 means every two lines, 3 means every three lines, and so on.

We type the following:

sed -n ‘1 ~ 2p’ coleridge.txt

You will not always know where the text you are looking for in the file is located, which means that line numbers will not always be of great help. However, you can also use sed to select lines containing matching text patterns. For example, let’s extract all the lines starting with “And”.

The insertion sign (^) represents the beginning of the line. We will include our search term in forward slashes (/). We also include a space after “And” so that words like “Android” are not included in the result.

Reading sed scripts can be a bit difficult at first. The / p stands for “print”, just like it did in the commands we used above. In the following command, however, a forward slash precedes it:

sed -n ‘/ ^ And / p’ coleridge.txt

Three lines starting with “And” are extracted from the file and displayed for us.

Make substitutions

In our first example, we showed you the following basic format for a sed substitution:

echo howtogonk | sed ‘s / gonk / geek /’

The s tells sed that it is a substitution. The first string is the search pattern, and the second is the text with which we want to replace this matching text. Of course, like everything about Linux, the devil is in the details.

We type the following to change all occurrences from “day” to “week” and give the sailor and the albatross more time to bond:

sed -n ‘s / day / week / p’ coleridge.txt

In the first line, only the second occurrence of “day” is changed. Indeed, sed stops after the first match on the line. We must add a “g” at the end of the expression, as shown below, to perform a global search so that all matches on each line are processed:

sed -n ‘s / day / week / gp’ coleridge.txt

This corresponds to three of the four in the first line. Since the first word is “Day” and sed is case sensitive, it does not consider this instance to be the same as “day”.

We type the following, adding an i to the command at the end of the expression to indicate case insensitivity:

sed -n ‘s / day / week / gip’ coleridge.txt

It works, but you may not always want to turn on case insensitivity for everything. In these cases, you can use a group of regular expressions to add pattern-specific case insensitivity.

For example, if we put characters in square brackets ([]), they are interpreted as “any character from this list of characters”.

We type the following and include “D” and “d” in the group, to make sure it matches both “Day” and “day”:

sed -n / s[Dd]ay / week / gp ‘coleridge.txt

We can also limit substitutions to sections of the file. Let’s say our file contains strange spacing in the first verse. We can use the following familiar command to see the first verse:

sed -n ‘1,4p’ coleridge.txt

We are going to find two spaces and replace them with one. We will do it globally so that action is repeated across the board. To be clear, the search pattern is space, the space asterisk (*), and the substitution string is a single space. The 1.4 limits the substitution to the first four lines of the file.

We put it all together in the following command:

sed -n ‘1,4 s / * / / gp’ coleridge.txt

That works well! The research model is what is important here. The asterisk (*) represents zero or more of the preceding character, which is a space. Thus, the search model searches for strings of one or more spaces.

If we substitute a single space for a sequence of several spaces, we will restore the regular spacing of the file, with a single space between each word. In some cases, this will also replace a single space with a single space, but it won’t affect anything, we will always get the desired result.

If we type the following and reduce the search pattern to a single space, you will immediately see why we should include two spaces:

sed -n ‘1,4 s / * / / gp’ coleridge.txt

Since the asterisk matches zero or more of the preceding character, it sees each non-space character as “zero space” and applies the substitution to it.

However, if we include two spaces in the search pattern, sed must find at least one space character before applying the substitution. This ensures that non-spatial characters will remain intact.

We type the following, using -e (expression) that we used previously, which allows us to make two or more substitutions simultaneously:

sed -n -e ‘s / motion / flutter / gip’ -e ‘s / ocean / gutter / gip’ coleridge.txt

We can get the same result if we use a semicolon (;) to separate the two expressions, like this:

sed -n ‘s / motion / flutter / gip; s / ocean / gutter / gip ‘coleridge.txt

When we exchanged “day” for “week” in the following command, the instance of “day” in the expression “well one day” was also exchanged:

sed -n / s[Dd]ay / week / gp ‘coleridge.txt

To avoid this, we can only attempt substitutions on lines which correspond to another model. If we modify the command to have a search pattern at the start, we will only consider the operation on the lines that match this pattern.

We type the following to make our matching model the word “after”:

sed -n ‘/ after / s /[Dd]ay / week / gp ‘coleridge.txt

This gives us the answer we want.

More complex substitutions

Let’s take a break from Coleridge and use sed to extract the names from the etc / passwd file.

There are shorter ways to do this (more on that later), but we’ll use the longer way here to demonstrate another concept. Each corresponding element in a search pattern (called sub-expressions) can be numbered (up to a maximum of nine elements). You can then use these numbers in your sed commands to reference specific subexpressions.

You must put the sub-expression in parentheses [()] to make it work. Parentheses must also be preceded by a forward slash () to prevent them from being treated as a normal character.

To do this, you need to type the following:

sed / s ([^:]* ). * / 1 / ‘/ etc / passwd

sed / s ([^:]* ). * /  1 / '/ etc / passwd in a terminal window

Let’s break this down:

sed / s: The sed command and the start of the substitution expression.
(: The opening parenthesis [(] surrounding the subexpression, preceded by a backslash ().
[^:]*: The first sub-expression of the search term contains a group in square brackets. The insertion sign (^) means “no” when used in a group. A group means that any character that is not a colon (:) will be accepted as a match.
): The closing parenthesis [)] with a previous backslash ().
. *: This second search sub-expression means “any character and any number”.
/ 1: The substitution part of the expression contains 1 preceded by a backslash (). This represents the text that matches the first subexpression.
/ ‘: The closing slash (/) and the single quotation mark (‘) end the sed command.

This means that we will search for any character string not containing a colon (:), which will be the first instance of corresponding text. Then we look for something else on this line, which will be the second instance of text matching. We will replace the entire line with the text corresponding to the first sub-expression.

Each line in the / etc / passwd file begins with a username ending with a colon. We match everything up to the first colon, then substitute this value for the entire line. So we isolated the usernames.

Release

Then we will put the second sub-expression in parentheses [()] so that we can also reference it by number. We will also replace 1 with 2. Our command will now replace the entire line with everything, from the first colon (:) to the end of the line.

We type the following:

sed / s ([^:]* ) (. * ) / 2 / ‘/ etc / passwd

These small changes reverse the direction of the command, and we get everything except usernames.

Exit from sed 's /  ([^:]* )  (. * ) /  2 / '/ etc / passwd in a terminal window

Now let’s see the quick and easy way to do it.

Our search term is from the first colon (:) to the end of the line. Since our substitution expression is empty (//), we will not replace the corresponding text with anything.

So we type the following, cutting everything from the first colon (:) at the end of the line, leaving only usernames:

sed ‘s /:.*// “/ etc / passwd

Let’s look at an example in which we reference the first and second matches in the same command.

We have a comma file (,) separating the first and last name. We want to list them as “last name, first name”. We can use cat, as shown below, to see what’s in the file:

cat geeks.txt

Like many sed commands, this one might seem impenetrable at first:

sed ‘s / ^ (. * ), (. * ) $ / 2, 1 / g’ geeks.txt

This is a substitution command like the others we used, and the search pattern is quite simple. We will detail it below:

sed / s: The normal substitution command.
^: Because the cursor is not in a group ([]), it means “The beginning of the line”.
(. * ),: The first sub-expression is any number of characters. It is placed in parentheses [()], each one being preceded by a backslash () so that we can reference it by a number. So far, our full search pattern results in a search from the start of the line to the first comma (,) for any number of characters.
(. * ): The next subexpression is (again) any number of any character. It is also placed in parentheses [()], both preceded by a backslash () so that we can reference the corresponding text by number.
$ /: The dollar sign ($) represents the end of the line and will allow our search to continue until the end of the line. We simply used it to introduce the dollar sign. We don’t really need it here, because the asterisk (*) would go to the end of the line in this scenario. The forward slash (/) ends the search pattern section.
2, 1 / g ‘: Because we have put our two sub-expressions in parentheses, we can designate them both by their number. Because we want to reverse the order, we type them as second match, first match. Numbers must be preceded by a backslash ().
/ g: This allows our command to work globally on each line.
geeks.txt: The file we are working on.

You can also use the Cut command (c) to replace entire lines that match your search pattern. We type the following to find a line containing the word “neck” and replace it with a new text string:

sed ‘/ neck / c Around my wrist was threaded’ coleridge.txt

Our new line now appears at the bottom of our extract.

Inserting lines and text

We can also insert new lines and text into our file. To insert new lines after the corresponding ones, we will use the Add (a) command.

Here is the file with which we will work:

cat geeks.txt

We have numbered the lines to make this a little easier to follow.

We type the following to find the lines that contain the word “He” and insert a new line below:

sed ‘/ He / a -> Inserted!’ geeks.txt

Inserted! ‘geeks.txt “command in a terminal window.” width = “646” height = “262” onload = “pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);” onerror = “this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);” />

We type the following and include the insert command (i) to insert the new line above those that contain the corresponding text:

sed ‘/ He / i -> Inserted!’ geeks.txt

Inserted! ‘geeks.txt “command in a terminal window.” width = “646” height = “262” onload = “pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);” onerror = “this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);” />

We can use the ampersand (&), which represents the original matched text, to add new text to a corresponding line. 1, 2, etc., represent corresponding sub-expressions.

To add text to the beginning of a line, we will use a substitution command that matches everything on the line, combined with a replacement clause that combines our new text with the original line.

To do all of this, we type the following:

sed ‘s /.*/–> Inserted & /’ geeks.txt

Insert the & / ‘geeks.txt “command in a terminal window.” width = “646” height = “212” onload = “pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);” onerror = “this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);” />

We type the following, including the G command, which will add a blank line between each line:

sed ‘G’ geeks.txt

If you want to add two or more blank lines, you can use G; G, G; G; G, etc.

Delete rows

The Delete command (d) deletes the lines that match a search pattern or those specified with line numbers or ranges.

For example, to delete the third line, we would type the following:

sed ‘3d’ geeks.txt

To delete the range of lines four to five, we type the following:

sed ‘4,5d’ geeks.txt

To delete lines outside of a range, we use an exclamation mark (!), As shown below:

sed ‘6.7! from geeks.txt

Save your changes

So far, all of our results have been printed in the terminal window, but we haven’t saved them anywhere yet. To make them permanent, you can either write your changes to the original file, or redirect them to a new one.

Replacing your original file requires some caution. If your sed command is incorrect, you can make changes to the original file that are difficult to reverse.

For peace of mind, sed can create a backup of the original file before executing its command.

You can use the in-place option (-i) to tell sed to write the changes to the original file, but if you add a file extension to it, sed will save the original file to a new one. It will have the same name as the original file, but with a new file extension.

To demonstrate this, we will search for all the lines containing the word “He” and delete them. We will also save our original file to a new one using the BAK extension.

To do all of this, we type the following:

sed -i’.bak ” /^.*He.*$/d ‘geeks.txt

We type the following to make sure that our backup file is unchanged:

geeks.txt.bak chat

We can also type the following to redirect the output to a new file and get a similar result:

sed -i’.bak ” /^.*He.*$/d ‘geeks.txt> new_geeks.txt

We use cat to confirm that the changes have been written to the new file, as shown below:

cat new_geeks.txt

new_geeks.txt “and” cat new_geeks.txt “commands in a terminal window.” width = “646” height = “307” onload = “pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);” onerror = “this.onerror = null; pagespeed.lazyLoadImages.loadIfVisibleAndMaybeBeacon (this);” />

Having seduced it all

As you have probably noticed, even this quick introduction to sed is quite long. There’s a lot to this order, and there’s even you can do more.

We hope, however, that these basic concepts have provided a solid foundation on which you can build on to continue learning more.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.