Home ›

Variable Mangling in Bash with String Operators

Mar 13, 2006  By Pat Eyler

 in

• Linux Journal

Here's a quick and updated HOWTO for using string operators in bash to manipulate variables.

Editor's Note: This article has been updated by its author. Thank you, Pat.
Have you ever wanted to change the names of many files at once? Or, have you ever needed to use a default value for a variable that has no value? These and many other options are available to you when you use string operators in bash and other Bourne-derived shells.
String operators allow you to manipulate the contents of a variable without having to write your own shell functions to do so. They are provided through "curly brace" syntax. Any variable can be displayed as ${foo} without changing its meaning. This functionality often is used to protect a variable name from surrounding characters.

$ export foo=foo 
$ echo ${foo}bar # foo exists so this works as expected
foobar
$ echo $foobar # foobar doesn't exist, so this doesn't

$

By the end of this article, you'll be able to use it for a whole lot more.
Three kinds of variable substitution are available for use: pattern matching, substitution and command substitution. I talk about the first two variables here and leave command substitution for another time.

Pattern Matching

In pattern matching, you can match from the left or from the right. The operators, along with their functions and examples, are shown below:
Operator: ${foo#t*is}
Function: deletes the shortest possible match from the left
Example:

$ export foo="this is a test"
$ echo ${foo#t*is}
is a test
$

Operator: ${foo##t*is}
Function: deletes the longest possible match from the left
Example:

$ export foo="this is a test"
$ echo ${foo##t*is}
a test
$

Operator: ${foo%t*st}
Function: deletes the shortest possible match from the right
Example:

$ export foo="this is a test"
$ echo ${foo%t*st}
this is a
$

Operator: ${foo%%t*st}
Function: deletes the longest possible match from the right
Example:

$ export foo="this is a test"
$ echo ${foo%%t*st}

$

Although the # and % identifiers may not seem obvious, they have a convenient mnemonic. The # key is on the left side of the $ key and operates from the left. The % key is on the right of the $ key and operates from the right. (This is true, at least, for US qwerty keyboards.)
The operators listed above can be used to do a variety of things. For example, the following script changes the extension of all .html files so they now are .htm files.

#!/bin/bash 
# quickly convert html filenames for use on a dossy system
# only handles file extensions, not file names
for i in *.html; do
   if [ -f ${i%l} ]; then 
      echo "${i%l} already exists"
   else 
      mv $i ${i%l}
   fi 
done

Substitution

Another kind of variable mangling you might want to employ is substitution. Four substitution operators are used in Bash, and they are shown below:
Operator: ${foo:-bar}
Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, return bar.
Example:

$ export foo=""
$ echo ${foo:-one}
one
$ echo $foo

$

Operator: ${foo:=bar}
Function: If $foo exists and is not null, return $foo. If it doesn't exist or is null, set $foo to bar and return bar.
Example:

$ export foo=""
$ echo ${foo:=one}
one

$ echo $foo
one
$

Operator: ${foo:+bar}
Function: If $foo exists and is not null, return bar. If it doesn't exist or is null, return a null.
Example:

$ export foo="this is a test"
$ echo ${foo:+bar}
bar
$

Operator: ${foo:?"error message"}
Function: If $foo exists and isn't null, return its value. If it doesn't exist or is null, print the error message. If no error message is given, it prints parameter null or not set. In a non-interactive shell, this aborts the current script. In an interactive shell, this simply prints the error message.
Example:

$ export foo="one"
$ for i in foo bar baz; do
> eval echo \${$i:?}
> done
one
bash: bar: parameter null or not set
bash: baz: parameter null or not set
$

The : in the above operators can be omitted. Doing so changes the behavior of the operator so that it simply tests for the existence of the variable. This, in turn, causes the creation of a variable, for example:

$ export foo="this is a test"
$ echo $bar

$ echo ${foo=bar}
this is a test
$ echo ${bar=bar}
bar
$ echo $bar
bar
$          

These operators can be used in a variety of ways. A good example would be, in the case when no arguments are given, to give a default value to a variable normally read from command-line arguments. This example is demonstrated in the following script:

#!/bin/bash 
export INFILE=${1-"infile"} 
export OUTFILE=${2-"outfile"}
cat $INFILE > $OUTFILE

Copyright (c) 2005, 2000 by Pat Eyler. Originally published in Linux Gazette issue 57. Copyright (c) 2000, Specialized Systems Consultants, Inc. The material in this article may be distributed only subject to the terms and conditions set forth in the Open Publication License, v1.0 or later.

Bash String Manipulation Examples – Length, Substring, Find and Replace

by Sasikala on July 23, 2010

In bash shell, when you use a dollar sign followed by a variable name, shell expands the variable with its value. This feature of shell is called parameter expansion.
But parameter expansion has numerous other forms which allow you to expand a parameter and modify the value or substitute other values in the expansion process. In this article, let us review how to use the parameter expansion concept for string manipulation operations.
This article is part of the on-going bash tutorial series. Refer to our earlier article on bash { } expansion.

1. Identify String Length inside Bash Shell Script

${#string}

The above format is used to get the length of the given bash variable.

$ cat len.sh
#! /bin/bash

var="Welcome to the geekstuff"

echo ${#var}

$ ./len.sh
24

To understand more about bash variables, read 6 Practical Bash Global and Local Variable Examples.

2. Extract a Substring from a Variable inside Bash Shell Script

Bash provides a way to extract a substring from a string. The following example expains how to parse n characters starting from a particular position.

${string:position}

Extract substring from $string at $position

${string:position:length}

Extract $length of characters substring from $string starting from $position. In the below example, first echo statement returns the substring starting from 15th position. Second echo statement returns the 4 characters starting from 15th position. Length must be the number greater than or equal to zero.

$ cat substr.sh
#! /bin/bash

var="Welcome to the geekstuff"

echo ${var:15}
echo ${var:15:4}

$ ./substr.sh
geekstuff
geek

Also, refer to our earlier article to understand more about http://www.thegeekstuff.com/2010/05/bash-shell-special-parameters/.

3. Shortest Substring Match

Following syntax deletes the shortest match of $substring from front of $string

${string#substring}

Following syntax deletes the shortest match of $substring from back of $string

${string%substring}

Following sample shell script explains the above two shortest substring match concepts.

$ cat shortest.sh
#! /bin/bash

filename="bash.string.txt"

echo ${filename#*.}
echo ${filename%.*}

$ ./shortest.sh
After deletion of shortest match from front: string.txt
After deletion of shortest match from back: bash.string

In the first echo statement substring ‘*.’ matches the characters and a dot, and # strips from the front of the string, so it strips the substring “bash.” from the variable called filename. In second echo statement substring ‘.*’ matches the substring starts with dot, and % strips from back of the string, so it deletes the substring ‘.txt’

4. Longest Substring Match

Following syntax deletes the longest match of $substring from front of $string

${string##substring}

Following syntax deletes the longest match of $substring from back of $string

${string%%substring}

Following sample shell script explains the above two longest substring match concepts.

$ cat longest.sh
#! /bin/bash

filename="bash.string.txt"

echo "After deletion of longest match from front:" ${filename##*.}
echo "After deletion of longest match from back:" ${filename%%.*}

$ ./longest.sh
After deletion of longest match from front: txt
After deletion of longest match from back: bash

In the above example, ##*. strips longest match for ‘*.’ which matches “bash.string.” so after striping this, it prints the remaining txt. And %%.* strips the longest match for .* from back which matches “.string.txt”, after striping  it returns “bash”.

5. Find and Replace String Values inside Bash Shell Script

Replace only first match

${string/pattern/replacement}

It matches the pattern in the variable $string, and replace only the first match of the pattern with the replacement.

$ cat firstmatch.sh
#! /bin/bash

filename="bash.string.txt"

echo "After Replacement:" ${filename/str*./operations.}

$ ./firstmatch.sh
After Replacement: bash.operations.txt

Replace all the matches

${string//pattern/replacement}

It replaces all the matches of pattern with replacement.

$ cat allmatch.sh
#! /bin/bash

filename="Path of the bash is /bin/bash"

echo "After Replacement:" ${filename//bash/sh}

$ ./allmatch.sh
After Replacement: Path of the sh is /bin/sh

Taking about find and replace, refer to our earlier articles – sed substitute examples and Vim find and replace.

Replace beginning and end

${string/#pattern/replacement

Following syntax replaces with the replacement string, only when the pattern matches beginning of the $string.

${string/%pattern/replacement

Following syntax replaces with the replacement string, only when the pattern matches at the end of the given $string.

$ cat posmatch.sh
#! /bin/bash

filename="/root/admin/monitoring/process.sh"

echo "Replaced at the beginning:" ${filename/#\/root/\/tmp}
echo "Replaced at the end": ${filename/%.*/.ksh}

$ ./posmatch.sh
Replaced at the beginning: /tmp/admin/monitoring/process.sh
Replaced at the end: /root/admin/monitoring/process.ksh