Tables are inspired at tables you can find in:
O’Really, Learning the bash Shell, 3rd Edition By Cameron Newham

Utilities

The most wanted utilities in unix should surely be the following ones:

Utility	Purpose
cat grep sort cut sed rt awk	Copy input to output Search for string in the input Sort lines in the input Extract columns from input Perform editing operation on input Translate characters in the input to other characters Full-featured text processing language with a syntax reminiscent of C

For example the grep command is very useful to fing text inside files like shown below:

grep –Hnr “Fabio Bombardi” .

This command search for the string “Fabio Bombardi” in all files, recursively (-r) , starting from the current directory (.), showing the line number (-n) where this string is located in the found file.

I/O Redirectors and pipelines

Redirector	Purpose
< >	Redirect the right side argument as input of the left side argument Redirect the output of the left side argument to the left right side argument

For example if your cp command is broken you can use the cat command in the following way:

cat < file1 > file2

This behaves like the command

cp file1 file2

Redirector	Purpose
cmd1 | cmd2	Redirect the output of the left side command cmd1 to the input of the right side command cmd2

For example you can use the command

ls -l | more

to send the list the file in the current directory as the input of the more command.

It’s possible to use the I/O redirection in conjunction with pipelines like in the following example:

cut –d: -f1 < /etc/passwd | sort

this command extract the first field (f1) in the file /etc/passwd, where fields are separated by colons (-d:) and sort that first field showing it on the output.

The above redirectors are the most used. But you may found a plethora of other useful redirectors, above all for a system programmer, like those listed in the table below.

Redirector

Purpose

> file < file >> file >| file n>| file <> file n <> file << label n> file n< file n>> file n>& n<& n>&m n<&m &> file <&- >&- n>&- n<&-

Direct standard output to file Take standard input from file Direct standard output to file; append to file if it already exists Force standard output to file even if noclobber is set Force output to file from file descriptor n even if noclobber is set Use file as both input and output for file descriptor n Use file as both input and output for file descriptor n Here-document; see text Direct file descriptor n to file Take file descriptor n from file Direct file descriptor n to file; append to file if it already exists Duplicate standard output to file descriptor n Duplicate standard input from file descriptor n File descriptor n is made to be a copy of the output file descriptor m File descriptor n is made to be a copy of the input file descriptor m Directs standard output and standard error to file Close the standard input Close the standard output Close the output from file descriptor n Close the input from file descriptor ni

For example if you want to redirect both the standard error and the standard output of the ls command to a logfile you could type what follows:

ls > logfile 2>&1

This will redirect the standard error (2) in the same place where the standard output (1) is directed. Since standard output is redirected to logfile hence also the standard error will be redirected there too.

If you want you could get the same result also with the following commnd:

ls &> logfile

Special characters and quoting

Special characters

Character	Purpose
~ # $ & * ( ) \ | [ ] { } ; ‘ “ < > / ? !	Home directory Comment Variable expression Background job String wildcar Start subshell End subshell Quote next character Pipe Start character-set wildcard End Character-set wildcard Smart command block End command block Shell command separator Strong quote Weak quote Input redirect Ouput redirect Pathname directory separator Single-character wildcard Pipeline logical NOT

Quoting

When you want to use special character without their special meaning you have to use quoting. If you surround a string of characters with single quotation marks (or quotes), you strip all characters within the quotes of any special meaning they might have.

For example if you want to print in standard output the string “2*3 > 5 is an invalid inequality” you have to type the following command:

echo ‘2 * 3 > 5 is an invalid inequality’

Otherwise you will get a file named 5 containing the list of 2, all the files in the current directory, and the string “3 is an invalid inequality”!

Another way to change the meaning of a character is to precede it with a backslash (\). This is called backslash-escaping the character.

Quoting and find

The most useful use of the quoting is along with the command find.

If you want to find all the character with .c extension you have to act like below:

find . -name ‘*.c’

Control key setting

To know your control-key setting you can type

stty all

And you will get something like that:

erase kill werase rprnt flush lnext susp intr quit stop eof

^? ^U ^W ^R ^O ^V ^Z/^Y ^C ^\ ^S/^Q ^D

Or type

stty -a

If your Unix version derives from System III or System V (this include also Linux) to obtain a similar list of control-key.

History Expansion

In order to see the list of the history c-shell command you can type the following command:

fc -l

Command	Purpose
! !! !n !-n !string !?string? ^string1^string2	Start a history substitution Refers to the last command Refers to command line n Refers to current command line minus n Refers to the most recent command starting with string Refers to the most recent command containing string. The ending ? is optional. Repeat the last command, replacing string1 with string2

Tabella 1 – event designator

For example, if your last command was the whoami command, you are able to retype it only by typing

!!

It’s also passible to refer to certain words in a previous command by the use of a word designator

Designator

Purpose

0 n ^ $ % x-y * x* x-

The zeroth (first) word in a line The nth word in a line The first argument (the second word) The last argument in a line The word matched by the most recent ?string search A range of words from x to y. –y is synonymous with 0-y. All words but the zeroth (fist). Synonymous with 1-$. If there is only one word on the line, an empty string is returned. Synonymous with x-$ The words from x to the second last word

Tabella 2 – word designator

The word designator follows the event designator, separated by a colon. It’s possible, for example, repeat the previous command without arguments by typing

!!:0

Or it’s possible to repeat the previous command with different arguments

!!:0 arg0 arg1 arg2

For example if you type

!!:0 --version

(in this case the same like !! –-version)

Lat’s say that your last typed command is whoami, in this case, typing the above command it’s like you would have typed

whoami --version

Event designator may also be followed by modifiers. The modifiers follow the word designator, if there is one.

Modifier

Purpose

h r e t p q x s/old/new

Remove a trailing pathname component, leaving the head Removes a trailing suffix of the form .xxx Removes all but the trailing suffix Removes all leading pathname components, leaving the tail Prints the resulting command but doesn’t execute it Quote the substituted word, escaping further substitutions Quote the substituted words, breaking them into words at blanks and newlines Substitutes new for old

Tabella 3 – modifiers

For example if you have just type the following command

grep -Hnr “Fabio Bombardi” .

And you want to retype the above command but to search for “Elena Bombardi” you could type:

!!:s/Fabio/Elena

And you get the following command:

grep -Hnr “Elena Bombardi” .

Setting the X server keyboard map type

If you want to set a different map, let’s say Italian, to your X sever, you could type the following command:

setxkbmap -layout it

Patterns and Pattern Matching

Patterns are strings that can contain wildcard characters.

Operator

Meaning

${variable#pattern} ${variable##pattern} ${variable%pattern} ${variable%%pattern} ${variable/pattern/string} ${variable//pattern/string}

If the pattern matches the beginning of the variable’s value, delete the shortest part that matches and return the rest. If the pattern matches the beginning of the variable’s value, delete the longest part that matches and return the rest. If the pattern matches the end of the variable’s value, delete the shortest part that matches and return the rest. If the pattern matches the end of the variable’s value, delete the longest part that matches and return the rest. The longest match to pattern in variable is replaced by string. In the first from, only the first match is replaced. In the second form, all the matches are replaced. If the pattern begins with a #, it must match at the start of the variable. If it begins with a %, it must match with the end of the variable. If string is null, the matches are deleted. If variable is @ or *, the operation is applied to each positional parameter in turn and the expansion is the result list.

Tabella 4 – Pattern-Matching Operators

If you want some exaples here you are:

$path /home/shadowsheep/testpatterns/hello.shadowsheep.test

$path##/*/ hello.shadowsheep.test

$path#/*/ shadowsheep/testpatterns/hello.shadowsheep.test

$path%.* /home/shadowsheep/testpatterns/hello.shadowsheep

$path%%.* /home/shadowsheep/testpatterns/hello

If you want to substitute all the occurrence in a string you may want to type:

echo ${PATH//:/’\n’}

/home/usr/bin\n/usr/local/bin\n/bin\n

If you want the echo command to interpret the backslashes character ( like \n) so you may want to type:

echo –e ${PATH//:/’\n’}

/home/usr/bin

/usr/local/bin

/bin

Lessons

Lesson 1

If you want to edit a text file without open a text editor you can use the echo command in the following way:

fbombardi@linux:~/workspace/shell_commands> echo -e "a\n\

a\n\

a\n\

mount -n -t nfs -o nolock,rsize=1024,wsize=1024 172.27.30.179:/home/nfs /NetShared\n\

b\n\

b\n\

end" > test.log

In this way you create a text file named test.log whose content is the following:

a

a

a

mount -n -t nfs -o nolock,rsize=1024,wsize=1024 172.27.30.179:/home/nfs /NetShared

b

b

end

Now if you want to type the command

mount -n -t nfs -o nolock,rsize=1024,wsize=1024 172.27.30.179:/mnt/nfs /NetShared

You can use the following shortcut:

fbombardi@linux:~/workspace/shell_commands> more test.log | grep "mount -n" | sed -e s/home/mnt/

Or even better you can use this shortcut:

fbombardi@linux:~/workspace/shell_commands> grep "mount -n" test.log | sed -e s/home/mnt/

I think that this should be useful when you have a command in a very full written file, otherwise I think that you could be faster in typing the command from scratch =).

Lesson 2

Here we are again. This time we’ll go deeper in the world of the I/O redirection. Let’s say we want to get rid of all the boring messages that the command make show us during the compiling and linking time. What we have to do? The answer is simple and is located in the line below:

make > /dev/null 2>&1

In this way all the boring messages will be redirected in the “black-hole” /dev/null.

Let’s see now that we want to decide which kind of messages to display. I’ll give you a gift showing you on of my favorite shell script:

#

# My Make

# Use this program to customize the make output messages

#

# @author Fabio Bombardi

# @location Datasensor

# @version 2.1.0

# @last-updated 2k6.09.24

#

if [ -n $2 ] && [ "$2" = "-silent" ]; then

break;

else

echo -e "\n"

echo -e \#

echo -e \# My Make

echo -e \# Use this program to customize the make output messages

echo -e \#

echo -e \# @author "\t\t" Fabio Bombardi

echo -e \# "\t\t\t\t" fbombardi@datasensor.com

echo -e \# @location "\t\t" Datasensor

echo -e \# @version "\t\t" 2.1.0

echo -e \# @last-updated "\t" 2k6.09.24

echo -e \#

echo -e "\n"

fi

if [ -z $1 ]; then

echo -e "usage: mymake <type-of-massage>\n\n\t<type-of-massage>=\n\t\twarning\n\t\terror\n\t\tall\n\t\tclean\n"

exit -1

fi

case $1 in

warning )

make 2>&1 | grep "warning" | cat ;;

error )

make 2>&1 | grep "Error" | cat ;;

all )

make 2>&1 | grep "*" | cat ;;

clean)

make clean ;;

* )

echo -e "\ndefault selection: warning\n"

$0 wanrning

esac

In the above shell script the line in bold shows us only the messages containing the word “warning” and hiding all the other messages.

Lesson 3

Let’s say now that we want not only to choose between the make output messages but also to give different colors to this messages (e.g. yellow for warnings and red for error).

In this case we may want to make the following changes in the case statement:

case $1 in

warning )

GREP_COLOR="1;33"

make 2>&1 | grep --color=always "warning" | cat ;;

error )

GREP_COLOR="0;31"

make 2>&1 | grep --color=always "Error" | cat ;;

all )

make 2>&1 | grep "*" | cat ;;

clean)

make clean ;;

* )

echo -e "\ndefault selection: warning\n"

$0 warning ;;

esac

The grep command has the option flag –color that along with the variable GREP_COLOR highlights the pattern word with the color set in the variable GREP_COLOR.

With the above changes we should get something like that:

img_processing.cpp:148: warning: right-hand operand of comma has no effect

img_processing.cpp:367: warning: converting to `unsigned char’ from `float’

img_processing.cpp:477: warning: converting to `int’ from `float’

img_processing.cpp:478: warning: converting to `int’ from `float’

Colors

We could choose, of course, among a plethora of color:

Dark gray: 1;30

Blue: 0;34

Light Blue: 1;34

Green: 0;32

Light green: 1;32

Cyan: 0;36

Light cyan: 1;36

Red: 0;31

Light red: 1;31

Purple: 0;35

Light purple: 1;35

Brown: 0;33

Yellow: 1;33

Light gray: 0;37

White: 1;37

Lesson 4

Let’s say now that we want not only to highlight the keywords (warning, error) but the entire lines!

What follows is what we could do to be successful:

case $1 in

warning )

echo -e "\033[1;33m"

make 2>&1 | grep "warning" | cat

echo -e "\033[1;37m" ;;

error )

echo -e "\033[1;31m"

make 2>&1 | grep --color=always "Error" | cat

echo -e "\033[1;37m" ;;

all )

make 2>&1 | grep "*" | cat ;;

clean)

echo -e "\033[1;32m"

make clean

echo -e "\033[1;37m\n" ;;

* )

echo -e "\033[1;35m"

echo -e "\ndefault selection: warning\n"

echo -e "\033[1;37m\n"

$0 warning ;;

esac

And we’ll get the following output:

img_processing.cpp:148: warning: right-hand operand of comma has no effect

img_processing.cpp:367: warning: converting to `unsigned char’ from `float’

img_processing.cpp:477: warning: converting to `int’ from `float’

img_processing.cpp:478: warning: converting to `int’ from `float’