You might have came across the word ‘script’ a lot of times, but what is the meaning of a script? Basically, a script is a program that contains a series of commands to be executed. These commands are executed by an interpreter. Anything you can put into a command line, you can put in a script. And, scripts are great for automating tasks.
If you find yourself repeating some commands frequently, you can, rather you should, create a script for doing it!

  • There are two special scripts attached at the end.

CLI is a command line interface. This user interface enables the user to give commands to interact with the device.

GUI is a graphical user interface. This user interface enables users to interact with devices with the help of graphical icons and visual indicators.

  • CLI gives better control to the user.
  • CLI is a best option for professionals who work on more programming languages.
  • It required less memory as compared to GUI.
  • The speed of the CLI is faster than GUI.

Shell is a UNIX term for an interface between a user and an operating system service. Shell provides users with an interface and accepts human-readable commands into the system and executes those commands which can run automatically and give the program’s output in a shell script.

An Operating is made of many components, but its two prime components are

  • Kernel
  • Shell

Shell Scripting is an open-source computer program designed to be run by the Unix/Linux shell. Shell Scripting is a program to write a series of commands for the shell to execute. It can combine lengthy and repetitive sequences of commands into a single and simple script that can be stored and executed anytime which, reduces programming efforts.

  • /bin/bash and /bin/sh is interactive shell
  • /sbin/nologin shell is non-interactive shell

The prompt, $, which is called the command prompt, is issued by the shell. While the prompt is displayed, you can type a command.

Shell reads your input after you press Enter. It determines the command you want executed by looking at the first word of your input. A word is an unbroken set of characters. Spaces and tabs separate words.

Following is a simple example of the date command, which displays the current date and time −

$ date

You can get the name of your shell prompt, with following command :

$ echo $SHELL

The $ sign stands for a shell variable, echo will return the text whatever you typed in.

#!/bin/bash
echo "My First Script!"

To run it,

$ chmod +x script.sh
$ ./script.sh

You might have noticed in above script that it starts with #!/bin/bash This is called shebang. It is basically to refer the path to the interpreter. There are many interpreters, some of them are: bash, zsh, csh and ksh, etc.

All the best people in life seem to like LINUX. — Steve Wozniak

To use bash: #!/bin/bash
To use zsh: #!/bin/zsh
To use ksh: #!/bin/ksh
To use csh: #!/bin/csh

Why Shebang?
# is often called sharp and ! is called Bang, hence the name sharp bang, but generally people say it shebang instead of sharp bang.

Note* If a script does not contain the shebang, the commands are executed using your shell, so there are chances that the code might run properly, but still, that isn’t the correct way of doing it!

Comments
Comments are started by a # sign, anything after pound sign on that line is ignored.

There are two main shells in Linux:

1. The Bourne Shell: The prompt for this shell is $ and its derivatives are listed below:

  • POSIX shell also is known as sh
  • Korn Shell also knew as sh
  • Bourne Again SHell also knew as bash (most popular)

2. The C shell: The prompt for this shell is %, and its subcategories are:

  • C shell also is known as csh
  • Tops C shell also is known as tcsh

ls : List the contents of a directory.

pwd : Print Working Directory.

cd : Change Directory

cat : Concatenate and Print the contents of file(s)

tac : Concatenate and Print the contents of file(s) in reverese.

rev : Reverse the lines of file(s)

man : Display the Manual Page for a command

cp : Copy file(s) or directory(ies)

mv : Move or Rename file or directory

rm : Remove file(s) or directory(ies)

mkdir : Make directory

rmdir : Remove “an empty” directory

date : Print or Set the system date and time.

touch : Change file(s) timestamps. If the file(s) don’t exist, it will be created.

more : Print file contents one screen at a time.

less : Similar to more , but allows both forward and backward movement in the file.

seq : Print a sequence of numbers between Start and End.

echo : Print line(s) of text to the standard output (screen).

cal : Display the Calendar (of month or entire year)

clear : Clear the terminal screen.

Daily use Examples of Shell scripting by System Admins

  • Monitoring your Linux system.
  • Data backup and creating snapshots.
  • Dumping Oracle or MySQL database for backup.
  • Creating email based alert systems.
  • Find out what processes are eating up your system resources.
  • Find out available and free memory.
  • Find out all logged in users and what they are doing.
  • Find out if all necessary network services are running or not. For example if the web server failed then send an alert to the system administrator via a pager or an email.
  • Find out all failed login attempts, if login attempts are continued repeatedly from the same network IP automatically block all those IPs accessing your network/service via firewall.
  • User administration as per your own security policies.
  • Find out information about local or remote servers.
  • Server configuration.

GREP command

The grep filter searches a file for a particular pattern of characters, and displays all lines that contain that pattern. The pattern that is searched in the file is referred to as the regular expression (grep stands for globally search for regular expression and print out).

$ grep [options] pattern [files]Options Description
-c : This prints only a count of the lines that match a pattern
-h : Display the matched lines, but do not display the filenames.
-i : Ignores, case for matching
-l : Displays list of a filenames only.
-n : Display the matched lines and their line numbers.
-v : This prints out all the lines that do not matches the pattern
-e exp : Specifies expression with this option. Can use multiple times.
-f file : Takes patterns from file, one per line.
-E : Treats pattern as an extended regular expression (ERE)
-w : Match whole word
-o : Print only the matched parts of a matching line,
with each such part on a separate output line.
-A n : Prints searched line and nlines after the result.
-B n :
Prints searched line and n line before the result.
-C n : Prints searched line and n lines after before the result.

To run multiple commands together

Use && Ex: date && cal. We can also use ||.

How to create multiple text files and directories?

To create multiple text file touch file name {} command is used

Ex. suppose we want create 4 files then we type #touch filename{1..4}

To create multiple directory mkdir filename {} command is used

Ex. suppose we want to create 4 directory then we type mkdir filename {1..4}

What is the use of head and tail command?

Head command is used for display started 10 lines

Tail command is used for display started 10 lines

How to find the current shell which you are using?

$echo $SHELL command is used for find current shell.

How to find an available shell in your system?

Cat /etc/shells command is used to find available shells in your system.

How will you pass and access arguments to a script in Linux?

For pass arguments in script “scriptname arg1 arg2 arg3 …”

For access arguments in script can be accessed inside the script as “$1 , $2 .. $n”

What is the significance of $#?

It represents the total number of arguments passed by string.

What is the difference between $* and $@?

$* consider the entire set of positional parameters as a single string but $@ treat each quoted argument as a separate argument.

What are the different types of variables used in Shell Script?

System defined variable : system defined variable created by os itself. These variables are generally defined in capital letters. It can be viewed by the “set” command.

User defined variable : it created by system users. Value of variable can be view by using “echo $variablename” command.

What is the difference between = and ==?

= use for assign value to variable

== use for string comparison

What are Shell Variables?

Variables store data in the form of characters and numbers. Similarly, Shell variables are used to store information and they can by the shell only.

For example, the following creates a shell variable and then prints it:

variable ="Hello"
echo $variable

Below is a small script which will use a variable.

#!/bin/sh
echo "what is your name?"
read name
echo "How do you do, $name?"
read remark
echo "I am $remark too!"

Note : We can assign the output of a command to a variable too Example: LIST=$(ls) , another example: SERVER_NAME=$(hostname)

Some Valid Variable Names:

THIS3VARIABLE=”ABC”
THIS_IS_VARIABLE=”ABC”
thisIsVariable=”ABC”

Some Invalid Variable Names:

4Number=”NUM”
This-Is-Var=”VAR”
# No special character apart from underscore is allowed!

Read-only Variables

Shell provides a way to mark variables as read-only by using the read-only command. After a variable is marked read-only, its value cannot be changed.

For example, the following script generates an error while trying to change the value of NAME −

#!/bin/shNAME="Vimal Daga"
readonly NAME
NAME="Preeti Daga"

The above script will generate the following result −

/bin/sh: NAME: This variable is read only.

Unsetting Variables

Unsetting or deleting a variable directs the shell to remove the variable from the list of variables that it tracks. Once you unset a variable, you cannot access the stored value in the variable.

Following is the syntax to unset a defined variable using the unset command

unset variable_name

The above command unsets the value of a defined variable. Here is a simple example that demonstrates how the command works −

#!/bin/shNAME="VD"
unset NAME
echo $NAME

Special variables

The $ character represents the process ID number, or PID, of the current shell

$echo $$

The above command writes the PID of the current shell.

  • $0 : The filename of the current script.
  • $n : These variables correspond to the arguments with which a script was invoked. Here n is a positive decimal number corresponding to the position of an argument (the first argument is $1, the second argument is $2, and so on).
  • $# : The number of arguments supplied to a script.
  • $* : All the arguments are double quoted. If a script receives two arguments, $* is equivalent to $1 $2.
  • $@ : All the arguments are individually double quoted. If a script receives two arguments, $@ is equivalent to $1 $2.
  • $? : The exit status of the last command executed.
  • $$ : The process number of the current shell. For shell scripts, this is the process ID under which they are executing.
  • $! : The process number of the last background command.

User Input

read command accepts STDIN (Standard Input)

read -p "PROMPT MESSAGE" VARIABLE

Here, “Prompt Message” will be prompted to the screen and whatever user enters next will be stored in VARIABLE

#!/bin/bash
read -p "Please Enter You Name: " NAME
echo "Your Name Is: $NAME"

Shell Arrays

If you are using the bash shell, here is the syntax of array initialization −

array_name=(value1 ... valuen)

After you have set any array variable, you access it as follows −

${array_name[index]}
  • Print all array indexes :
$ echo ${!array[@]}
  • Remove elements from an array with id 3
$ Unset array[3]

Test Operators

Tests are used for decision making. [ condition-to-test-for ] Example: `[ -e /etc/passwd ]

  1. File Test Operations
-d FILE_NAM  # True if FILE_NAM is a directory
-e FILE_NAM # True if FILE_NAM exists
-f FILE_NAM # True if FILE_NAM exists and is a regular file
-r FILE_NAM # True if FILE_NAM is readable
-s FILE_NAM # True if FILE_NAM exists and is not empty
-w FILE_NAM # True if FILE_NAM has write permission
-x FILE_NAM # True if FILE_NAM is executable

2. String Test Operations

-z STRING  # True if STRING is empty
-n STRING # True if STRING is not empty
STRING1 = STRIN2 # True if strings are equal
STRING1 != STRIN2 # True if strings are not equal

3. Relational Operators

var1 -eq var2  # True if var1 is equal to var2
var1 -ne var2 # True if var1 not equal to var2
var1 -lt var2 # True if var1 is less than var2
var1 -le var2 # True if var1 is less than or equal to var2
var1 -gt var2 # True if var1 is greater than var2
var1 -ge var2 # True if var1 is greater than or equal to var2

Decision Making

Just like any script, shell scripts can make decisions based on conditions. We can use If-Else (or, If-Elif-Else) and case statements for decision making.

  1. The if statement
if [ condition-is-true ]
then
command 1
command 2
...
...
command N
fi

The if-elif Ladder

if [ condition-is-true ]
then
command 1
elif [ condition-is-true ]
then
command 2
elif [ condition-is-true ]
then
command 3
else
command 4
fi

2. Case Statements
The case statements are an alternative for if statements which are a little easier to read than complex if elif ladder. If you find yourself using an if statement to compare the same variable against some different/discrete values, you can use a case statements instead of if-elif ladder.

Syntax:

case "$VAR" in
pattern_1)
# commands when $VAR matches pattern 1
;;
pattern_2)
# commands when $VAR matches pattern 2
;;
*)
# This will run if $VAR doesnt match any of the given patterns
;;
esac

Example:

#!/bin/bash
read -p "Enter the answer in Y/N: " ANSWER
case "$ANSWER" in
[yY] | [yY][eE][sS])
echo "The Answer is Yes :)"
;;
[nN] | [nN][oO])
echo "The Answer is No :("
;;
*)
echo "Invalid Answer :/"
;;
esac

Iterative Statements: Loops

Loops can execute a block of code a number of times and are basically used for performing iterations.

1. The For Loop

for VARIABLE_NAME in ITEM_1 ITEM_N
do
command 1
command 2
...
...
command N
done

For example:

#!/bin/bash
COLORS="red green blue"
for COLOR in $COLORS
do
echo "The Color is: ${COLOR}"
done

Another way of using for loop:

for (( VAR=1;VAR<N;VAR++ ))
do
command 1
command 2
...
...
command N
done

Here’s an example of how you can make a script to rename each file with .txt format to new-<old-name>.txt

#!/bin/bash
FILES=$(ls *txt)
NEW="new"
for FILE in $FILES
do
echo "Renaming $FILE to new-$FILE"
mv $FILE $NEW-$FILE
done

2. The While Loop

While loop repeats a series of commands for as long as the given condition holds true.

while [ CONNDITION_IS_TRUE ]
do
# Commands will change he entry condition
command 1
command 2
...
...
command N
done

Condition can be any test or command. If the test/command retrns a 0 exit status, it means that the condition is true and commands will be executed. If the command returns a non-zero exit status, it the loop will stop its iterations. If the condition is false initially, then the commands inside the loop will never get executed.

Example: Reading a file line by line

#!/bin/bash
LINE=1
while read CURRENT_LINE
do
echo "${LINE}: $CURRENT_LINE"
((LINE++))
done < /etc/passwd
# This script loops through the file /etc/passwd line by line

Nesting while Loops

It is possible to use a while loop as part of the body of another while loop.

Syntax

while command1 ; # this is loop1, the outer loop
do
Statement(s) to be executed if command1 is true
while command2 ; # this is loop2, the inner loop
do
Statement(s) to be executed if command2 is true
done
Statement(s) to be executed if command1 is true
done

Example

Here is a simple example of loop nesting. Let’s add another countdown loop inside the loop that you used to count to nine −

#!/bin/sha=0
while [ "$a" -lt 10 ] # this is loop1
do
b="$a"
while [ "$b" -ge 0 ] # this is loop2
do
echo -n "$b "
b=`expr $b - 1`
done
echo
a=`expr $a + 1`
done

This will produce the following result. It is important to note how echo -n works here. Here -n option lets echo avoid printing a new line character.

0
1 0
2 1 0
3 2 1 0
4 3 2 1 0
5 4 3 2 1 0
6 5 4 3 2 1 0
7 6 5 4 3 2 1 0
8 7 6 5 4 3 2 1 0
9 8 7 6 5 4 3 2 1 0

Note:
continue statement is used to take the flow of control to the next iteration. Any statement after continue statement is hit will not be executed and the flow of control will shift to the next iteration.
break statement can be used in while loops (or other loops like for loop) to end the loops. Once the break statement is hit, the flow of control will move outside the loop.

Positional Parameters

Some arguments or parameters can be passed when we call the script.
For Example: $ ./script.sh param1 param2 param3 param4

All the parameters will be stored in various variables:

$0 -- "script.sh"
$1 -- "param1"
$2 -- "param2"
$3 -- "param3"
$4 -- "param4"
$@ -- array of all positional parameters

These variables can be accessed anywhere in the script just like any other global variable.

Exit Statuses

Every command returns an exit status, also called the return code which ranges from 0 to 255. Exit status are used for error checking.

  • 0 means success
  • Any code other than 0 means an error condition.

To find out what an exit status for a command means, one can look for the documentations or manual using man or info command.
$? contains the return code of previously executed command.

Example: A script which will print Reachable/Unreachable Host/server using ping command.

#!/bin/bash
HOST="google.com"
ping -c 1 $HOST # -c is used for count, it will send the request, number of times mentioned
RETURN_CODE=$?
if [ "$RETURN_CODE" -eq "0" ]
then
echo "$HOST reachable"
else
echo "$HOST unreachable"
fi

Custom Exit Statuses
Exit command is used to explicitly define the return code. If we do not define the exit status of the shell script explicitly, then by default the exit status of the last command executed is taken as the exit status of the script. We can use exit command anywhere in the script, and as soon as exit command is encountered, the shell script will stop executing.

exit 0
exit 1
exit 2
...
...
exit 255

Functions

A set of instructions which can be reused/called any time from the main program whenever the need for those instructions arrives. Consider a situation that you have a particular module that does a particular job, now let’s suppose that job has to be done 20 (say) times in the main program (for example calculating maximum number in an array of numbers). Now, if you write the code for that 20 times, your code would become very large, however, if we write a function for that, and call that function whenever it is required, the code would remain short, easy to read and much more modularized.

Using functions reduces overall length/size of script. Moreover, functions make program easier to maintain, because they make the code divided into modules making particular place to edit and troubleshoot available in case of bugs/errors.

Note:
- Whenever you find yourself repeating a set of instructions, make a function for that. (A function must be defined before use).
- It is a good practice to define all your functions at the top before starting the main program or main instructions.

Syntax:

function function_name() {
command 1
command 2
command 3
...
...
command N
}

To call a function: simply write it’s name on a line in the script.

#!/bin/bash
function myFunc () {
echo "Shell Scripting Is Fun!"
}
myFunc

Pass Parameters to a Function

You can define a function that will accept parameters while calling the function. These parameters would be represented by $1, $2 and so on.

Following is an example where we pass two parameters Zara and Ali and then we capture and print these parameters in the function.

#!/bin/sh# Define your function here
Hello () {
echo "Hello World $1 $2"
}
# Invoke your function
Hello vimal daga

Upon execution, you will receive the following result −

$./test.sh
Hello World vimal daga

To remove the definition of a function from the shell, use the unset command with the -f option. This command is also used to remove the definition of a variable to the shell.

$ unset -f function_name

Positional Parameters In Functions

Just like a shell script, functions can also accept parameters. The first parameter is stored in $1 The second parameter is stored in $2 and so on. $@ contains all the parameters.
Note: $0 is still the name of script itself, not the name of function.
To provide parameters, just write them after the function name with a white space in between. myFunc param1 param2 param3 ...

Scope Of Variables

Global Scope: All variables have, by default, global scope. Having a global scope means that the value of that variable can be accessed from anywhere in the script. Variables must be defined before it is used.
Local Scope: Local variables can be accessed only from within the function. Local variables are created using local keyword and only functions can have local variables. It is a good practice to keep variables inside a function local.

Wildcards

A character or a string patterns that is used to match file and directory names is/are called wildcard(s). The process used to expand wildcard pattern into a list of files and/or directories (or basically paths) is called Globbing.
Wild Cards can be used with most of the commands that require file/dir path as argument. (Example ls,rm,cp etc).

Some Commonly Used Wildcards

* = Matches zero or more characters
Example:*.txt hello.* great*.md

? = matches exactly one character
Example: ?.md Hello?

[ ] = A character class
This wildcard is used to match any of the characters included between the square brackets (Matching exactly one character).
Example: He[loym], [AIEOU]

[!] = maches characters not included within brackets
It matches exactly one character.
Example: To match a consonant: [!aeiou]

Predefined named character classes

  • [[:alpha:]]
  • [[:alnum:]]
  • [[:space:]]
  • [[:upper:]]]
  • [[:lower:]]
  • [[:digit:]]

Matching wildcard characters: In case we have to match wildcard characters themselves like *, or ?, we can go with escape character - \
Example: *\? –> will match all files that end with a question mark.

Debugging

A bug is an error in a computer program/software that causes it to produce an unexpected or an incorrect result. Most of the bugs are cased by errors in the code and it’s design. To fix an error, try to reach to the root of that unexpected behaviour.

The bash shell provides some options that can help you in debugging your script. You can use these options by updating first line of the script.

Some Options:

1. -x option
It prints commands and arguments as they execute. It is called print debugging, tracing or an x-trace. We can use it by modifying the first line #!/bin/bash -x

2. -e option
It stands for “Exit on error”. This will cause your script to exit immediately if a command exits with a non-zero exit status.

3. -v option
It prints shell commands/input lines as they are read.

Note* These options can be combined, and more than one options can be used at a time!

#!/bin/bash-xe
#!/bin/bash-ex
#!/bin/bash-x-e
#!/bin/bash-e-x

Shell Input/Output Redirections

The output from a command normally intended for standard output can be easily diverted to a file instead. This capability is known as output redirection.

If the notation > file is appended to any command that normally writes its output to standard output, the output of that command will be written to file instead of your terminal.

Check the following who command which redirects the complete output of the command in the users file.

$ who > users

Notice that no output appears at the terminal. This is because the output has been redirected from the default standard output device (the terminal) into the specified file. You can check the users file for the complete content −

$ cat users

If a command has its output redirected to a file and the file already contains some data, that data will be lost. Consider the following example −

$ echo line 1 > users
$ cat users
line 1

You can use >> operator to append the output in an existing file as follows −

$ echo line 2 >> users
$ cat users
line 1
line 2

Just as the output of a command can be redirected to a file, so can the input of a command be redirected from a file. As the greater-than character > is used for output redirection, the less-than character < is used to redirect the input of a command.

The commands that normally take their input from the standard input can have their input redirected from a file in this manner. For example, to count the number of lines in the file users generated above, you can execute the command as follows −

$ wc -l users
2 users

Upon execution, you will receive the following output. You can count the number of lines in the file by redirecting the standard input of the wc command from the file users

$ wc -l < users
2

Note that there is a difference in the output produced by the two forms of the wc command. In the first case, the name of the file users is listed with the line count; in the second case, it is not.

In the first case, wc knows that it is reading its input from the file users. In the second case, it only knows that it is reading its input from standard input so it does not display file name.

$ awk ‘{print $5}’
$ grep –c “ABC” filename
$ awk '{print $3}' $1

/etc/shadow file has all the users listed.

$ awk –F ‘:’ ‘{print $1}’ /etc/shadow|uniq –u
$ sed ’50i\ABCD’ filename
$ cut -f 6 -d “:” /etc/passwd

f represent field, d represent field separation, etc/passwd is a one file.

AWK Command

The awk command is used for text processing in Linux. Although, the sed command is also used for text processing, but it has some limitations, so the awk command becomes a handy option for text processing. It provides powerful control to the data.

The Awk is a powerful scripting language used for text scripting. It searches and replaces the texts and sorts, validates, and indexes the database.

It is one of the most widely used tools for the programmer, as they write the scaled-down effective program in the form of a statement to define the text patterns and designs.

It acts as a filter in Linux. It is also referred as gawk (GNU awk) in Linux.

This command is named by using the first letter of the name of three people who wrote the original version of this command in 1977. Their names are Alfred Aho, Peter Weinberger, and Brian Kernighan and they were from AT & T Bell Laboratories.

Various features of the Awk command are as follows:

  • It scans a file line by line.
  • It splits a file into multiple fields.
  • It compares the input text or a segment of a text file.
  • It performs various actions on a file like searching a specified text and more.
  • It formats the output lines.
  • It performs arithmetic and string operations.
  • It applies the conditions and loops on output.
  • It transforms the files and data on a specified structure.
  • It produces the format reports.

Syntax:

The Awk command is used as follows:

$ awk options ‘selection _criteria {action }’ input-file > output-file

The options can be:

  • -f program files: It reads the source code of the script written on the awk command
  • -F fs: It is used as the input field separator.
$ awk --help

On executing this code, you get the following result −

Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options : GNU long options: (standard)
-f progfile --file=progfile
-F fs --field-separator=fs
-v var=val --assign=var=val
Short options : GNU long options: (extensions)
-b --characters-as-bytes
-c --traditional
-C --copyright
-d[file] --dump-variables[=file]
-e 'program-text' --source='program-text'
-E file --exec=file
-g --gen-pot
-h --help
-L [fatal] --lint[=fatal]
-n --non-decimal-data
-N --use-lc-numeric
-O --optimize
-p[file] --profile[=file]
-P --posix
-r --re-interval
-S --sandbox
-t --lint-old
-V --version

Built In Variables In Awk

Awk’s built-in variables include the field variables — $1, $2, $3, and so on ($0 is the entire line) — that break a line of text into individual words or pieces called fields.

NR: NR command keeps a current count of the number of input records. Remember that records are usually lines. Awk command performs the pattern/action statements once for each record in a file.

NF: NF command keeps a count of the number of fields within the current input record.

FS: FS command contains the field separator character which is used to divide fields on the input line. The default is “white space”, meaning space and tab characters. FS can be reassigned to another character (typically in BEGIN) to change the field separator.

RS: RS command stores the current record separator character. Since, by default, an input line is the input record, the default record separator character is a newline.

OFS: OFS command stores the output field separator, which separates the fields when Awk prints them. The default is a blank space. Whenever print has several parameters separated with commas, it will print the value of OFS in between each parameter.

ORS: ORS command stores the output record separator, which separates the output lines when Awk prints them. The default is a newline character. print automatically outputs the contents of ORS at the end of whatever it is given to print.

  • To pass value on run time we have to make commands in non-interactive way. Ex: passwd required password. So, echo <password> | passwd bob * awk ‘{ print NR }’ my.txt → Print record no.
  • awk ‘{ print $(NF-1) }’ my.txt → Very useful when data is unstructured. It retrieves second last value from each line.
  • awk has END option to print output from last.
  • awk has BEGIN keyword where we can write further script eg: for loop or etc.

TRANSPOSE Function using AWK

Write the command for getting the total number of false client hits.

Total no of false client hit : aws ‘$9==404 { print $0’ access_log | sort | uniq -c | wc -l

How to get filter dd:mm:yy from date output?

filter dd:mm:yyyy using date +%e : %b : %G

How to sort and count the line log file?

awk ‘{ print $1 }’ access_log | sort | uniq -c | sort -n -k1

Watch Command

watch is used to run any arbitrary command at regular intervals and displays the output of the command on the terminal window.

It is useful when you have to execute a command repeatedly and watch the command output change over time. For example, you can use the watch command to monitor the system uptime or disk usage.

The watch utility is a part of the procps (or procps-ng) package which is pre-installed on nearly all Linux distributions.

The syntax for the watch command is as follows:

watch [OPTIONS] COMMAND

Run Command Every 2 Seconds

By default, the watch command runs after every 2 seconds and shows output on the terminal. It keeps running until interrupted by the user.

watch date

Run Command Every N Seconds

Watch command update interval can be overridden. It means you can change the update interval from 2 seconds to your desired interval as well.

watch -n <interval-in-seconds> <command>

Highlighting The Differences Between Updates

You can also highlight the differences between previous output and current output. So instead of reading the whole output, you can keep an eye on the changes.

watch -d <command>

Hide Header in output

Watch command output shows a header containing time interval, command, system name, and system date. However, if you do not want to see this portion, you can hide it as well.

watch -t <command>

Exit On Change

By default, the watch command keeps running until it is interrupted manually by the user. However, You can make it exit if it identifies a change in output as well.

watch -g <command>watch -n 10 -g date

PATH in Linux

The PATH is an environment variable. It is a colon delimited list of directories that your shell searches through when you enter a command. All executables are kept in different directories on the Linux and Unix like operating systems.

Linux sed Command | Linux Stream Editor

Linux ‘sed’ command stands for stream editor. It is used to edit streams (files) using regular expressions. But this editing is not permanent. It remains only in display, but in actual, file content remains the same.

Primarily, it is used for text substitution; additionally, it can be used for other text manipulation operations like insert, delete, search, and more. The sed command allows us to edit files without opening them. Regular expression support makes it a more powerful text manipulation tool.

Syntax: sed [OPTION]… {script-only-if-no-other-script} [input-file]…

It might sound crazy, but the Linux sed command is a text editor without an interface. You can use it from the command line to manipulate text in files and streams.

The Power of sed

The sed command is a bit like chess: it takes an hour to learn the basics and a lifetime to master them (or, at least a lot of practice). We’ll show you a selection of opening gambits in each of the main categories of sed functionality.

sed is a stream editor that works on piped input or files of text. It doesn’t have an interactive text editor interface, however. Rather, you provide instructions for it to follow as it works through the text. This all works in Bash and other command-line shells.

With sed you can do all of the following:

  • Select text
  • Substitute text
  • Add lines to text
  • Delete lines from text
  • Modify (or preserve) an original file

Ex : sed -i ’63 s/bash/sh/g’ /etc/passwd

g is for global if we use in a function. 63 is the record no which has to be updated where found bash replace with sh. -i option updates the file.

Projects

1. User-Script

Features:

* useradd: Add new users

* userdel: Delete existing user

* userlock & backup: Lock the account of user and create backup of the user workspace

2. DoS-Attack-Detection-Prevention

Features:

* Analyze httpd server logs

* Detect whether anyone is trying to perform Dos attack, if yes then it can take appropriate action against it.

ARTH | IIEC Rise | GoogleCloudReady Facilitator | Proficient Photographer | Quantum Computing Enthusiast