Different Types of Shells to Declare

#!/usr/bin/sh OR #!/bin/sh Declares a Bourne shell
#!/usr/bin/ksh OR #!/bin/ksh Declares a Korn shell
#!/usr/bin/csh OR #!/bin/csh Declares a C shell
#!/usr/bin/bash OR #!/bin/bash Declares a Bourne-Again shell

Linux Commands

rename

As you know, the mv command renames files. For example,

$ mv oldname newname
renames the file oldname to newname. However, what if you don’t know the filenames yet? The rename command comes in really handy here.

rename .log .log.‘date +%F-%H:%M:%S‘ *

replaces all files with the extension .log with .log.. So sqlnet.log becomes sqlnet.log.2006-09-12-23:26:28.

alias and unalias

Suppose you want to check the ORACLE_SID environment variable set in your shell. You will have to type:

echo $ORACLE_HOME

As a DBA or a developer, you frequently use this command and will quickly become tired of typing the entire 16 characters. Is there is a simpler way?

There is: the alias command. With this approach you can create a short alias, such as “os”, to represent the entire command:

alias os=’echo $ORACLE_HOME’

Now whenever you want to check the ORACLE_SID, you just type “os” (without the quotes) and Linux executes the aliased command.

However, if you log out and log back in, the alias is gone and you have to enter the alias command again. To eliminate this step, all you have to do is to put the command in your shell’s profile file. For bash, the file is .bash_profile (note the period before the file name, that’s part of the file’s name) in your home directory. For bourne and korn shells, it’s .profile, and for c-shell, .chsrc.

You can create an alias in any name. For instance, I always create an alias for the command rm as rm -i, which makes the rm command interactive.

alias rm=’rm -i’

Whenever I issue an rm command, Linux prompts me for confirmation, and unless I provide “y”, it doesn’t remove the file—thus I am protected form accidentally removing an important file. I use the same for mv (for moving the file to a new name), which prevents accidental overwriting of existing files, and cp (for copying the file).

Here is a list of some very useful aliases I like to define:

alias bdump=’cd $ORACLE_BASE/admin/$ORACLE_SID/bdump’
alias l=’ls -d .* –color=tty’
alias ll=’ls -l –color=tty’
alias mv=’mv -i’
alias oh=’cd $ORACLE_HOME’
alias os=’echo $ORACLE_SID’
alias rm=’rm -i’
alias tns=’cd $ORACLE_HOME/network/admin’

Painless Changes to Owner, Group, and Permissions

# ls -l
total 8
-rw-r–r– 1 test users 70 Aug 4 04:02 file1
-rwxr-xr-x 1 oracle dba 132 Aug 4 04:02 file2
-rwxr-xr-x 1 oracle dba 132 Aug 4 04:02 file3
-rwxr-xr-x 1 oracle dba 132 Aug 4 04:02 file4
-rwxr-xr-x 1 oracle dba 132 Aug 4 04:02 file5
-rwxr-xr-x 1 oracle dba 132 Aug 4 04:02 file6

and you need to change the permissions of all the files to match those of file1. Sure, you could issue chmod 644 * to make that change—but what if you are writing a script to do that, and you don’t know the permissions beforehand? Or, perhaps you are making several permission changes and based on many different files and you find it infeasible to go though the permissions of each of those and modify accordingly.

A better approach is to make the permissions similar to those of another file. This command makes the permissions of file2 the same as file1:

chmod –reference file1 file2

Now if you check:

# ls -l file[12]
total 8
-rw-r–r– 1 test users 70 Aug 4 04:02 file1
-rw-r–r– 1 oracle dba 132 Aug 4 04:02 file2

The file2 permissions were changed exactly as in file1. You didn’t need to get the permissions of file1 first.

You can also use the same trick in group membership in files. To make the group of file2 the same as file1, you would issue:

# chgrp –reference file1 file2
# ls -l file[12]
-rw-r–r– 1 test users 70 Aug 4 04:02 file1
-rw-r–r– 1 oracle users 132 Aug 4 04:02 file2

Of course, what works for changing groups will work for owner as well. Here is how you can use the same trick for an ownership change. If permissions are like this:

# ls -l file[12]
-rw-r–r– 1 test users 70 Aug 4 04:02 file1
-rw-r–r– 1 oracle dba 132 Aug 4 04:02 file2

You can change the ownership like this:

# chown –reference file1 file2
# ls -l file[12]
-rw-r–r– 1 test users 70 Aug 4 04:02 file1
-rw-r–r– 1 test users 132 Aug 4 04:02 file2

Note that the group as well as the owner have changed.

UNIX COMMANDS- AIX Command Crib Sheet

oslevel Returns operating system level

whence (program) Returns full path of program
whereis (program) Returms full path of program

what (program) Displays identifying info from the executable
like version number, when compiled.

lslpp -L all list all installed software
lslpp -L (program set name) Check if software installed
lslpp -f Lists filesets vs packages
lslpp -ha Lists installation history of filesets

instfix -ik (fix number eg IX66617) Checks id fix is installed
instfix -ik 4330-02_AIX_ML

compress -c file.txt > file.Z Create a compressed file.

uuencode (infile) (extract-file-name) > (output file)
Converts a binary file to an ASCII file for transfer by modem or email

uudecode (encoded file)
Extracts a binary file from encoded file and calls it the extract-file-name

examples :-

uuencode maymap maymap > maymap.enc
uuencode maymap.enc

od -c /tmp Displays contents of the /tmp directory file
ls -i Lists files with their inode numbers
echo * Lists files, can be used if ls is corrupt/missing

alog -o -t boot View the boot log

chtz (timezone eg GMT0BST) Changes the timezone in /etc/environment file
chlang (language eg En_GB) Changes the language in /etc/environment file

ar -v -t (archive file) List contents of an archive
ar -v -x (archive file) Extracts the archive
ar -v -t /usr/lib/libC-r.a Lists contents of the libC_r.a library

find /source -print | cpio -pdm /target
Copying directories using cpio, creates /target/source directory.

dump -nTv (binary executable) Displays the contents of an executable file

dump -c Displays string information
dump -o Displays object file headers
dump -l Displays line numbers
dump -s Displays the text section

snap -ao /dev/rmt0 Create a snapshot onto tape
snap -ad (directory) Create a snapshot into a named directory other
than the default (/tmp/ibmsupt)

/usr/dt/bin/dtconfig -d Disables desktop logins
/usr/dt/bin/dtconfig -e Enables desktop logins
/var/dt/Xpid PID of the dtlogin process

——————————————————————————–
TERMINALS
——————————————————————————–

tty Displays what the tty/pty number of the terminal is.

termdef reports the termtype setup in smit for the tty port
that termdef is run on.

chdev -l (device eg tty1) -a term=vt100 Sets tty to a vt100 terminal type

penable tty0 adds getty line into /etc/inittab for tty0 and starts getty
pdisable tty0 disables the getty line and disables getty

penable / pdisable -a option is for all

stty erase ^? Set backspace key for vt100 terminals
stty erase ^H Set backspace key for wyse50 terminals

lscons Displays the console device
chcons -a login=enable (device eg /dev/tty1) Changes the console device

Create ttys on ports 0 to 7 on adapter sa2 :-

for i in 0 1 2 3 4 5 6 7
do
mkdev -c tty1 -t tty -s rs232 -p sa2 -w$i -a login=enable -a term=vt100
done

portmir -t /dev/tty0 Mirror current terminal onto /dev/tty0
portmir -o Turns off port mirroring

——————————————————————————–
NETWORK
——————————————————————————–

host (ip or hostname) Resolves a hostname / ip address

hostname Displays hostname
hostname (hostname) Sets the hostname until next reboot

chdev -l (device name) -a hostname=(hostname) Changes hostname permanently
chdev -l inet0 -a hostname=thomas

ifconfig (device name) Displays network card settings
ifconfig (device name) up Turns on network card
ifconfig (device name) down Turns off network card
ifconfig (device name) detach Removes the network card from the
network interface list
ifconfig en0 inet 194.35.52.1 netmask 255.255.255.0 up

ifconfig lo0 alias 195.60.60.1 Create alias ip address for loopback

route (add/delete) (-net/-host) (destination) (gateway)
Adds or deletes routes to other networks or hosts, does not update
the ODM database and will be lost at reboot.
route add -net 194.60.89.0 194.60.90.4

lsattr -EHl inet0 Displays routes set in ODM and hostname
odmget -q “name=inet0” CuAt Displays routes set in ODM and hostname

refresh -s inetd Refresh inetd after changes to inetd.conf
kill -1 (inetd PID) Refresh inetd after changes to inted.conf

netstat -i Displays interface statistics
entstat -d (ethernet adapter eg en0) Displays ethernet statistics

arp -a Displays ip to mac address table from arp cache

no -a Displays network options use -o to set individual options or
-d to set individual options to default.
no -o option=value (this value is reset at reboot)
no -o “ipforwarding=1”

traceroute (name or ipaddress) Displays all the hops from source to
destination supplied.

ping -R (name or ipaddress) Same as traceroute except repeats.

——————————————————————————–
N.F.S.
——————————————————————————–

exportfs Lists all exported filesystems

exportfs -a Exports all fs’s in /etc/exports file

exportfs -u (filesystem) Un-exports a filesystem

mknfs Configures and starts NFS services

rmnfs Stops and un-configures NFS services

mknfsexp -d /directory Creates an NFS export directory

mknfsmnt Creates an NFS mount directory

mount hostname:/filesystem /mount-point Mount an NFS filesystem

nfso -a Display NFS Options
nfso -o option=value Set an NFS Option
nfso -o nfs_use_reserved_port=1

——————————————————————————–
BACKUPS
——————————————————————————–

MKSYSB
——

mkszfile -f Creates /image.data file (4.x onwards)
mkszfile -X Creates /fs.size file (3.x)

mksysb (device eg /dev/rmt0)

CPIO ARCHIVE
————

find (filesystem) -print | cpio -ocv > (filename or device)

eg find ./usr/ -print | cpio -ocv > /dev/rmt0

CPIO RESTORE
————

cpio -ict < (filename or device) | more Lists archive
cpio -icdv < (filename or device)
cpio -icdv < (filename or device) ("files or directories to restore")

eg cpio -icdv < /dev/rmt0 "tcpip/*" Restore directory and contents
cpio -icdv new.filename
Strips out ^M characters from ascii files that have been transferred as binary.
To enter crontrol characters type ctrl v then ctrl ? where ? is whatever
ctrl character you need.

——————————————————————————–
DEVICES
——————————————————————————–

lscfg lists all installed devices
lscfg -v lists all installed devices in detail
lscfg -vl (device name) lists device details

bootinfo -b reports last device the system booted from
bootinfo -k reports keyswitch position
1=secure, 2=service, 3=normal

bootinfo -r reports amount of memory (/ by 1024)
bootinfo -s (disk device) reports size of disk drive
bootinfo -T reports type of machine ie rspc

lsattr -El sys0 -a realmem reports amount of useable memory

mknod (device) c (major no) (minor no) Creates a /dev/ device file.
mknod /dev/null1 c 2 3

lsdev -C lists all customised devices ie installed
lsdev -P lists all pre-defined devices ie supported
lsdev -(C or P) -c (class) -t (type) -s (subtype)

chdev -l (device) -a (attribute)=(new value) Change a device attribute
chdev -l sys0 -a maxuproc=80

lsattr -EH -l (device) -D Lists the defaults in the pre-defined db
lsattr -EH -l sys0 -a modelname

rmdev -l (device) Change device state from available to defined
rmdev -l (device) -d Delete the device
rmdev -l (device) -SR S stops device, R unconfigures child devices

lsresource -l (device) Displays bus resource attributes of a device.

Power Management (PCI machines)
——————————-

pmctrl -a Displays the Power Management state

rmdev -l pmc0 Unconfigure Power Management
mkdev -l pmc0 Configure Power Management

——————————————————————————–
TAPE DRIVES
——————————————————————————–

rmt0.x where x = A + B + C

A = density 0 = high 4 = low
B = retension 0 = no 2 = yes
C = rewind 0 = no 1 = yes

tctl -f (tape device) fsf (No) Skips forward (No) tape markers
tctl -f (tape device) bsf (No) Skips back (No) tape markers
tctl -f (tape device) rewind Rewind the tape
tctl -f (tape device) offline Eject the tape
tctl -f (tape device) status Show status of tape drive

chdev -l rmt0 -a block_size=512 changes block size to 512 bytes
(4mm = 1024, 8mm = variable but
1024 recommended)

bootinfo -e answer of 1 = machine can boot from a tape drive
answer of 0 = machine CANNOT boot from tape drive

diag -c -d (tape device) Hardware reset a tape drive.

tapechk (No of files) Checks Number of files on tape.

(filename) Output crontab entrys to a file
crontab (filename) Enter a crontab from a file
crontab -r Removes all crontab entrys
crontab -v Displays crontab submission time.

/var/adm/cron/cron.allow File containing users allowed crontab use.
/var/adm/cron/cron.deny File containing users denied crontab use.
/var/adm/cron/crontab Directory containing users crontab entries.

at (now + 2 minutes, 13:05, etc) {return} Schedule a job using at
Command or schell script {return}
{CTRL D}

at -l
atq Lists out jobs scheduled to run via at command

at -r (at job No)
atrm (at job No) Removes an at job scheduled to run.

/var/adm/cron/at.allow File containing users allowed at use.
/var/adm/cron/at.deny File containing users denied at use.
/var/adm/cron/atjobs Directory containing users at entries.

——————————————————————————–
SECURITY
——————————————————————————–

groups Lists out the groups that the user is a member of
setgroups Shows user and process groups

chmod abcd (filename) Changes files/directory permissions

Where a is (4 SUID) + (2 SGID) + (1 SVTX)
b is (4 read) + (2 write) + (1 execute) permissions for owner
c is (4 read) + (2 write) + (1 execute) permissions for group
d is (4 read) + (2 write) + (1 execute) permissions for others

-rwxrwxrwx -rwxrwxrwx -rwxrwxrwx
||| ||| |||
– – –
| | |
Owner Group Others

-rwSrwxrwx = SUID -rwxrwSrwx = SGID drwxrwxrwt = SVTX

chown (new owner) (filename) Changes file/directory owners
chgrp (new group) (filename) Changes file/directory groups

chown (new owner).(new group) (filename) Do both !!!

umask Displays umask settings
umask abc Changes users umask settings

where ( 7 – a = new file read permissions)
( 7 – b = new file write permissions)
( 7 – c = new file execute permissions)

eg umask 022 = new file permissions of 755 = read write and execute for owner
read —– and execute for group
read —– and execute for other

mrgpwd > file.txt Creates a standard password file in file.txt

passwd Change current user password

pwdadm (username) Change a users password

pwdck -t ALL Verifies the correctness of local authentication

lsgroup ALL Lists all groups on the system
mkgroup (new group) Creates a group
chgroup (attribute) (group) Change a group attribute
rmgroup (group) Removes a group

——————————————————————————–
USERS
——————————————————————————–

passwd -f Change current users gecos (user description)
passwd -s Change current users shell

chfn (username) Changes users gecos
chsh (username) (shell) Changes users shell

env Displays values of environment variables
printenv

id Displays current user’s uid and gid details
id (user) Displays user uid and gid details

whoami Displays current user details
who am i (or who -m)

who Displays details of all users currently logged in.
w
who -b Displays system reboot time

uptime Displays number of users logged in, time since last
reboot, and the machine load averages.

lslicense Displays number of current user licensese
chlicense -u (number) Changes the number of user licenses

lsuser ALL Lists all users details
lsuser (username) Lists details for user
lsuser -a(attribute) (username or ALL) Lists user attributes
lsuser -a home ALL

mkuser -a(attributes) (newuser) Add a new user

chuser (attributes) (user) Change a user
chuser login=false (user) Lock a user account

rmuser -p (user) Removes a user and all entries in security files

usrck -t ALL Checks all the user entires are okay.

fuser -u (logical volume) Displays processes using the files in that LV

lsattr -D -l sys0 -a maxuproc Displays max number of processes per user
chdev -l sys0 -a maxuproc=(number) Changes max number of processes per user

——————————————————————————–
REMOTE USERS
——————————————————————————–

ruser -a -f (user) Adds entry into /etc/ftpusers file
ruser -a -p (host) Adds entry into /etc/host.lpd file
ruser -a -r (host) Adds entry into /etc/hosts.equiv file

ruser -d -f (user) Deletes entry in /etc/ftpusers file
ruser -d -p (host) Deletes entry in /etc/host.lpd file
ruser -d -r (host) Deletes entry in /etc/hosts.equiv file

ruser -s -F Shows all entries in /etc/ftpusers file
ruser -s -P Shows all entries in /etc/host.lpd file
ruser -s -R Shows all entries in /etc/hosts.equiv file

ruser -X -F Deletes all entries in /etc/ftpusers file
ruser -X -P Deletes all entries in /etc/host.lpd file
ruser -X -R Deletes all entries in /etc/hosts.equiv file

——————————————————————————–
INITTAB
——————————————————————————–

telinit S Switches to single user mode.
telinit 2 Switches to multi user mode.
telinit q Re-examines /etc/inittab

lsitab -a Lists all entries in inittab
lsitab (ident eg tty1) Lists the tty1 entry in inittab

mkitab (“details”) Creates a new inittab entry
chitab (“details”) Ammends an existing inittab entry

rmitab (ident eg tty1) Removes an inittab entry.

chitab “tty1:2:respawn:/usr/bin/getty /dev/tty1”

——————————————————————————–
ODM
——————————————————————————–

odmget -q “name=lp1” CuDv |more Gets lp1 info from pre-defined database.

odmget -q “name-lp1” CuAt |more Gets lp1 info from customised database.

odmdelete -o CuAt -q “name=lp1” Deletes lp1 info from customised db.

odmget -q “name=lp1” CuAt > lp1.CuAt Export ODM info to text file.
odmadd < lp1.CuAt Import ODM info from text file.

——————————————————————————–
ERROR LOGGING
——————————————————————————–

/usr/lib/errdemon -l Displays errorlog attributes.
/usr/lib/errdemon Starts error logging.
/usr/lib/errstop Stops error logging.

errpt Displays summary errorlog report.
errpt -a Displays detailed errorlog report.
errpt -j (identifier) Displays singe errorlog report.

Note : errorlog classes are H=Hardware S=Software O=Information V=Undetermined

errclear (days) Deletes all error classes in the errorlog.
errclear -d (class) (days) Deletes all error class entries in errlog.

Note : The errclear command will delete all entries older than the numbers of
days specified in the days paramenter. To delete ALL entries used 0.

errlogger "message up to 230 chrs"
Enters an operator notifaction message into the errorlog.

——————————————————————————–
PERFORMANCE MONITORING
——————————————————————————–

vmstat (drive) (interval) (count) Reports virtual memory statistics.
vmstat hdisk0 5 20

vmstat -s Diplays number of paging events since system start.
vmstat -f Diplays number of forks since system start.
vmstat -i Diplays number of interupts by device since system start.

iostat (drive) (interval) (count) Reports i/o and cpu statistics.
iostat hdisk0 5 20

iostat -d (drive) (interval) (count) Limits report to drive statistics.
iostat -t (interval) (count) Limits report to tty statistics.

sar -u -P ALL 10 10 Displays %usr %sys %wio %idle for all processors

——————————————————————————–
DOS DISKETTES
——————————————————————————–

dosdir Reads directory listing of a diskette
dosdir (directory) Reads directory listing of a named directory

dosread -D/dev/fd0 C41.TXT c41.txt Gets C41.TXT from diskette drive fd0

dosread -D/dev/fd0 DIRECTORY/C41.TXT c41.txt
(-D option can be dropped if using fd0)

doswrite -D/dev/fd0 (unixfile) (dosfile) Writes a file to diskette

dosdel (dosfile) Deletes a dos file on diskette

dosformat Formats the diskette

——————————————————————————–
SENDMAIL
——————————————————————————–

sendmail -bi Creates new aliase db from /etc/aliase file.
newaliases

sendmail -bp Displays the contents of the mail queue
mailq

sendmail -q Processe the sendmail queue NOW

sendmail -bt -d0.4 < /dev/null
Prints out sendmail version, compile defines and system information

refresh -s sendmail Restart sendmail
kill -l (sendmail PID)

——————————————————————————–
SP / PSSP
——————————————————————————–

dsh (command) Runs the command on all the nodes

Efence Diplays which node are currently fenced
Efence (node number) Fences the node

Eunfence (node number) Unfences the node

Estart Starts the switch

spmon -q Starts SP monitor in gui
spmon -d -G Diag info, lists LED and switch info for all nodes
spmon -L frame1/node3 Displays LED for node 3 in frame 1

spmon -p off frame1/node3 Powers off the node
spmon -p on frame1/node3 Powers on the node

spled Diplays all the nodes LED's in a updating gui

s1term -w (frame number) (node number) Opens serial terminal (read and write)
s1term (frame number) (node number) Opens serial terminal (read only)

Example :-

s1term 1 1 Opens a serial terminal to console port on frame 1 node 1
which is read only. When rebooting a node use read only.

splstdata -e Lists site environment database information
-d Displays df command from each node
-n Lists node configuration
-h Diplays lscfg command from each node
-s Lists switch node information
-b Lists boot/installation information
-a Lists LAN database information
-i Displays netstat -in command from each node

ORA-27054: NFS file system where the file is created or resides is not mounted with correct options

NFS Mount Options

You must mount NFS volumes used for storing database files with special mount options on the host where the database server is running. When mounting an NFS file system, Oracle recommends that you use the same mount point options that your NAS vendor used when certifying the device. Refer to your device documentation or contact your vendor for information about recommended mount-point options.

In general, most vendors recommend that you use the NFS mount options listed in the following table.

Option Requirement Description
hard Mandatory Generate a hard mount of the NFS file system. If the connection to the server fails or is temporarily lost, connection attempts are made until the NAS device responds.
bg Optional Try to connect in the background if connection fails.
tcp Optional Use the TCP protocol rather than UDP. TCP is more reliable than UDP.
nfsvers=3 Optional Use NFS version 3. Oracle recommends that you use NFS version 3 where available, unless the performance of version 2 is higher.
suid Optional Allow clients to run executables with SUID enabled. This option is required for Oracle software mount points.
rsize Mandatory The number of bytes used when reading from the NAS device. This value should be set to the maximum database block size supported by this platform. A value of 8192 is often recommended for NFS version 2 and 32768 is often recommended for NFS version 3.
wsize Mandatory The number of bytes used when writing to the NAS device. This value should be set to the maximum database block size supported by this platform. A value of 8192 is often recommended for NFS version 2 and 32768 is often recommended for NFS version 3.
nointr (or intr) Optional Do not allow (or allow) keyboard interrupts to kill a process that is hung while waiting for a response on a hard-mounted file system.

Note: Different vendors have different recommendations about this option. Contact your vendor for advice.

actime=0 or noac Mandatory Disable attribute caching.

Note: You must specify this option for NFS file systems where you want to install the software. If you do not use this option, Oracle Universal Installer will not install the software in the directory that you specify.

The mandatory mount options comprise the minimum set of mount options that you must use while mounting the NFS volumes. These mount options are essential to protect the integrity of the data and to prevent any database corruption. Failure to use these mount options may result in the generation of file access errors. Refer to your operating system or NAS device documentation for more information about the specific options supported on your platform.

For AIX:
cio,rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3,timeo=600

For HP-UX:
forcedirectio,rw,bg,hard,nointr,rsize=32768,wsize=32768,proto=tcp,noac,vers=3,timeo=600,suid

DBSTART SCRIPT DOES NOT WORK WITH KSH SHELL on HP-UX

DBSTART script does not work with KSH shell and it gives the following error:

/base/oracle/product/V10.2.0.3/bin/dbstart[89]: test: argument expected
Failed to auto-start Oracle Net Listener using /bin/tnslsnr
Processing Database instance “DGED00”: log file
/base/oracle/product/V10.2.0.3/startup.log
logout

This happens when running dbstart as a part of the startup procedures. It can be reproduced using the command line as follows:
su – oracle -c “${ORACLE_HOME}/bin/dbstart”

1/ The result of this command is :
/base/oracle/product/V10.2.0.3/bin/dbstart[89]: test: argument expected
Failed to auto-start Oracle Net Listener using /bin/tnslsnr
Processing Database instance “DGED00”: log file
/base/oracle/product/V10.2.0.3/startup.log
logout

2/ The content of startup.log is :
startinst[79]: /base/oracle/product/V10.2.0.3/dbs/spfileDGED00.ora: unknown
test operator
logger: illegal option — s
Usage: logger [ -t tag ] [ -p pri ] [ -i ] [ -f file ] [message …]
logger: illegal option — s
Usage: logger [ -t tag ] [ -p pri ] [ -i ] [ -f file ] [message …]
logger: illegal option — s
Usage: logger [ -t tag ] [ -p pri ] [ -i ] [ -f file ] [message …]

WORKAROUND:

In the dbstart script, there is a test on existence of some files :

if [ -e $SPFILE -o -e $SPFILE1 -o -e $PFILE ] ; then
When changing it to be (as in version 10.2.0.2), it works fine:
if [ -f $SPFILE -o -f $SPFILE1 -o -f $PFILE ] ; then

ksh on HPIA does not support test -e argument and so fails.
Passing over to DDR as generic as maybe the generic code
could use a more widely supported option such as -r
(as -e will show that a file exists even if it cannot be read
but -r will only short the file exists if it can be read
and -r should honor symbolic links).

vmstat command

The first tool to use is the vmstat command, which quickly provides compact information about various system resources and their related performance problems.
The vmstat command reports statistics about kernel threads in the run and wait queue, memory, paging, disks, interrupts, system calls, context switches, and CPU activity. The reported CPU activity is a percentage breakdown of user mode, system mode, idle time, and waits for disk I/O.
Note: If the vmstat command is used without any options or only with the interval and optionally, the count parameter, such as vmstat 2 10; then the first line of numbers is an average since system reboot.
As a CPU monitor, the vmstat command is superior to the iostat command in that its one-line-per-report output is easier to scan as it scrolls and there is less overhead involved if there are a lot of disks attached to the system. The following example can help you identify situations in which a program has run away or is too CPU-intensive to run in a multiuser environment.

# vmstat 2
kthr memory page faults cpu
—– ———– ———————— ———— ———–
r b avm fre re pi po fr sr cy in sy cs us sy id wa
1 0 22478 1677 0 0 0 0 0 0 188 1380 157 57 32 0 10
1 0 22506 1609 0 0 0 0 0 0 214 1476 186 48 37 0 16
0 0 22498 1582 0 0 0 0 0 0 248 1470 226 55 36 0 9

2 0 22534 1465 0 0 0 0 0 0 238 903 239 77 23 0 0
2 0 22534 1445 0 0 0 0 0 0 209 1142 205 72 28 0 0
2 0 22534 1426 0 0 0 0 0 0 189 1220 212 74 26 0 0
3 0 22534 1410 0 0 0 0 0 0 255 1704 268 70 30 0 0
2 1 22557 1365 0 0 0 0 0 0 383 977 216 72 28 0 0

2 0 22541 1356 0 0 0 0 0 0 237 1418 209 63 33 0 4
1 0 22524 1350 0 0 0 0 0 0 241 1348 179 52 32 0 16
1 0 22546 1293 0 0 0 0 0 0 217 1473 180 51 35 0 14

This output shows the effect of introducing a program in a tight loop to a busy multiuser system. The first three reports (the summary has been removed) show the system balanced at 50-55 percent user, 30-35 percent system, and 10-15 percent I/O wait. When the looping program begins, all available CPU cycles are consumed. Because the looping program does no I/O, it can absorb all of the cycles previously unused because of I/O wait. Worse, it represents a process that is always ready to take over the CPU when a useful process relinquishes it. Because the looping program has a priority equal to that of all other foreground processes, it will not necessarily have to give up the CPU when another process becomes dispatchable. The program runs for about 10 seconds (five reports), and then the activity reported by the vmstat command returns to a more normal pattern.

Optimum use would have the CPU working 100 percent of the time. This holds true in the case of a single-user system with no need to share the CPU. Generally, if us + sy time is below 90 percent, a single-user system is not considered CPU constrained. However, if us + sy time on a multiuser system exceeds 80 percent, the processes may spend time waiting in the run queue. Response time and throughput might suffer.

To check if the CPU is the bottleneck, consider the four cpu columns and the two kthr (kernel threads) columns in the vmstat report. It may also be worthwhile looking at the faults column:

* cpu

Percentage breakdown of CPU time usage during the interval. The cpu columns are as follows:
o us

The us column shows the percent of CPU time spent in user mode. A UNIX® process can execute in either user mode or system (kernel) mode. When in user mode, a process executes within its application code and does not require kernel resources to perform computations, manage memory, or set variables.
o sy

The sy column details the percentage of time the CPU was executing a process in system mode. This includes CPU resource consumed by kernel processes (kprocs) and others that need access to kernel resources. If a process needs kernel resources, it must execute a system call and is thereby switched to system mode to make that resource available. For example, reading or writing of a file requires kernel resources to open the file, seek a specific location, and read or write data, unless memory mapped files are used.
o id

The id column shows the percentage of time which the CPU is idle, or waiting, without pending local disk I/O. If there are no threads available for execution (the run queue is empty), the system dispatches a thread called wait, which is also known as the idle kproc. On an SMP system, one wait thread per processor can be dispatched. The report generated by the ps command (with the -k or -g 0 option) identifies this as kproc or wait. If the ps report shows a high aggregate time for this thread, it means there were significant periods of time when no other thread was ready to run or waiting to be executed on the CPU. The system was therefore mostly idle and waiting for new tasks.
o wa

The wa column details the percentage of time the CPU was idle with pending local disk I/O and NFS-mounted disks. If there is at least one outstanding I/O to a disk when wait is running, the time is classified as waiting for I/O. Unless asynchronous I/O is being used by the process, an I/O request to disk causes the calling process to block (or sleep) until the request has been completed. Once an I/O request for a process completes, it is placed on the run queue. If the I/Os were completing faster, more CPU time could be used.

A wa value over 25 percent could indicate that the disk subsystem might not be balanced properly, or it might be the result of a disk-intensive workload.

For information on the change made to wa, see Wait I/O time reporting.
* kthr

Number of kernel threads in various queues averaged per second over the sampling interval. The kthr columns are as follows:
o r

Average number of kernel threads that are runnable, which includes threads that are running and threads that are waiting for the CPU. If this number is greater than the number of CPUs, there is at least one thread waiting for a CPU and the more threads there are waiting for CPUs, the greater the likelihood of a performance impact.
o b

Average number of kernel threads in the VMM wait queue per second. This includes threads that are waiting on filesystem I/O or threads that have been suspended due to memory load control.

If processes are suspended due to memory load control, the blocked column (b) in the vmstat report indicates the increase in the number of threads rather than the run queue.
o p

For vmstat -I The number of threads waiting on I/Os to raw devices per second. Threads waiting on I/Os to filesystems would not be included here.
* faults

Information about process control, such as trap and interrupt rate. The faults columns are as follows:
o in

Number of device interrupts per second observed in the interval. Additional information can be found in Assessing disk performance with the vmstat command.
o sy

The number of system calls per second observed in the interval. Resources are available to user processes through well-defined system calls. These calls instruct the kernel to perform operations for the calling process and exchange data between the kernel and the process. Because workloads and applications vary widely, and different calls perform different functions, it is impossible to define how many system calls per-second are too many. But typically, when the sy column raises over 10000 calls per second on a uniprocessor, further investigations is called for (on an SMP system the number is 10000 calls per second per processor). One reason could be “polling” subroutines like the select() subroutine. For this column, it is advisable to have a baseline measurement that gives a count for a normal sy value.
o cs

Number of context switches per second observed in the interval. The physical CPU resource is subdivided into logical time slices of 10 milliseconds each. Assuming a thread is scheduled for execution, it will run until its time slice expires, until it is preempted, or until it voluntarily gives up control of the CPU. When another thread is given control of the CPU, the context or working environment of the previous thread must be saved and the context of the current thread must be loaded. The operating system has a very efficient context switching procedure, so each switch is inexpensive in terms of resources. Any significant increase in context switches, such as when cs is a lot higher than the disk I/O and network packet rate, should be cause for further investigation.

Redirection on unix

Redirection on unix

Redirecting stdout and stderr
ls > file – redirect output to file
ls 2> filename – redirect error to file named filename
ls 2>&1 file – redirect error to where output is going to file named file
ls 1>&2 file – redirect stdout to stderr to file named file
ls > file 2> err – redirect output to file and error to file named err
exec 2> std.err – Redirect all error messages to file named std.err

NMON Analyzer: AIX server performance stats

This is a useful application provided by IBM

NMON_Analyser is designed to complement NMON (Nigel’s Monitor) in analysing and reporting performance problems; it produces graphs for virtually all sections of output created using the “spreadsheet output” mode of NMON as well as doing some additional analyses for ESS, EMC and FAStT subsystems. It will also work with files produced by topasout and with other tools that produce data in “NMON” format. It is written in VBA for Excel and will work with the 2002 or 2003 editions.

NMON UserGuide v33

NMON Excel File

List Unix users that can startup, shutdown and admin Databases

#!/bin/ksh
rem ———————————————————————–
rem Filename: sysdba.sh
rem Purpose: List Unix users that can startup, shutdown and admin Databases
rem Author: Frank Naude
rem ———————————————————————–

echo “Users that can startup, shutdown and admin Oracle Databases:”
echo

grep `grep ^dba /etc/group | cut -d: -f3` /etc/passwd

AWK : Linux

AWK: The Linux Administrators’ Wisdom Kit
By Emmett Dulaney

Learning Linux? This introduction to the invaluable AWK text-manipulation tool will be invaluable.

The AWK utility, with its own self-contained language, is one of the most powerful data processing engines in existence — not only in Linux, but anywhere. The limits to what can be done with this programming and data-manipulation language (named for the last initials of its creators, Alfred Aho, Peter Weinberger, and Brian Kernighan) are the boundaries of one’s own knowledge. It allows you to create short programs that read input files, sort data, process it, perform arithmetic on the input, and generate reports, among myriad other functions.

What Is AWK?

To put it the simplest way possible, AWK is a programming-language tool used to manipulate text. The language of the AWK utility resembles the shell-programming language in many areas, although AWK’s syntax is very much its own. When first created, AWK was designed to work in the text-processing arena, and the language is based on executing a series of instructions whenever a pattern is matched in the input data. The utility scans each line of a file, looking for patterns that match those given on the command line. If a match is found, it takes the next programming step. If no match is found, it then proceeds to the next line.

While the operations can get complex, the syntax for the command is always:

awk ‘{pattern + action}’ {filenames}

where pattern represents what AWK is looking for in the data, and action is a series of commands executed when a match is found. Curly brackets ({}) are not always required around your program, but they are used to group a series of instructions based on a specific pattern.

Understanding Fields

The utility separates each input line into records and fields. A record is a single line of input, and each record consists of several fields. The default-field separator is a space or a tab, and the record separator is a new line. Although both tabs and spaces are perceived as field separators by default (multiple blank spaces still count as one delimiter), the delimiter can be changed from white space to any other character.

To illustrate, look at the following employee-list file saved as emp_names:

46012 DULANEY EVAN MOBILE AL
46013 DURHAM JEFF MOBILE AL
46015 STEEN BILL MOBILE AL
46017 FELDMAN EVAN MOBILE AL
46018 SWIM STEVE UNKNOWN AL
46019 BOGUE ROBERT PHOENIX AR
46021 JUNE MICAH PHOENIX AR
46022 KANE SHERYL UNKNOWN AR
46024 WOOD WILLIAM MUNCIE IN
46026 FERGUS SARAH MUNCIE IN
46027 BUCK SARAH MUNCIE IN
46029 TUTTLE BOB MUNCIE IN

As AWK reads the input, the entire record is assigned to the variable $0. Each field, as split with the field separator, is assigned to the variables $1, $2, $3, and so on. A line contains essentially an unlimited number of fields, with each field being accessed by the field number. Thus, the command

awk ‘{print $1,$2,$3,$4,$5}’ names

would result in a printout of

46012 DULANEY EVAN MOBILE AL
46013 DURHAM JEFF MOBILE AL
46015 STEEN BILL MOBILE AL
46017 FELDMAN EVAN MOBILE AL
46018 SWIM STEVE UNKNOWN AL
46019 BOGUE ROBERT PHOENIX AR
46021 JUNE MICAH PHOENIX AR
46022 KANE SHERYL UNKNOWN AR
46024 WOOD WILLIAM MUNCIE IN
46026 FERGUS SARAH MUNCIE IN
46027 BUCK SARAH MUNCIE IN
46029 TUTTLE BOB MUNCIE IN

An important item of noteworthiness is that AWK interprets the five fields as being separated by white space, but when it prints the display, there is only one space between each field. By virtue of the ability to address each field with a unique number, you can choose to print only certain fields. For example, to print only the names from each record, select only the second and third fields to print:

$ awk ‘{print $2,$3}’ emp_names
DULANEY EVAN
DURHAM JEFF
STEEN BILL
FELDMAN EVAN
SWIM STEVE
BOGUE ROBERT
JUNE MICAH
KANE SHERYL
WOOD WILLIAM
FERGUS SARAH
BUCK SARAH
TUTTLE BOB
$

You can also specify that the fields print in any order, regardless of how they exist in the record. Thus, to show only the name fields, and reverse them so the first name is shown, then the last:

$ awk ‘{print $3,$2}’ emp_names
EVAN DULANEY
JEFF DURHAM
BILL STEEN
EVAN FELDMAN
STEVE SWIM
ROBERT BOGUE
MICAH JUNE
SHERYL KANE
WILLIAM WOOD
SARAH FERGUS
SARAH BUCK
BOB TUTTLE
$

Working with Patterns

You can select the action to take place only on certain records, and not on all records, by including a pattern that must be matched. The simplest form of pattern matching is that of a search, wherein the item to be matched is included in slashes (/pattern/). For example, to perform the earlier action only on those employees who live in Alabama:

$ awk ‘/AL/ {print $3,$2}’ emp_names
EVAN DULANEY
JEFF DURHAM
BILL STEEN
EVAN FELDMAN
STEVE SWIM
$

If you do not specify what fields to print, the entire matching entry will print:

$ awk ‘/AL/’ emp_names
46012 DULANEY EVAN MOBILE AL
46013 DURHAM JEFF MOBILE AL
46015 STEEN BILL MOBILE AL
46017 FELDMAN EVAN MOBILE AL
46018 SWIM STEVE UNKNOWN AL
$

Multiple commands for the same set of data can be separated with a semicolon (;). For example, to print names on one line and city and state on another:

$ awk ‘/AL/ {print $3,$2 ; print $4,$5}’ emp_names
EVAN DULANEY
MOBILE AL
JEFF DURHAM
MOBILE AL
BILL STEEN
MOBILE AL
EVAN FELDMAN
MOBILE AL
STEVE SWIM
UNKNOWN AL
$

If the semicolon were not used (print $3,$2,$4,$5), all would appear on the same line. On the other hand, if the two print statements were given separately, an altogether different result would occur:

$ awk ‘/AL/ {print $3,$2} {print $4,$5}’ emp_names
EVAN DULANEY
MOBILE AL
JEFF DURHAM
MOBILE AL
BILL STEEN
MOBILE AL
EVAN FELDMAN
MOBILE AL
STEVE SWIM
UNKNOWN AL
PHOENIX AR
PHOENIX AR
UNKNOWN AR
MUNCIE IN
MUNCIE IN
MUNCIE IN
MUNCIE IN
$

Fields three and two are given only when AL is found in the listing. Fields four and five, however, are unconditional and always print. Only the commands within the first set of curly braces are active for the command (/AL/) immediately preceding.

The result is altogether cumbersome to read and can use cleaning up a bit. First, insert a space and comma between city and state. Next, leave a blank line after each two-line display:

$ awk ‘/AL/ {print $3,$2 ; print $4″, “$5″\n”}’ emp_names
EVAN DULANEY
MOBILE, AL

JEFF DURHAM
MOBILE, AL

BILL STEEN
MOBILE, AL

EVAN FELDMAN
MOBILE, AL

STEVE SWIM
UNKNOWN, AL
$

Between the fourth and fifth fields, a comma and a space are added (between the quotation marks), and after the fifth field, a new line character (\n) is printed. All the special characters that can be used with the echo command can be also be used with AWK print statements, including:

* \n (new line)
* \t (tab)
* \b (backspace)
* \f (formfeed)
* \r (carriage return)

Thus, to read all five fields, which were originally separated by tabs, and print them with tabs as well, you could program

$ awk ‘{print $1″\t”$2″\t”$3″\t”$4″\t”$5}’ emp_names
46012 DULANEY EVAN MOBILE AL
46013 DURHAM JEFF MOBILE AL
46015 STEEN BILL MOBILE AL
46017 FELDMAN EVAN MOBILE AL
46018 SWIM STEVE UNKNOWN AL
46019 BOGUE ROBERT PHOENIX AR
46021 JUNE MICAH PHOENIX AR
46022 KANE SHERYL UNKNOWN AR
46024 WOOD WILLIAM MUNCIE IN
46026 FERGUS SARAH MUNCIE IN
46027 BUCK SARAH MUNCIE IN
46029 TUTTLE BOB MUNCIE IN
$

You can search for more than one pattern match at a time by placing the multiple criteria in consecutive order and separating them with a pipe (|) symbol:

$ awk ‘/AL|IN/’ emp_names
46012 DULANEY EVAN MOBILE AL
46013 DURHAM JEFF MOBILE AL
46015 STEEN BILL MOBILE AL
46017 FELDMAN EVAN MOBILE AL
46018 SWIM STEVE UNKNOWN AL
46024 WOOD WILLIAM MUNCIE IN
46026 FERGUS SARAH MUNCIE IN
46027 BUCK SARAH MUNCIE IN
46029 TUTTLE BOB MUNCIE IN
$

This finds every match for Alabama and Indiana residents. A problem occurs, however, when you try to find the people who live in Arizona:

$ awk ‘/AR/’ emp_names
46019 BOGUE ROBERT PHOENIX AR
46021 JUNE MICAH PHOENIX AR
46022 KANE SHERYL UNKNOWN AR
46026 FERGUS SARAH MUNCIE IN
46027 BUCK SARAH MUNCIE IN
$

Employees 46026 and 46027 do not live in Arizona; however, their first names contain the character sequence being searched for. The important thing to remember is that when pattern matching in AWK, as in grep, sed, or most other Linux/Unix commands, look for a match anywhere in the record (line) unless told to do otherwise. To solve this problem, it is necessary to tie the search to a particular field. This goal is accomplished by means of a tilde (˜) and a specification to a specific field, as the following example illustrates:

$ awk ‘$5 ˜ /AR/’ emp_names
46019 BOGUE ROBERT PHOENIX AR
46021 JUNE MICAH PHOENIX AR
46022 KANE SHERYL UNKNOWN AR
$

The opposite of the tilde (signifying a match) is a tilde preceded by an exclamation mark (!˜). These characters tell the program to find all lines matching the search sequence, providing the sequence does not appear in the specified field:

$ awk ‘$5 !˜ /AR/’ names
46012 DULANEY EVAN MOBILE AL
46013 DURHAM JEFF MOBILE AL
46015 STEEN BILL MOBILE AL
46017 FELDMAN EVAN MOBILE AL
46018 SWIM STEVE UNKNOWN AL
46024 WOOD WILLIAM MUNCIE IN
46026 FERGUS SARAH MUNCIE IN
46027 BUCK SARAH MUNCIE IN
46029 TUTTLE BOB MUNCIE IN
$

In this case, it displayed all lines that do not have AR in the fifth field — including the two Sarah’s entries that do have AR, but in the third field instead of the fifth one.

Braces and Field Separators

The bracket characters play an important part in the AWK commands. The actions that appear between them spell out what will take place and when it will take place. When only one set of brackets is used:

{print $3,$2}

all the actions between occur at the same time. When more than one set of brackets is used:

{print $3}{print $2}

the first sequence of commands is carried out until completion, then the second sequence kicks in. Notice the difference between the two listings that follow:

$ awk ‘{print $3,$2}’ names
EVAN DULANEY
JEFF DURHAM
BILL STEEN
EVAN FELDMAN
STEVE SWIM
ROBERT BOGUE
MICAH JUNE
SHERYL KANE
WILLIAM WOOD
SARAH FERGUS
SARAH BUCK
BOB TUTTLE
$

$ awk ‘{print $3}{print $2}’ names
EVAN
DULANEY
JEFF
DURHAM
BILL
STEEN
EVAN
FELDMAN
STEVE
SWIM
ROBERT
BOGUE
MICAH
JUNE
SHERYL
KANE
WILLIAM
WOOD
SARAH
FERGUS
SARAH
BUCK
BOB
TUTTLE
$

To reiterate the findings with multiple sets of brackets, the commands within the first set are carried out until completion; processing then moves to the second set. If there were a third set, it would go to that set on completion of the second set, and so on. In the generated printout, there are two separate print commands, so the first one is carried out, followed by the second, causing the display for each entry to appear on two lines instead of one.

The field separator differentiating one field from another need not always be white space; it can be any discernible character. To illustrate, assume the emp_names file separated the fields with colons instead of tabs:

$ cat emp_names
46012:DULANEY:EVAN:MOBILE:AL
46013:DURHAM:JEFF:MOBILE:AL
46015:STEEN:BILL:MOBILE:AL
46017:FELDMAN:EVAN:MOBILE:AL
46018:SWIM:STEVE:UNKNOWN:AL
46019:BOGUE:ROBERT:PHOENIX:AR
46021:JUNE:MICAH:PHOENIX:AR
46022:KANE:SHERYL:UNKNOWN:AR
46024:WOOD:WILLIAM:MUNCIE:IN
46026:FERGUS:SARAH:MUNCIE:IN
46027:BUCK:SARAH:MUNCIE:IN
46029:TUTTLE:BOB:MUNCIE:IN
$

If you attempted to print the last name by specifying that you wanted the second field with

$ awk ‘{print $2}’ emp_names

you would end up with twelve blank lines. Because there are no spaces in the file, there are no discernible fields beyond the first one. To solve the problem, AWK must be told that a character other than white space is the delimiter, and there are two methods by which to inform AWK of the new field separator: Use the command-line parameter -F, or specify the variable FS within the program. Both strategies work equally well, with one exception, as illustrated by the following example:

$ awk ‘{FS=”:”}{print $2}’ emp_names

DURHAM
STEEN
FELDMAN
SWIM
BOGUE
JUNE
KANE
WOOD
FERGUS
BUCK
TUTTLE
$

$ awk -F: ‘{print $2}’ emp_names
DULANEY
DURHAM
STEEN
FELDMAN
SWIM
BOGUE
JUNE
KANE
WOOD
FERGUS
BUCK
TUTTLE
$

In the first command, a blank line is incorrectly returned for the very first record, while all the others work as they should. It is not until the second record is read that the field separator is recognized and properly acted on. This shortcoming can be corrected by using a BEGIN statement (more on that later). The -F works much like a BEGIN and is able to correctly read the first record and act on it as it should.

As I mentioned at the start of this article, the default display/output field separator is a blank space. This feature can be changed within the program by using the Output Field Separator (OFS) variable. For example, to read the file (separated by colons) and display it with dashes, the command would be

$ awk -F”:” ‘{OFS=”-“}{print $1,$2,$3,$4,$5}’ emp_names
46012-DULANEY-EVAN-MOBILE-AL
46013-DURHAM-JEFF-MOBILE-AL
46015-STEEN-BILL-MOBILE-AL
46017-FELDMAN-EVAN-MOBILE-AL
46018-SWIM-STEVE-UNKNOWN-AL
46019-BOGUE-ROBERT-PHOENIX-AZ
46021-JUNE-MICAH-PHOENIX-AZ
46022-KANE-SHERYL-UNKNOWN-AR
46024-WOOD-WILLIAM-MUNCIE-IN
46026-FERGUS-SARAH-MUNCIE-IN
46027-BUCK-SARAH-MUNCIE-IN
46029-TUTTLE-BOB-MUNCIE-IN
$

FS and OFS, (input) Field Separator and Output Field Separator, are but a couple of the variables that can be used within the AWK utility. For example, to number each line as it is printed, use the NR variable in the following manner:

$ awk -F”:” ‘{print NR,$1,$2,$3}’ emp_names
1 46012 DULANEY EVAN
2 46013 DURHAM JEFF
3 46015 STEEN BILL
4 46017 FELDMAN EVAN
5 46018 SWIM STEVE
6 46019 BOGUE ROBERT
7 46021 JUNE MICAH
8 46022 KANE SHERYL
9 46024 WOOD WILLIAM
10 46026 FERGUS SARAH
11 46027 BUCK SARAH
12 46029 TUTTLE BOB
$

To find all lines with employee numbers between 46012 and 46015:

$ awk -F”:” ‘/4601[2-5]/’ emp_names
46012 DULANEY EVAN MOBILE AL
46013 DURHAM JEFF MOBILE AL
46015 STEEN BILL MOBILE AL
$

Adding Text

Text may be added to the display in the same manner as control sequences or other characters are. For example, to change the delimiter from spaces to colons, the command could be

awk ‘{print $1″:”$2″:”$3″:”$4″:”$5}’ emp_names > new_emp_names

In this case, the character (:), enclosed in quotation marks (“/”), is added between each of the fields. This value within the quotation marks can be anything. For example, to create a database-like display of the employees living in Alabama:

$ awk ‘$5 ~ /AL/ {print “NAME: “$2”, “$3″\nCITY-STATE:
“$4”, “$5″\n”}’ emp_names

NAME: DULANEY, EVAN
CITY-STATE: MOBILE, AL

NAME: DURHAM, JEFF
CITY-STATE: MOBILE, AL

NAME: STEEN, BILL
CITY-STATE: MOBILE, AL

NAME: FELDMAN, EVAN
CITY-STATE: MOBILE, AL

NAME: SWIM, STEVE
CITY-STATE: UNKNOWN, AL
$

Math Operations

In addition to the textual possibilities AWK provides, it also offers a full range of arithmetic operators, including the following:

+ adds numbers together
– subtracts
* multiplies
/ divides
^ performs exponential mathematics
% gives the modulo
++ adds one to the value of a variable
+= assigns the result of an addition operation to a variable
— subtracts one from a variable
-= assigns the result of a subtraction operation to a variable
*= assigns the result of multiplication
/= assigns the result of division
%= assigns the result of a modulo operation

For example, assume the following file exists on your machine detailing the inventory in a hardware store:

$ cat inventory
hammers 5 7.99
drills 2 29.99
punches 7 3.59
drifts 2 4.09
bits 55 1.19
saws 123 14.99
nails 800 .19
screws 80 .29
brads 100 .24
$

The first order of business is to compute the value of each item’s inventory by multiplying the value of the second field (quantity) by the value of the third field (price):

$ awk ‘{print $1,”QTY: “$2,”PRICE: “$3,”TOTAL: “$2*$3}’ inventory
hammers QTY: 5 PRICE: 7.99 TOTAL: 39.95
drills QTY: 2 PRICE: 29.99 TOTAL: 59.98
punches QTY: 7 PRICE: 3.59 TOTAL: 25.13
drifts QTY: 2 PRICE: 4.09 TOTAL: 8.18
bits QTY: 55 PRICE: 1.19 TOTAL: 65.45
saws QTY: 123 PRICE: 14.99 TOTAL: 1843.77
nails QTY: 800 PRICE: .19 TOTAL: 152
screws QTY: 80 PRICE: .29 TOTAL: 23.2
brads QTY: 100 PRICE: .24 TOTAL: 24
$

If the lines themselves are unimportant, and you want only to determine exactly how many items are in the store, you can assign a generic variable to increment by the number of items in each record:

$ awk ‘{x=x+$2} {print x}’ inventory
5
7
14
16
71
194
994
1074
1174
$

According to this data, 1,174 items are in the store. The first time through, the variable x had no value, so it took the value of the first line’s second field. The next time through, it retained the value of the first line and added the value from the second line, and so on, until it arrived at a cumulative total.

The same process can be applied to determining the total value of the inventory on hand:

$ awk ‘{x=x+($2*$3)} {print x}’ inventory
39.95
99.93
125.06
133.24
198.69
2042.46
2194.46
2217.66
2241.66
$

Thus, the value of the 1,174 items is $2,241.66. Although this procedure is good for getting a total, it does not look at all pretty, and it would need sanitizing for an actual report. Sprucing up the display a bit can be easily accomplished with a few additions:

$ awk ‘{x=x+($2*$3)}{print $1,”QTY: “$2,”PRICE: “$3,”TOTAL: “$2*$3,”BAL: “x}’ inventory
hammers QTY: 5 PRICE: 7.99 TOTAL: 39.95 BAL: 39.95
drills QTY: 2 PRICE: 29.99 TOTAL: 59.98 BAL: 99.93
punches QTY: 7 PRICE: 3.59 TOTAL: 25.13 BAL: 125.06
drifts QTY: 2 PRICE: 4.09 TOTAL: 8.18 BAL: 133.24
bits QTY: 55 PRICE: 1.19 TOTAL: 65.45 BAL: 198.69
saws QTY: 123 PRICE: 14.99 TOTAL: 1843.77 BAL: 2042.46
nails QTY: 800 PRICE: .19 TOTAL: 152 BAL: 2194.46
screws QTY: 80 PRICE: .29 TOTAL: 23.2 BAL: 2217.66
brads QTY: 100 PRICE: .24 TOTAL: 24 BAL: 2241.66
$

This procedure gives a listing of each record while assigning a total value to the inventory and keeping a running balance of the store’s inventory.

BEGIN and END

Actions can be specified to take place prior to the actual start of processing or after it has been completed with BEGIN and END statements respectively. BEGIN statements are most commonly used to establish variables or display a header. END statements, on the other hand, can be used to continue processing after the program has finished.

In an earlier example, a complete value of the inventory was generated with the routine

awk ‘{x=x+($2*$3)} {print x}’ inventory

This routine provided a display for each line in the file as the running total accumulated. There was no other way to specify it, and not having it print at each line would have resulted in it never printing. With an END statement, however, this problem can be circumvented:

$ awk ‘{x=x+($2*$3)} END {print “Total Value of Inventory: “x}’ inventory
Total Value of Inventory: 2241.66
$

The variable x is defined, and it processes for each line; however, no display is generated until all processing has completed. While it’s useful as a standalone routine, it an also be put with the earlier listing to add even more information and a more complete report:

$ awk ‘{x=x+($2*$3)} {print $1,”QTY: “$2,”PRICE:
“$3,”TOTAL: “$2*$3} END {print “Total Value of Inventory: ” x}’ inventory

hammers QTY: 5 PRICE: 7.99 TOTAL: 39.95
drills QTY: 2 PRICE: 29.99 TOTAL: 59.98
punches QTY: 7 PRICE: 3.59 TOTAL: 25.13
drifts QTY: 2 PRICE: 4.09 TOTAL: 8.18
bits QTY: 55 PRICE: 1.19 TOTAL: 65.45
saws QTY: 123 PRICE: 14.99 TOTAL: 1843.77
nails QTY: 800 PRICE: .19 TOTAL: 152
screws QTY: 80 PRICE: .29 TOTAL: 23.2
brads QTY: 100 PRICE: .24 TOTAL: 24
Total Value of Inventory: 2241.66
$

The BEGIN command words in the same fashion as END, but it establishes items that need to be done before anything else is accomplished. The most common purpose of this procedure is to create headers for reports. The syntax for this routine would resemble

$ awk ‘BEGIN {print “ITEM QUANTITY PRICE TOTAL”}’

Input, Output, and Source Files

The AWK tool can read its input from a file, as was done in all examples up to this point, or it can take input from the output of another command. For example:

$ sort emp_names | awk ‘{print $3,$2}’

The input of the awk command is the output from the sort operation. In addition to sort, any other Linux command can be used — for example, grep. This procedure allows you to perform other operations on the file before pulling out selected fields.

Like the shell, AWK uses the output-redirection operators > and >> to put its output into a file rather than to standard output. The symbols react like their counterparts in the shell, so > creates the file if it doesn’t exist, and >> appends to the existing file. Examine the following example:

$ awk ‘{print NR, $1 ) > “/tmp/filez” }’ emp_names
$ cat /tmp/filez
1 46012
2 46013
3 46015
4 46017
5 46018
6 46019
7 46021
8 46022
9 46024
10 46026
11 46027
12 46029
$

Examining the syntax of the statement, you can see that the output redirection is done after the print statement is complete. You must enclose the file name in quotes, or else it is simply an uninitialized AWK variable, and the combination of instructions generates an error from AWK. (If you use the redirection symbols improperly, AWK gets confused about whether the symbol means “redirection” or is a relation operator.)

Output into pipes in AWK also resembles the way the same action would be accomplished in a shell. To send the output of a print command into a pipe, follow the print command with a pipe symbol and the name of the command, as in the following:

$ awk ‘{ print $2 | “sort” }’ emp_names
BOGUE
BUCK
DULANEY
DURHAM
FELDMAN
FERGUS
JUNE
KANE
STEEN
SWIM
TUTTLE
WOOD
$

As was the case with output redirection, you must enclose the command in quotes, and the name of the pipe is the name of the command being executed.

Commands used by AWK can come from two locations. First, they can be specified on the command line, as illustrated. Second, they can come from a source file. If such is the case, AWK is alerted to this occurrence by means of the -f option. To illustrate:

$ cat awklist
{print $3,$2}
{print $4,$5,”\n”}
$

$ awk -f awklist emp_names
EVAN DULANEY
MOBILE AL

JEFF DURHAM
MOBILE AL

BILL STEEN
MOBILE AL

EVAN FELDMAN
MOBILE AL

STEVE SWIM
UNKNOWN AL

ROBERT BOGUE
PHOENIX AR

MICAH JUNE
PHOENIX AR

SHERYL KANE
UNKNOWN AR

WILLIAM WOOD
MUNCIE IN

SARAH FERGUS
MUNCIE IN

SARAH BUCK
MUNCIE IN

BOB TUTTLE
MUNCIE IN

$

Notice that the apostrophes are not used anywhere within the source file or when calling it at the command line. They are only for use in differentiating the commands on the command line from file names.

If simple output cannot handle the intricate detail you want in your programs, try the more complex output available with the printf command, the syntax for which is

printf( format, value, value …)

This syntax is like that of the printf command in the C language, and the specifications for the format are the same. You define the format by inserting a specification that defines how the value is to be printed. The format specification consists of a % followed by a letter. Like the print command, printf does not have to be enclosed in parentheses, but using them is considered good practice.

The following table lists the various specifications available for the printf command.

Specification Description
%c Prints a single ASCII character
%d Prints a decimal number
%e Prints a scientific notation representation of numbers
%f Prints a floating-point representation
%g Prints %e or %f; whichever is shorter
%o Prints an unsigned octal number
%s Prints an ASCII string
%x Prints an unsigned hexadecimal number
%% Prints a percent sign; no conversion is performed

You can supply some additional formatting parameters between the % and the character. These parameters further refine how the value is printed:

Parameter Description
– Left justifies the expression in the field
width Pads the field to the specified width as needed (a leading zero pads the field with zeros)
.prec Maximum string width or the maximum number of digits to the right of the decimal point

The printf command enables you to control and translate the value from one format to another. When you want to print the value of a variable, simply provide a specification that instructs printf how to print the information (normally enclosed in double quotes). You must include a specification parameter for each variable that is being passed to printf; if too few parameters are included, printf does not print all the values.