----------------------------
                           Mitigating the Obfuscation
                            Circumscribing the Three
                              Primary Unix Shells
                                       or
                            "Ohhh, so THAT's what
                             they were doing at
                             Berkeley in the 70s!"
                          ----------------------------

                                AZSAGE Meeting
                              September 16, 1997
                         Ralph Luethy, Sun Instructor
                         Business & Industry Institute


This presentation is in two parts.  The first part will examine those
features common across all shells while the second compares and
contrasts the three primary Unix shells: the Bourne shell, C shell,
and Korn shell.

PART I
------

A Brief Overview and History

    - Bourne Shell
        Steve Bourne of AT&T
        /bin/sh and /sbin/sh (why two of them?)
        Faster, smaller (207K RAM)
        Standard default shell for UNIX and all startup scripts
        Uses the $ prompt

    - C Shell
        Bill Joy (Co-founder Sun)
        /bin/csh
        Slower, bigger (261K RAM)
        Added history
        More builtins (echo, math, testing)
        Uses the % prompt

    - Korn Shell   (/bin/ksh)
        David Korn
        Written by AT&T just like Bourne shell
        /bin/ksh
        Bigger (365K RAM) but often faster than Bourne/C
        Newer shell to Sun
        Interactive command-line history
        More builtins (print, math, testing)
        Combines best features of Bourne and C shell
        Can run Bourne shell scripts

The Nature of a Shell Script in the Unix Environment

    - A floor wax and a dessert topping

        A command-line interpreter (e.g., % echo a*)
        A programming language

    - Violence in the Unix home (killing parents and forking children)

        How a shell find commands
            C/Korn aliases and functions are searched
            Then builtins
            Then utility using PATH
        Forking
        Parent sleeps
        Execing

    - Execution vs. sourcing

        % cd
        % vi tryit

        -------------
        |cd /tmp
        |pwd
        |

        % chmod +x tryit

        % tryit
        /tmp

        % pwd
        what directory are we in?

        % source tryit
        /tmp

        % pwd
        what directory are we in now?

    - The Unix philosophy
        Small, focused utilities

    - Standard in, Standard out, Standard error

        File        Name       Destination/Source
      Descriptor

         0          stdin          Keyboard
         1          stdout         Screen/Window
         2          stderr         Screen/Window

    - Redirection and pipes

        % vi message
        % mailx -s "About that meeting" fred < message

        % ls > ls.out
        % find / -print | wc -l

    - Environmental vs. local

        Shell Variables
        ---------------
        % set name=Fred
        % echo $name
        Fred
        % csh
        % echo $name
        name: Undefined variable    <-----
        % exit
        % echo $name
        Fred

        Environment Variables
        ---------------------
        % setenv NAME Gloria
        % echo $NAME
        Gloria
        % csh
        % echo $NAME
        Gloria                      <-----
        % exit

Metacharacters, A Favorite Among Unix Neophytes

    - Quotations: double vs. single

        # Use to preserve spacing
        % echo -n "  Name:  "

        # Interpreted vs. literal
        % set name=Fred
        % echo "$name"
        % echo '$name'

        # You may have problem with C shell history
        % echo $name\!    <-- Ok
        % echo "$name!"   <-- Nope

    - The second most mysterious command line in all of Unix

        % find / -name core -mtime +7 -exec rm {} \; -print

    - The MOST mysterious command syntax in all of Unix

        alias cd 'cd \!* ; set prompt="[\! ^H] `whoami`@`hostname` $cwd % "'

        Gives you this prompt -> [15] luethy@olympus /tmp %

Wildcards vs. Regular Expressions (RE)

    Regular expressions are used by many Unix utilities, like awk, grep
    and sed.  Even vi uses regular expressions.

         Description                     Wildcard      RE
    ------------------------             --------   --------
    List or range                           []         []
    Escape                                  \          \
    Single unknown character                ?          .
    0 or more of preceeding character                *
    0 or more of any character              *          .*
    Beginning of line                                 ^RE
    End of line                                       RE$
    Beginning of word                                 \

    Sample RE:      \

    Sample text:    Although he was filled with courage
                    and blessed with amazing speed,
                    he came in fourth because he
                    fell down rounding the south turn.

A Basic Shell Script Example

    - The ALL IMPORTANT first line

    % vi first

    ----------------
    |#!/bin/wc
    |How many lines, words, and characters
    |are in this file?

    ----------------
    |#!/bin/csh -f
    |ls

    ----------------
    |#
    |ls

    ----------------
    |ls

    - Comments

        # Here's a comment
        cd /tmp  # here's an end-of-line comment

    - Making it executable (or not)

        % chmod +x first
        % first
        OR...
        % csh first


                                   ---------
                                    PART II
                                   ---------

Differences between the three primary Unix shells: the Bourne shell,
C shell, and Korn shell.

Builtins (output provided by "man shell_builtins")

                 command                      built into
                 -------                     ------------
                 alias                       csh, ksh
                 bg                          csh, ksh, sh
                 break                       csh, ksh, sh
                 case                        csh, ksh, sh
                 cd                          csh, ksh, sh
                 chdir                       csh, sh
                 continue                    csh, ksh, sh
                 dirs                        csh
                 echo                        csh, ksh, sh
                 eval                        csh, ksh, sh
                 exec                        csh, ksh, sh
                 exit                        csh, ksh, sh
                 export                      ksh, sh
                 fc                          ksh
                 fg                          csh, ksh, sh
                 for                         ksh, sh
                 foreach                     csh
                 function                    ksh
                 getopts                     ksh, sh
                 glob                        csh
                 goto                        csh
                 hash                        ksh, sh
                 hashstat                    csh
                 history                     csh
                 if                          csh, ksh, sh
                 jobs                        csh, ksh, sh
                 kill                        csh, ksh, sh
                 let                         ksh
                 limit                       csh
                 login                       csh, ksh, sh
                 logout                      csh, ksh, sh
                 nice                        csh
                 newgrp                      ksh, sh
                 notify                      csh
                 onintr                      csh
                 popd                        csh
                 print                       ksh
                 pushd                       csh
                 pwd                         ksh, sh
                 read                        ksh, sh
                 readonly                    ksh, sh
                 rehash                      csh
                 repeat                      csh
                 return                      ksh, sh
                 select                      ksh
                 set                         csh, ksh, sh
                 setenv                      csh
                 shift                       csh, ksh, sh
                 source                      csh
                 stop                        csh, ksh, sh
                 suspend          |          csh, ksh, sh
                 switch           |          csh
                 test             |          ksh, sh
                 time             |          csh
                 times            |          ksh, sh
                 trap             |          ksh, sh
                 type             |          ksh, sh
                 typeset          |          ksh
                 ulimit           |          ksh, sh
                 umask            |          csh, ksh, sh
                 unalias          |          csh, ksh
                 unhash           |          csh
                 unlimit          |          csh
                 unset            |          csh, ksh, sh
                 unsetenv         |          csh
                 until            |          ksh, sh
                 wait             |          csh, ksh, sh
                 whence           |          ksh
                 while            |          csh, ksh, sh

Setting and displaying Variables

    - All shells

        echo $variable  # to display content of any variable

    - Bourne and Korn

        name=Fred   # shell (local) variable
        set         # display all shell variables
        export name # make it environment (global) variable
        env         # display env. variables
        unset name  # unset variable
    
    - Korn

        export name=Fred # make shell AND env. variable with one statement
        unset name       # unset both shell and env. variable

    - C shell

        set name=fred   # shell variable
        set             # display shell variables
        setenv NAME Joe # environment variable
        env             # display env. variables

A User's Environment

    - Startup order

            Bourne: /etc/profile, $HOME/.profile
            Korn:   /etc/profile, $HOME/.profile, $HOME/.kshrc
            Cshell: /etc/.login,  $HOME/.cshrc,   $HOME/.login

    - Startup file contents for Bourne shell users:

        /etc/profile:   environment variables to be shared by all Bourne users

        $HOME/.profile: personal local/environment variables and startup stuff

    - Startup file contents for Korn shell users:

        /etc/profile:   environment variables to be shared by all Bourne users

        $HOME/.profile: personal environment variables and startup stuff

        $HOME/.kshrc:   shell variables, aliases

    - Startup file contents for C shell users:

        /etc/.login:  environment variables to be shared by all C shell users

        $HOME/.login: personal environment variables, startup stuff and
                      "set path=()"

        $HOME/.cshrc: shell variables, aliases,

Arrays

    - Bourne shell

        No array support

    - Korn shell

        name[0]=Fred     # initialize
        name[10]=Joe     # elements between 0 and 10 are not created
        echo ${name[0]}  # display single element
        echo ${name[*]}  # all elements, displays "Fred Joe"
        echo ${#name[*]} # number of elements in array
    
    - C shell

        set names=(Joe Gloria Fred)
        echo $names[1]   # this displays "Joe" (no 0 index)
        set list=`ls`    # create array containing filenames
        echo $list[1]    # displays first filename

Aliases

    - Bourne shell

        No aliases (use functions instead)

    - Korn shell

        $ alias home='cd ; ls'
    
    - C shell

        % alias home 'cd ; ls'

Generating Standard Output (stdout)

    - The /usr/bin/echo utility and the Bourne/Korn shell builtin:

        Used with with no parameters generates a CR
        May suppress a CR on the end of a line with "\c"
            (e.g., echo "Name: \c")

    - The C shell echo builtin

        Used with no parameters generates no output
        May suppress a CR on the end of a line with the "-n" option
            (e.g., echo -n "Name: ")

Getting Shell Input

    - Bourne and Korn shell:

        echo "Name: \c"
        read name
        echo $name
    
    - C shell:

        echo -n "Name: "
        set name = $<
        echo $name

    - Korn shell:

        read name?"Enter name: "    
        echo $name

History

    - Bourne shell

        No history
    
    - C shell

        % set history=50
        % ls /tmpp
        /tmpp: No such file or directory
        % history
        1  set history=20
        2  ls /tmpp
        3  history           [display history]
        % !s                 [rerun last command that started with s]
        % !2                 [rerun command number 2]
        % !2:s/tmpp/tmp      [rerun command number 2, substitue tmpp with tmp]
        ls /tmp
        ...
        % !!                 [repeat last command]

    - Korn shell

        $ vi ~/.kshrc
        set -o vi
        $ . ~/.kshrc
        $ ls /tmpp
        $ [press escape key then vi's "hjkl" keys to navigate]
        $ history
        $ r

Variables and Command Line Parameters

    - Bourne, Korn, and C shell

        $1, $2, ... $9 - command line parameters
        $#             - Number of command line parameters
        $0             - Name of command
        $10            - Wrong!  Interpreted as $1 with 0 appended
        shift          - Shift args one position to left

Exit Status

    - Bourne and Korn

        $?
    
    - C shell

        $status

Command Substitution

    - All shells use the accent grave characters to surround a command.
      The command will be run by the shell and its output will substitute
      the command.

      % set today=`date`
      $ today=`date`
      $ echo "It is now: `date`"

    - Korn shell uses something different in addition to accent graves.

      $ today=$(date)

      You can use the following syntax to set the value of a variable
      to the contents of a file.

      $ list=$(< /etc/passwd)
    
    - Table driven mail example

      % vi list
      [email protected]
      [email protected]
      [email protected]
      [email protected]

      % mailx -s "Meeting" `cat list` < message

Math

    - The expr utility is available to any shell but is required when
      doing math in the Bourne shell.

        total=`expr $t1 + $t2`

    - The C shell

        @ total = 10
        @ total++
        @ total = 5 + 1

    - Korn Shell

        let i=i+4
        let "i = i + 1"

        if let "total == 100"    OR...   if (( total == 100 ))
        then
            ...
        fi

Debugging

    - Bourne and Korn shell

        #!/bin/sh
        set -v  # verbose, outputs line as typed
        set -x  # echo, outputs line as interpreted
        set +v  # verbose off
        set +x  # echo off
    
    - C shell

        #!/bin/csh -xv # turn echo and/or verbose on
        set verbose    # verbose on
        set echo       # echo on
        unset verbose  # verbose off
        unset echo     # echo off

Testing

    - All shells may use the test utility (use man test).  The test
      utility is also spelled "[".

      -r filename     True if filename exists and is readable.
 
      -w filename     True if filename exists and is writable.
 
      -x filename     True if filename exists and is executable.
 
      -f filename     True if filename exists  and is a regular file.
 
      -d filename     True if filename exists and is a directory.
 
      -h filename     True if filename exists and is a symbolic link.
 
      -c filename     True if filename exists and is a character special file.
 
      -b filename     True if filename exists and is a block special file.
 
      -p filename     True if filename exists and is a named pipe (fifo).
 
      -u filename     True if filename exists and its set-user-ID bit is set.
 
      -g filename     True if filename exists and its set-group-ID bit is set.
 
      -k filename     True if filename exists and its  sticky bit is set.
 
      -s filename     True if filename exists and has a size greater than zero.
 
      -z s1           True if the length of string s1 is zero.
 
      -n s1           True if the length of the string s1 is non-zero.
 
      s1 = s2         True if strings s1 and s2 are identical.
 
      s1 != s2        True if strings s1 and s2 are not identical.
 
      s1              True if s1 is not the null string.
 
      exp1 -eq n2     True if the integers exp1 and exp2 are alge-
                      braically equal.  Any of the comparisons -ne,
                      -gt, -ge, -lt, and -le may be used  in  place
                      of -eq.
 
      -L filename     True if filename exists and is a symbolic link.
 
      !               Unary negation operator.
 
      -a              Binary and operator.
 
      -o              Binary or operator (-a has higher precedence than -o).
 
      (condition)     Parentheses for grouping.  Notice also that
                      parentheses are meaningful to the shell and,
                      therefore, must be quoted.

    - C shell builtin file testing

      -r has read permission
      -w has write permission
      -x has execute permission
      -e exists
      -o is owner
      -z zero size
      -f a normal file

    - Bourne/Korn builtin use same syntax as test utility

Flow Control

    - Bourne and Korn

        # infinite while loop
        while :
        do
            echo "^G Hello"
        done

        # Conditional while/do/done
        while ( test -f flagfile )
        do
            echo "File still there"
        done

        # If construct
        if test $# = 0
        then
            echo "Not enough arguments"
        fi

        # alternate syntax to above
        if [ $# = 0 ] ; then
            echo no arguments
        fi

        if test $# -ne 2
        then
            echo "Usage: crypt  "
            exit
        fi

        if test "$1" = "-o"
        then
            echo "-o option was used"
        fi

        if test "`whoami`" != "root"
        then
            echo "Sorry, you must be root"
        fi

        if test $# -ne 2
        then
            echo "Usage: command arg1 arg2"
            exit
        elif test "`whoami`" != "root"
        then
            echo "Must be root"
            exit
        else
            echo "Ok to proceed"
        fi

        # case construct
        echo -n "Choice: "
        read ans
        case $ans in
            1) echo "Choice 1 entered."
                ;;
            *) echo "Try again"
                ;;
        esac

        # using while loop to read a file in one line at a time
        while read i
        do
            echo "$i"
        done < /etc/passwd

        # for loop
        for filename in `ls`
        do
            echo $filename
        done

        # Repeat something n times
        for filename in 1 2 3 4 5 6 7 8 9 0 1 2
        do
            echo $filename
        done

    - C shell

        # while loop
        @ count = 0
        @ max = 10
        while( $count < $max )
            echo $count
            @ count++
        end

        # foreach loop
        foreach filename ( passwd hosts group )
            echo $filename
        end

        # foreach and a file's contents
        foreach filename ( `cat /etc/passwd` )
            echo $filename
        end

        # switching
        echo -n "Enter Choice: "
        set ans = $<
        switch ( $ans )
            case "1":
                ...
                breaksw
            case "2":
                ...
                breaksw
            case [yY]*:
                echo you answered yes
                breaksw
            default:
                echo -n "Choice selected is not available. "
                goto begin
                breaksw
        endsw

        # Using if to examine command line parameters
        if ( $#argv != 2 ) then
            echo "Usage: command arg1 arg2"
            exit 1
        endif 

        # using if and builtin testing to examine existence of a file
        if( -f addscene ) then
            echo "found file"
        else
            echo "did not find file"
        endif

        # if file does not exist
        if( ! -f addscene ) then
            echo "file does not exist"
        else
            echo "file exists"
        endif

        # search file for string the hard way
        #!/bin/csh
        grep root /etc/passwd
        if ( $status == 0 ) then
            echo yep
        endif

        # search file for string the sleek way
        if { grep root /etc/passwd > /dev/null } then
            echo "found root in /etc/passwd"
        else
            echo "did not find root in /etc/passwd"
        endif

        # if and command exit status
        if ! { /bin/rm dummy } then
            echo "could not remove file"
        else
            echo "file removed"
        endif

        # while and command line arguments
        @ count = 0
        while ( $count <= $#argv )
            echo $argv[$count]
            @ count++
        end

                 ==============================================
                  Miscellaneous Advanced Features and Examples
                 ==============================================

Catching interrupts and trapping

    - Bourne and Korn

        # do a "man -s 5 signal" to list all possible signals
        # 2 is for interrupts (ctrl-c)
        trap "echo you interrupted me!; exit" 2
        sleep 30

    - C shell

        #!/bin/csh
        onintr catch
        echo running
        touch /tmp/tempfile
        sleep 30
        exit 0

        catch:
            echo Caught Interrupt\!  Cleaning up...
            rm /tmp/tempfile

    - Korn Shell

        #!/bin/ksh
        function cleanupAndExit {
            echo cleaning up...
            rm -f /tmp/tempfile
            exit 0
        }

        # Common trap constants
        #   EXIT (the program ends normally or on calling exit)
        #   INT  (user types ctrl-c)
        trap cleanupAndExit INT
        sleep 60.

Functions

    - Bourne and Korn

        hello() {
            echo hello $1
        }

        hello there

    - Korn

        function hello {
            echo hello $1
        }

        hello Fred
    
    - C shell
        No functions (aliases and subshells are as close
        as you can get)

Setuid

    % su
    # vi ckd

    -----------------
    |#!/bin/csh -fb
    |touch /hello

    OR...

    -----------------
    |#!/bin/ksh -p
    |touch /hello

    OR...

    -----------------
    |#!/bin/sh -p
    |touch /hello

    # chmod 4755 ckd
    # exit
    % ckd

Korn shell Autoloading

    $ mkdir -p ~/.autoload/math
    $ cd ~/.autoload/math
    $ vi math_min
    ------------------------------
    |function math_min
    |{
    |   if(( $1 <= $2 )) ; then
    |       print $1
    |   elif (( $2 < $1 )) ; then
    |       print $2
    |   fi
    |
    |   return 0
    |}
    
    $ export FPATH=~/.autoload/math
    $ math_min 5 10
    5

Sed

    # delete line 5 only
    sed '5d' $testdata > $outfile

    # delete lines 5 thru 10
    sed '5,10d' $testdata > $outfile

    # delete lines 45 to the end
    sed '45,$d' $testdata > $outfile

    # delete any line containing the word 'nobody'
    sed '/nobody/d' $testdata > $outfile

    # delete from any line containing 'dog' to
    # the next line containing 'cat'
    sed '/nobody/,/sync/d' $testdata > $outfile

    # delete first line ending with "csh" thru line 23
    sed '/csh$/,23d' $testdata > $outfile

    # replace all occurences of "Diag" or "diag" with "Big Wheel"
    sed 's/[dD]iag/Big Wheel/g' $testdata > $outfile

    # replace ":65534:" with "-->:65534:<--".  The "&" character
    # is replaced with the search pattern.
    sed 's/:65534:/-->&<--/g' $testdata > $outfile

    # Put "EOL" on end of lines starting with first occurrence
    # of "root" down to next line containing "news".  Will
    # repeat if "root" and "news" found again.
    sed '/root/,/news/s/$/& EOF/' $testdata > $outfile

    # delete blank lines (characters between [ and ] are space and tab.
    # i.e. "[]"
    sed '/^[    ]*$/d' $testdata > $outfile

    # do multiple sed edits (need -e for each)
    sed -e 's/root/ROOT/' -e "s/daemon/DEVIL/" $testdata > $outfile

    # Use the '-n' option (no output) and the 'p' (print) option.  Only
    # those lines with "uucp" will be sent to stdout
    sed -n '/uucp/p' $testdata > $outfile

    # use a sed script instead of a command line instruction
    sed -f /tmp/script.sed /tmp/passwd

    # with given data will reverse columns in each line
    Jane 133
    Robert 455
    Mary 98733

    sed 's/\([a-zA-Z]*\) \([0-9]*\)/\2 \1/' /tmp/textfile

Awk/Nawk

    # uses default action, will print each line
    nawk '/.*/' $data

    # use default pattern, will print all lines
    nawk '{ print }' $data

    # use given pattern and action
    nawk '/Fred/ { print $2, $1 }' filename

    # examine sixth column of ps output (both examples do same thing)
    ps -ef | nawk '$6 == "console" { print $0 }'
    ps -ef | nawk '{ if( $6 ~ "console" ) print $0 }'

    # Demonstrates how to pass a shell variable inside an awk statement.
    set name=root
    nawk 'BEGIN { FS=":" } ; /^'$name':/ { print $0 }' /etc/passwd

    # same as above
    awk -F: '/^root/ { print $6 }' /etc/passwd

    # process those lines with a single digit on the end of the line
    nawk '/[0-9]$/ { print $4 "\t" $8 }' datafile

    #####
    # This command demonstrates how to set and use a variable
    nawk '{ total = $8 ; print $1, total }' $data

    #####
    # This command demonstrates how to keep a running total of the
    # eighth field and then print it.
    nawk '{ total += $8 } ; END { print total }' $data

    #####
    # use NR (number of current record)
    nawk '{ print NR, $1 }' $data

    #####
    # use ARGC and ARGV, call like this: nawk -f tryawk 1 2 3
    # The curly brace following "BEGIN" must be on the same line.
    # NOTE: ARGV is not available in awk, just nawk.
    BEGIN {
        print ARGC - 1 " arguments passed"
        print "The first argument is " ARGV[1]
    }

    #####
    # Example of the if command that makes sure user
    # has passed in enough arguments.  Try it like this:
    #   nawk -f tryawk
    # then this:
    #   nawk -f tryawk arg1 arg2
    BEGIN {
        if( ARGC - 1 < 1 )
        {
            print "usage: " ARGV[0] " arg1 arg2"
        }
    }

    #####
    # Example of the while loop.  Prints first four fields of
    # each record using $i variable.
    {
        i = 1
        while( i <= 3 )
        {
            printf( "%s ", $i )
            i++
        }

        printf( "\n" )
    }

    #####
    # This for loop example prints all fields one per line
    {
        for( i = 1 ; i <= NF ; i++ )
            print $i
    }

    #####
    # Example of using an array
    BEGIN { OFS = "\t" }

    # assign first field of each record to list array
    # ($0 is the entire record/line)
    {
        list[NR] = $1
    }

    # when finished reading records display them
    END {
        print "the list of regions is:"
        for( i = 1 ; i < NR ; i++ )
            print list[i]
    }

    #####
    # print records in reverse order
    nawk '{a[NR] = $0 } \
        END { \
            for( c = NR; c > 0; c-- ) \
                print c, a[c] \
        }' cats

    #####
    # Example of how to use words for array indices (associative arrays)
    {
        a[$1]++
    }
    END { for( c in a ) print c, a[c] }

    #####
    # The "contains" relational operator
    nawk '{ if( $2 ~ /E/ ) print $2, $1 }' $data

    #####
    # The "not contains" relational operator
    nawk '{ if( $2 !~ /E/ ) print $2, $1 }' $data

    #####
    # The "==" equality operator
    nawk '{ if( $2 == NW ) print $2, $1 }' $data

    #####
    # The "!=" equality operator
    nawk '{ if( $2 != NW ) print $2, $1 }' $data

    #####
    # The && logical AND operator
    nawk '{ if( $2 ~ /E/ && $2 ~ /W/ ) print $2, $1 }' $data

    #####
    # The logical OR operator
    nawk '{ if( $0 ~ /E/ || $0 ~ /W/ ) print $2, $1 }' $data

    #####
    # The logical AND operator
    nawk '{ if( $2 ~ /E/ && $2 ~ /W/ ) print $2, $1 }' $data

    #####
    # This example of the "next" statement displays all lines
    # where the fifth field is 4.5 or greater.
    nawk '{ if( $5 < 4.5 ) next; print $1 }' $data

    #####
    # Writing your own functions
    function tryit() {
        print "tryit running"
        return 10
    }

    BEGIN { print "tryit() returned: " tryit() }

    # how to use a shell variable within an awk statement
    set count = 5
    awk 'BEGIN { printf ( "%03d\n", '$count' ) ; exit }'