Processes and Multitasking Terminology can be confusing Multiprogramming

Processes and Multitasking Terminology can be confusing Multiprogramming - the system can execute more than one program at the same time Multiuser - system can simultaneously service more than one online terminal Multitasking - system can execute two or more tasks at the same time In common usage, these all refer to the same thing UNIX System Programming

Multitasking Operating Systems Multitasking OSs are designed to perform a complex juggling trick They must: Allocate resources, such as CPU cycles and memory, and assign priorities so each process receives adequate attention UNIX System Programming

Multitasking Operating Systems Higher priority jobs need more or larger CPU time- slices without neglecting lower priority jobs Jobs that are waiting for some resource (such as user input, input from disk, or a shared output such as a printer) need to handled without wasting CPU time UNIX System Programming

Multitasking on a Single CPU Obviously, a single CPU cannot run multiple process simultaneously The OS simulates simultaneity by switching between tasks at a high rate Each switch is a time-slice Since thousands or hundreds of thousands of CPU cycles can go by between user keystrokes, this gives the appearance of simultaneous operation This resource allocation, priority processing, and timeslicing is all done by the scheduler UNIX System Programming

Process States Unix uses several process states to determine the current condition of a process Runnable Stopped Page Wait UNIX System Programming

Process States Non-Interruptable wait Typically for disk I/O or NFS requests Sleeping Idle Terminated UNIX System Programming

Unix Scheduling Algorithm Unix schedules tasks in this order: Highest priority task that is Ready-to-Run and loaded in memory and preempted Ties for priority are broken by time spent waiting (also known as Round-Robin scheduling) If no one is ready to run, the kernel idles until the next time-slice UNIX System Programming

Unix Images and Processes Each process receives a unique numerical process identifier (pid) when it is started Even if the same program is run multiple times, each instance will have a unique PID UNIX System Programming

Unix Images and Processes A process has an image in RAM Sharable Text Segment Data Segment Stack Segment UNIX System Programming Context Saved Registers Open Files Process State

The Shell is a Process The principle process you interact with is the shell The shell can run some commands (built-ins) itself but for most commands, it creates a separate process It usually waits for the command process to finish and then gives you a new shell prompt UNIX System Programming

The Shell is a Process What if you could tell the shell not to wait? You could then instruct the shell to do something else while the first command was running in the background Voila! Multiprocessing in action! UNIX System Programming

Background Execution Caveats Some processes do not lend themselves to background execution vi, for example, expects user input and regular screen output Commands that produce screen displays can create havoc with each other if running concurrently UNIX System Programming

Background Execution Caveats However, Unix processes such as print spooling seem to be made for background processing Background processing should normally only be used with commands that are getting their input from some place other than standard input and writing their output to other than standard output UNIX System Programming

Background Processes Using & The simplest way to create a background process is to invoke the command with a trailing & Example: % sort addresses > mail_list & [1] 21885 % UNIX System Programming

Background Processes Using & This will sort the file addresses in the background and place the results in the file mail_list 21885 is the unique pid of this instance of the sort process [1] is the job number that is assigned by the current shell UNIX System Programming

wait What if you initiate several background processes and then you decide that you need to wait until they have all finished before doing something? UNIX System Programming

wait will make the shell wait on all the background processes to complete before giving you another prompt If wait immediately returns a prompt, there are no processes executing in the background UNIX System Programming

show Proces. Ses - ps Common options: a - include processes that are not owned by you x - show processes that don’t have a controlling terminal r - show only running processes f – full details u – for a specific user e - everyone Example: clyde% ps -a UNIX System Programming

Example of ps ps -a PID TT STAT 26129 p 1 S 26632 p 1 R 25408 p 2 IW TIME COMMAND 0: 00 -csh (csh) 0: 00 ps -a 0: 00 -csh (csh) 25937 p 2 IW 0: 00 vim lab 3 S -> Sleeping R -> Running IW -> Idle (and swapped out) UNIX System Programming

jobs Lists the active jobs under shell job control Syntax: jobs [-l] -l List process IDs in addition to normal information clyde% [1] [2] [3] + clyde% jobs Running fist. scr Stopped (tty output) vi Stopped (tty input) ed + is the current process - is the previous job UNIX System Programming

Controlling Background Processes Commands available for job control bg - background fg - foreground stop kill ^Z Reference to a job begins with a %. By itself, % refers to the current job (and most of the commands default to the current job) UNIX System Programming

Back. Ground - bg Run the current job (or job_no) in the background Syntax: bg [%job_no] - the job number assigned when the task was initially put in the background, can be viewed with the jobs command job_no Example: bg %3 will make job number 3 resume execution in the background UNIX System Programming

stop Stop the current or specified background job Syntax: stop [%job_no] - the job number assigned when the task was initially put in the background, can be viewed with the jobs command job_no Example stop %1 will stop background job number 1 until either an fg or bg command is executed UNIX System Programming

^Z or Ctrl-Z ^Z stops the current foreground job and puts it in the background To re-enable execution, either use bg %job_no or fg %job_no Example %vi wordcount. c ^Z will suspend execution of this instance of vi and place it into the background until an fg command is executed UNIX System Programming

kill - signal a process kill is somewhat strangely named Syntax: kill [-sig_no] pid kill -l (display list of signals) -sig_no - signal number to send pid - process id of process to receive signal Default sig_no is 15, or request-process-termination UNIX System Programming

kill - signal a process kill -9 pid terminates the process with extreme prejudice As usual, you can only kill your own processes unless you are the superuser UNIX System Programming

nice - Modify Process Priority nice increments or decrements the process priority value Syntax: nice [+n|-n] [command] +n|-n - increments or decrements the process priority value by n UNIX System Programming

nice - Modify Process Priority The higher the process priority value, the lower the priority Only the superuser can lower the process priority value (raise the process priority) UNIX System Programming

find - Find files find recursively descends the directory hierarchy for each pathname in in the pathname-list looking for files that have certain filenames, permissions, or modifications times. find can also execute a program for each file it locates. UNIX System Programming

find - Find files Syntax: find directory-list expression - the pathnames of one or more directories to be searched. All subdirectories of all listed directories are searched to all levels. directory-list - a logical expression of one or more criteria as described later. A space only separating criteria is a logical AND operator. A -o separator is a logical OR operator. An ! negates any criteria it precedes. expression UNIX System Programming

Caveats Special characters within expression must be escaped () to keep the shell from interpreting them. These are characters such as parenthesis (), square brackets [ ], question marks ? , and asterisks *. UNIX System Programming

Caveats Each element within expression is a separate argument and must be separated from the others with spaces. There must be a space on both sides of each parenthesis, exclamation point, criterion, or other element. When you escape special characters with a , the spaces go on each side of the character pair [ UNIX System Programming

find Criteria -name filename matches if the file being evaluated matches filename -type filetype matches if the file has a matching file type d - directory f - ordinary l - symbolic link UNIX System Programming

find Criteria -links +/-n -user name matches if the file has the number of links specified by +/-n matches if the file is owned by name (login name) -group name matches if file belongs to group UNIX System Programming

find Criteria -inum n matches if the files inode number is n -size +/-n[c] matches if file size is the size, in blocks, specified by +/-n, or characters if +/-nc UNIX System Programming

find Criteria -atime +/-n matches if the file was last accessed the number of days ago specified by +/-n. The access time of searched directories is changed when this option is used. -mtime +/-n matches if the file was last modified +/-n days ago UNIX System Programming

find Criteria -newer filename matches if file is newer than filename -print all files match, causes find to print the pathname UNIX System Programming

find Criteria -ls all files match, similar to -print but displays more information such as inode number, size, permissions, number of lines, etc. UNIX System Programming

find Criteria -exec command ; included at the end of a criteria list, command is executed on all files that meet the preceding criteria. Must be terminated with an escapes semicolon (; ). {} within the command string represents the name of the file being evaluated. E. g. , find ……. –exec rm { } ; UNIX System Programming

find Criteria -ok command ; same as -exec only it displays the commands about to be executed and waits for a y from standard input before proceeding UNIX System Programming

More Caveats You must explicitly tell find to display matching file names with the -print criteria. You can use a -a (-o)between criteria for logical AND (OR) conditions. + is equivalent to greater than (>) - equates to less than (<) UNIX System Programming

find Examples find. -name a* finds all files in your current directory and all of its subdirectories with names beginning with an a. However, since no -print criteria was given, it does not display them. find. -name a* -print same as before but prints the pathnames of the files it finds UNIX System Programming

find Examples find. -type f -mtime -1 ! -name *. o -print | cpio -o. B > /dev/rmt 0 finds all files that have been modified within the past day (-mtime -1) excluding object files (! -name *. o) and pipes them to an archive program (cpio) UNIX System Programming

find Examples find. ( -name core -o -name junk ) -print -exec rm {} ; used to clean up a disk, finds all files whose name is core OR junk (notice the -o), displays their pathname, and deletes (rm) them if used with -ok instead of -exec, find will prompt you with an > character and wait for you to enter a y before deleting each file UNIX System Programming

find Examples find. ( -name *. cc -o -name a. out ) -print –ok chmod 700 {} ; find. -name [0 -9]* -ok chmod 700 {} ; UNIX System Programming

What does this do? % chown –R us. /base All your base are belong to us UNIX System Programming

Multi-Command Lines Multiple commands can be entered on a single line by separating them with a ; wc chapter. * ; who | wc will operate on chapter. *, then wc will operate on who UNIX System Programming

Multi-Command Lines You can also group commands using parenthesis (wc chapter. * ; who) | wc The first wc will operate on chapter. *, then who will run, and then the final wc will operate on the combined output of (wc chapter. * ; who) The character, with space separators, can be used to extend commands across multiple lines UNIX System Programming

Multi-line Command Example who | wc is equivalent to who | wc Q. What happens if you do: who ? (i. e. , a space and enter follows ) UNIX System Programming

show file DIFFerences - diff Displays the differences between two files Syntax: diff [ -biw ] [-c | -e ] file-name 1 file-name 2 -b - ignore trailing blanks (spaces and TABs at end of and treat all other strings of blanks as equivalent UNIX System Programming line)

show file DIFFerences - diff -i - ignore the case of letters -w -ignore all blanks (spaces and TABs); i. e. , a=b will compare equal to = a b -c - produce listing of differences with lines of context UNIX System Programming

list UNIQue items - uniq Examines a file for duplicate lines. Duplicate lines are only detected if they are consecutive. Syntax: uniq [ -cdu] [ +|-n ] input-file [ output-file] -c - Supercede the -u and -d options and generate an output report with each line preceded by an occurrence count -d - write one copy of only the duplicated lines UNIX System Programming

list UNIQue items - uniq -u - write only those lines which are not duplicated NOTE: the default output is the union (combination) of -d and -u The n arguments specify an initial portion of each line to skip +n - skip the first n characters -n - skip the first n fields and any blanks before them UNIX System Programming

C Shell Variables There are two types of shell variables: Environmental shell variables Ordinary shell variables These both have variables that either take on values or act as switches Switches are on if they are declared and off if they are not UNIX System Programming

C Shell Variables Environmental shell variables are inherited by all child shells while ordinary shell variables must be defined for each instance of the shell Shell variables are usually defined in either the. login or the. cshrc files UNIX System Programming

What Shell Variables Are In Use? set will display all ordinary shell variables and there values setenv will display all currently defined environmental variables and their values Traditionally, the ordinary variables have lowercase names and the environmental variables are all uppercase UNIX System Programming

Setting Variables Environmental shell variables are set using setenv variable[=value] Ordinary shell variables are set using set variable[=value] UNIX System Programming

Setting Variables Note that the text shows spaces on both sides of the =. This will not work. To unset a defined shell variable, use the unset and unsetenv commands unset variable unsetenv variable UNIX System Programming

Referring to Shell Variables To refer to shell variables, preface the variable name with a $ Examples echo $HOME echo This is the value of the history variable: $history echo environment path is: My $PATH echo current path is: My $path echo favorite shell, My $shell UNIX System Programming a great shell. is

Common Environmental Variables HOME - your home directory as defined in /etc/passwd PATH - your directory search path TERM - your terminal type SHELL - your default shell, used when forking other shells USER - your user name UNIX System Programming

Common Ordinary Shell Variables history - enables the history function and defines number of commands to savehist - specifies the number of commands saved when logging out shell - pathname of the shell home - your home directory prompt - sets the prompt string UNIX System Programming

Common Ordinary Shell Variables echo - if set, the shell displays each command before executing it (switch) filec - enables file name completion (switch) ignoreeof - tells shell to ignore ^D so you must use exit or logout to leave a shell (switch) noclobber - prevents you from accidentally overwriting a file when you redirect output (switch) UNIX System Programming

Redirecting Standard Input Redirection of standard input works in a similar manner < is the standard input redirection operator command [options] [arguments] < input_filename UNIX System Programming

Redirecting Standard Input options and arguments are the same as you are familiar with command is any executable program input_filename is the place (file) that contains the data you want to use for input to command Obviously, input_filename must exist UNIX System Programming

Using Input and Output Redirection Standard input and standard output redirection can be used together on the same command [options] [arguments] < input_filename > output_filename E. g. , crypt < password > cryptext sort < unsorted_file > sorted_file UNIX System Programming

Using Input and Output Redirection or command [options] [arguments] < input_filename >> output_filename E. g. , foo < input_file >> output_file UNIX System Programming

Redirecting Standard Error Under the C shell, you can redirect both standard output and standard error by using the > or >> symbols followed by an & UNIX System Programming

Redirecting Standard Error If you don’t do this, commands like find may clutter up your screen with error messages while you are trying to accomplish other work. find / -name stdlib. c -print >& whereitis or find / -name stdlib. c -print >>& whereitis UNIX System Programming

Pipes (Inter-Process Communication) Suppose you want a printed sorted list of who is currently logged on to the system You could issue the following command sequence who > temp_file sort < temp_file > sorted_file lpr sorted_file rm temp_file rm sorted_file UNIX System Programming

Pipes A pipe will let you do this in one command line without having to use temporary files who | sort | lpr UNIX System Programming

So What Is a Pipe? The shell uses a pipe ( | ) to connect the standard output of one command directly to the standard input of another This has the same effect of redirecting the standard output of one command to a temporary file, and then using that file as the standard input to another command UNIX System Programming

So What Is a Pipe? You can use pipes with any command that accepts input from either a file specified on the command line or standard input Pipes can also be used with commands that only accept data from standard input UNIX System Programming

So What Happens in a Pipe? Consider the command string ls | wc -l The shell opens a pipe for communication between ls and wc are then executed as concurrently running tasks UNIX System Programming

So What Happens in a Pipe? As ls writes data to the pipe, wc can read the data from the pipe Neither process knows that it is using a pipe instead of a standard file UNIX System Programming

Pipes Because the processes are running concurrently and not writing to intermediate files, the entire sequence is more efficient and should complete quicker. UNIX System Programming

Pipes If the writing process produces data much more quickly than the reading process and accept it, the pipe can fill up In that case, the kernel puts the writing process to sleep until the reader has a chance to catch up UNIX System Programming

Common Mistakes Confusing redirection and pipes is a common error that can be deadly to your files A pipe ( | ) will take the output from a command connect it to the input of another command Redirection ( > ) will take the output from a command create or overwrite a file with the data UNIX System Programming

tee Copy standard input to standard output and one or more files Unix Command Standard output file-list UNIX System Programming

tee Syntax: tee [ -ai ] file-list -a - append to output file rather than overwrite, default -i is to overwrite (replace) the output file - ignore interrupts file-list - one or more file names for capturing output UNIX System Programming

tees and Pipes So, what happens if you use tee in a pipe? ls | tee dir_list | sort -r What will this do? Why would I want to do it? UNIX System Programming

tees and Pipes What about this ? a. out | tee output_file Why do this? UNIX System Programming

Filters A filter is a command that process an input stream of data (from standard input) to produce an output stream (on standard output. ) A command line that uses a filter use a pipe to connect the filter’s input to the standard output of another command or filter UNIX System Programming

Filters Another pipe may connect the filter’s standard output to another command or filter’s standard input Interactive utilities, such as mail and vi, cannot be used as filters UNIX System Programming

Commonly Used Filters sort - sort a file tr - translate characters to different characters spell - check the spelling of a list of words wc - count the number of words, characters and lines in a file UNIX System Programming

Commonly Used Filters grep - search files for a pattern head and tail - list first/last part of a file sed - stream editor awk - pattern-action pair programming language UNIX System Programming

Filter Examples who | sort - display a sorted list of who is logged on ls | wc -w - display the number of entries in a directory tr A-Z a-z <myfile >yourfile - make all characters in myfile lower case and saves output in yourfile UNIX System Programming

TRanslate - tr Copies standard input to standard output with substitution or deletion of selected characters Syntax: tr [ -ds ] [ string 1 ] [ string 2 ] -d - delete all input characters contained in string 1 UNIX System Programming

TRanslate - tr -s - squeeze all strings of repeated output characters that are in string 2 to single characters tr provides only simple text processing. It does not allow the full power of regular expressions. If you need the power of regular expressions, you need to use sed UNIX System Programming

TRanslate - tr reads from standard input. Any character that does not match a character in string 1 is passed to standard output unchanged Any character that does match a character in string 1 is translated into the corresponding character in string 2 and then passed to standard output UNIX System Programming

TRanslate - tr Examples tr ‘s’ ‘z’ replaces all instances of s with z tr ‘so’ ‘zx’ replaces all instances of s with z and o with x tr ‘a-z’ ‘A-Z’ replaces all lower case characters with upper case characters UNIX System Programming

tr As you can see, tr establishes a single character to single character (or 1 -to-1) mapping. This mapping is case sensitive. The output of tr can be redirected to a file or piped to another filter and its input can be redirected from a file or piped from another command UNIX System Programming

tr This implies that certain characters must be protected from the shell by quotes or , such as: spaces : ; & ( ) | ^ < > [ ] ! newline TAB Example tr o replaces all o’s with a blank (space) UNIX System Programming

tr Non-printing characters can also be specified Bell 7 Backspace 10 CR Escape 33 Formfeed 14 Newline 12 TAB 11 15 UNIX System Programming

tr string 1 and string 2 can use ranges of characters as follows: tr a-z A-Z translates all lower case to upper case tr a-m A-M translates only lower case a through m to upper case A UNIX System Programming though M

tr Ranges must be in ascending ASCII order What would happen if: tr 1 -9 9 -1 < some_file UNIX System Programming

tr The tr -d option lets you delete any character matched in string 1. string 2 is not allowed with the -d option Examples tr -d a-z deletes all lower case characters tr -d aeiou deletes all vowels. UNIX System Programming

tr tr -dc aeiou deletes all characters except vowels (note: this includes spaces, well) TABS, and newlines as Normal usage would be with pipes or redirection tr -d 15 < in_file > out_file UNIX System Programming

tr Example from the tr man page. How does this work? tr -cs A-Za-z ‘12’ <in_file > out_file It replaces all characters that are not a letter (-c) with a newline ( 12 ) and then squeezes multiple newlines into a single newline (-s) UNIX System Programming

More tr Examples The following commands are equivalent tr abcdef [a-c][d-f] xyzabc [x-z][a-c] This command implements the rotate 13 encryption tr ‘[A-M][N-Z][a-m][n-z]’‘[N-Z][A-M][n-z][a-m]’ < temp UNIX System Programming

More tr Examples To make the text intelligible again, reverse the arguments tr [N-Z][A-M][n-z][a-m] [A-M][N-Z][a-m][n-z] The -d option will cause tr to delete selected characters echo If you can read this you can spot the missing vowels | tr -d aeiou UNIX System Programming

tr Gotchas The syntax varies between BSD and System V BSD uses a-z System V uses [a-z] System V allows [x*n] to indicate n occurrences of x If you leave out n, then x is duplicated as many times as necessary UNIX System Programming

tr Gotchas In BSD, if string 2 does not contain as many characters as string 1, the last character of string 2 is duplicated as many times as necessary System V fails to translate these additional characters Would tr solve the lab problem of capitalizing all the initial characters of each word in a sentence? Try it! UNIX System Programming

What If? Suppose your boss wants a tool that will: Accept any number of files to check, report each line that has doubled words, highlight the doubled words (using ANSI escape sequences) and display the file name with each line. UNIX System Programming

What If? Work across lines even when the word at the end of a line is repeated at the beginning of the next line Find doubled words despite capitalization, punctuation, or amount of separating whitespace Find doubled words even if they are separated by HTML tags UNIX System Programming

What If? Wow, this sounds like a BIG programming job. Better not make any plans for this weekend! UNIX System Programming

You Can Have Your Weekend Back! If you know regular expressions, you can have your weekend back Using the regular expression engine from Perl, this problem can be solved in only a few lines of code UNIX System Programming

You Can Have Your Weekend Back! $/ = ‘. n’; while (<>) { next if !s/b([az]+)((s|<[^>]+>)+)(1b)/e[7 m$1e[m$2e[7 m$4e[m/ig; s/^([^e]*n)+//mg; s/^/$ARGV: /mg; print; Code was written by Jeffrey E. F. Friedl as an example in Mastering Regular Expressions O’Reilly Publishing } UNIX System Programming