Tool Time Session 3: Unix Text Tools

> grep '(\?[0-9]\{3\})\?-[0-9]\{3\}-[0-9]\{4\}' phone-numbers.txt
(218)-589-6764
(507)-209-5649
952-474-0698
612-266-0909
...

> grep -E '\(?[0-9]{3}\)?-[0-9]{3}-[0-9]{4}' phone-numbers.txt 
(218)-589-6764
(507)-209-5649
952-474-0698
612-266-0909
(218)-781-1788
...

> grep -E '\(?[0-9]{3}\)?-[0-9]{3}-[0-9]{4}' phone-numbers-irregular.txt 
(218)-589-6764 Landline, from Dalton, MN(state),USA (507)-209-5649
952-474-0698 Landline, from Minneapolis, MN(state),USA 612-266-0909
Landline, from Saint Paul, MN(state),USA (218)-781-1788 Landline, from
(507)-510-6175 Landline, from Sherburn, MN(state),USA 952-843-4789
Landline, from Minneapolis, MN(state),USA 320-254-3105 Landline, from
...

> grep -o -E '\(?[0-9]{3}\)?-[0-9]{3}-[0-9]{4}' phone-numbers-irregular.txt
(218)-589-6764
(507)-209-5649
952-474-0698
612-266-0909
...

> grep -l -E '\(?[0-9]{3}\)?-[0-9]{3}-[0-9]{4}' phone-numbers* gettysburg.txt 
phone-numbers-irregular.txt
phone-numbers.txt
> 

> grep -r -l -E '\(?[0-9]{3}\)?-[0-9]{3}-[0-9]{4}' .
./phone-numbers.txt
./phone-numbers-irregular.txt
./search-dir/subdir/phone-numbers-irregular.txt
./search-dir/phone-numbers.txt
./search-dir/phone-numbers-irregular.txt

>  ls 
cs123  cs456  cs789  names-files

>  grep -r -i farrar cs*
cs456/grades-CS456.csv:Mi Farrar,farrar@college.edu,1269,28.64
cs789/grades-CS789.csv:Mi Farrar,farrar@college.edu,3708,68.23

>  grep -r -i blea cs*
cs789/grades-CS789.csv:Meri Blea,blea@college.edu,155,3.24

>  grep -r -i mcnelly cs*
cs123/grades-CS123.csv:Olympia Mcnelly,mcnelly@college.edu,1628,97.64

wget https://amyklobuchar.com/ -r -k -p 

> sed 'pattern1 operation1; pattern2 operation2; ...' file1.txt file2.txt

# default no transformations, original lines
> sed '' gettysburg.txt |head -3
Four score and seven years ago our fathers brought forth on this continent, a
new nation, conceived in Liberty, and dedicated to the proposition that all men
are created equal.

# matches only line 1, substitue 'F' for 'P'
> sed '1 s/F/P/' gettysburg.txt |head -3
Pour score and seven years ago our fathers brought forth on this continent, a
new nation, conceived in Liberty, and dedicated to the proposition that all men
are created equal.

# matches only line 1, substitue 'F' or 'f' for 'P' for first occurrence
> sed '1 s/[Ff]/P/g' gettysburg.txt |head -3
Pour score and seven years ago our Pathers brought Porth on this continent, a
new nation, conceived in Liberty, and dedicated to the proposition that all men
are created equal.

# matches all lines, substitue 'F' or 'f' for 'P' globally (all occurrences)
> sed 's/[Ff]/P/g' gettysburg.txt |head -3
Pour score and seven years ago our Pathers brought Porth on this continent, a
new nation, conceived in Liberty, and dedicated to the proposition that all men
are created equal.

# 2 actions, swapping 'Ff' for 'P' globally, swapping 'Pp' for 'F' globally
> sed 's/[Ff]/P/g; s/[Pp]/F/g' gettysburg.txt |head -3
Four score and seven years ago our Fathers brought Forth on this continent, a
new nation, conceived in Liberty, and dedicated to the FroFosition that all men
are created equal.

# don't print by default -n, print lines 6-12
> sed -n '6,12p' gettysburg.txt 
nation so conceived and so dedicated, can long endure. We are met on a great
battle-field of that war. We have come to dedicate a portion of that field, as a
final resting place for those who here gave their lives that that nation might
live. It is altogether fitting and proper that we should do this.

But, in a larger sense, we can not dedicate -- we can not consecrate -- we can
not hallow -- this ground. The brave men, living and dead, who struggled here,

# print only lines 13 and 19
> sed -n '13p; 19p' gettysburg.txt 
have consecrated it, far above our poor power to add or detract. The world will
they gave the last full measure of devotion -- that we here highly resolve that

# print only lines which match regex 'dead' : grep-like behavior
> sed -n '/dead/p' gettysburg.txt 
not hallow -- this ground. The brave men, living and dead, who struggled here,
that from these honored dead we take increased devotion to that cause for which
these dead shall not have died in vain -- that this nation, under God, shall

# find all .html files and search them for the phone# pattern 123-456-7890
> find . -name '*.html' -exec grep -E '([0-9]{3})-([0-9]{3})-([0-9]{4})' {} \;
...
        <strong>Phone:</strong> 603-668-4321<br>
        <strong>Phone:</strong> 603-352-1234<br>
        <strong>Phone:</strong> 603-668-4321<br>
...

# find all .html files, run sed on them
# sed will replace xxx-xxx-xxxx with (xxx) xxx-xxxx in the file
# sed will make a backup of the original with the extension '.bk' 

> find . -name '*.html' -exec sed -E -i.bk 's/([0-9]{3})-([0-9]{3})-([0-9]{4})/\(\1\) \2-\3/g' {} \;

# search for files with original pattern of phone number
> find . -name '*.html' -exec grep -E -o '[0-9]{3}-[0-9]{3}-[0-9]{4}' {} \;

# no matches reported

# search for files with new pattern of phone number
> find . -name '*.html' -exec grep -E '\([0-9]{3}\) [0-9]{3}-[0-9]{4}' {} \;
...
        <strong>Phone:</strong> (603) 352-1234<br>
        <strong>Phone:</strong> (603) 668-4321<br>
        <strong>Phone:</strong> (603) 668-4321<br>
...

# shell script wizardry to move all backup files to their originals
> for f in $(find . -name '*.bk');do mv $f ${f/.bk/};done

> cat hello.awk 
#!/bin/awk -f

BEGIN{
  print "Hello world!"
}
> awk -f hello.awk 
Hello world!

#!/bin/awk -f

/dead/ {                        #matches lines with 'dead'
  print "This line is morbid:",$0
}

$2 == "hallow" {                  #first 'field' is "dead"
  print "This line is holy:  ",$0
}

NR==3, NR==7 {                  #matches lines 3 to 7
  print "Thes lines are in the middle",NR,":",$0
}

> awk -f patterns.awk gettysburg.txt 
Thes lines are in the middle 3 : are created equal.
Thes lines are in the middle 4 : 
Thes lines are in the middle 5 : Now we are engaged in a great civil war, testing whether that nation, or any
Thes lines are in the middle 6 : nation so conceived and so dedicated, can long endure. We are met on a great
Thes lines are in the middle 7 : battle-field of that war. We have come to dedicate a portion of that field, as a
This line is morbid: not hallow -- this ground. The brave men, living and dead, who struggled here,
This line is holy:   not hallow -- this ground. The brave men, living and dead, who struggled here,
This line is morbid: that from these honored dead we take increased devotion to that cause for which
This line is morbid: these dead shall not have died in vain -- that this nation, under God, shall

#!/bin/awk -f
{
  for(i=1; i<=NF; i++){
    print "Line",NR,"Field",i,":",$i;
  }
}

> awk -f printfields.awk gettysburg.txt 
Line 1 Field 1 : Four
Line 1 Field 2 : score
Line 1 Field 3 : and
Line 1 Field 4 : seven
Line 1 Field 5 : years
Line 1 Field 6 : ago
Line 1 Field 7 : our
Line 1 Field 8 : fathers
Line 1 Field 9 : brought
Line 1 Field 10 : forth
Line 1 Field 11 : on
Line 1 Field 12 : this
Line 1 Field 13 : continent,
Line 1 Field 14 : a
Line 2 Field 1 : new
Line 2 Field 2 : nation,
Line 2 Field 3 : conceived
Line 2 Field 4 : in
...

> awk -F , -f printfields.awk grades.csv |head
Line 1 Field 1 : Henrietta Gamez
Line 1 Field 2 : gamez@college.edu
Line 1 Field 3 : 9240
Line 1 Field 4 : 59.39
Line 2 Field 1 : Lang Singleton
Line 2 Field 2 : singleton@college.edu
Line 2 Field 3 : 3063
Line 2 Field 4 : 57.89
...

#!/bin/awk -f

# frequency.awk: calculates the frequency of each word that appears in
# a text file and prints it out (in unsorted order). Leverages awk's
# built-in associative arrays along with several other features.

{                               # match every line
  gsub(/[^a-zA-Z ]/," ");       # eliminate non-word characters like ',' and '!'
  for(i=1; i<=NF; i++){         # iterate over each word
    counts[$i] = counts[$i]+1   # use ith word as a key, increment its frequency
  }                             # leverages built-in semantics: not present -> 0
}
END{                            # after processing all lines
  for(key in counts){           # iterate over all keys in the counts array
    print key,":",counts[key]   # print the key (word) and its count
  }
}

# Run script on gettysburg.txt, shows word frequency in unsorted order

> awk -f frequency.awk gettysburg.txt 
God : 1
detract : 1
honored : 1
before : 1
their : 1
people : 3
...
Lincoln : 1
proper : 1
who : 3
which : 2

# pipe results to sort and ask to sort on 3rd field (count) in reverse
# numeric order to get top 10 most frequent words

> awk -f frequency.awk gettysburg.txt | sort -k 3rn |head
that : 13
the : 9
here : 8
to : 8
we : 8
a : 7
and : 6
can : 5
for : 5
have : 5