From 32324580098bf7075968a76b3bb5d1a93ea75e2f Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 13:16:31 -0400 Subject: [PATCH 01/14] Added a reference topic for I/O redirection --- _commands/basics/io_redirection.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 _commands/basics/io_redirection.md diff --git a/_commands/basics/io_redirection.md b/_commands/basics/io_redirection.md new file mode 100644 index 000000000..9942e37e2 --- /dev/null +++ b/_commands/basics/io_redirection.md @@ -0,0 +1,17 @@ +--- + +I/O Redirection +--------------- + +We often want to redirect input and output to and from files. The I/O redirection operators `<`, `>`, and `>>` allow us to do this. + +~~~ bash +$ echo "I love bash!" > message.txt +$ cowsay -w < message.txt +$ echo "I love bash so much!" >> message.txt +$ cosway -w < message.txt +~~~ + + + +### Good stuff coming From 86cbdc4460d91b77ab267676e3bcea050cacdc46 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 14:09:29 -0400 Subject: [PATCH 02/14] Moved I/O redirection to the tutorials folder --- _commands/{basics => tutorials}/io_redirection.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) rename _commands/{basics => tutorials}/io_redirection.md (56%) diff --git a/_commands/basics/io_redirection.md b/_commands/tutorials/io_redirection.md similarity index 56% rename from _commands/basics/io_redirection.md rename to _commands/tutorials/io_redirection.md index 9942e37e2..e3a4027b2 100644 --- a/_commands/basics/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -3,7 +3,7 @@ I/O Redirection --------------- -We often want to redirect input and output to and from files. The I/O redirection operators `<`, `>`, and `>>` allow us to do this. +The I/O redirection operators `<`, `>`, and `>>` allow us to read and write input and output from files. ~~~ bash $ echo "I love bash!" > message.txt @@ -14,4 +14,7 @@ $ cosway -w < message.txt -### Good stuff coming +### Basic Usage + +#### `<` operator + From 8707649f1597008e0578f81fa06a280ca2d2c7d3 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 14:30:45 -0400 Subject: [PATCH 03/14] Added basic usage section --- _commands/tutorials/io_redirection.md | 41 +++++++++++++++++++++++++++ test.txt | 1 + 2 files changed, 42 insertions(+) create mode 100644 test.txt diff --git a/_commands/tutorials/io_redirection.md b/_commands/tutorials/io_redirection.md index e3a4027b2..1a9ff7fd4 100644 --- a/_commands/tutorials/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -16,5 +16,46 @@ $ cosway -w < message.txt ### Basic Usage +#### `>` operator +Puts the output from the program on the left of the operator into the file named on the right of the operator. Will overwrite all of current contents of the file, and will create the file if it doesn't exist yet. + +~~~ bash +$ echo "Text in a file" > message.txt +$ cat message.txt +Text in a file +~~~ + +#### `>>` operator +Appends the output from the program on the left of the operator to the contents of the file named on the right of the operator. Will not overwrite any of the current contents of the file, and will create the file if it doesn't already exist. + +~~~ bash +$ echo "Text in a file" >> message.txt +$ echo "More text in a file" >> message.txt +$ cat message.txt +Text in a file +More text in a file +~~~ + #### `<` operator +Feeds the contents of the file on the right into the standard input stream of the program on the left. Will not modify any of the contents of the file, and will throw an error if it doesnt already exits. + +~~~ bash +$ echo "I love bash" > message.txt +$ cowsay -w < message.txt + + _____________ +< I love bash > + ------------- + \ ^__^ + \ (OO)\_______ + (__)\ )\/\ + ||----w | + || || +~~~ + +### Advanced Usage: Standard Streams + + + + diff --git a/test.txt b/test.txt new file mode 100644 index 000000000..9daeafb98 --- /dev/null +++ b/test.txt @@ -0,0 +1 @@ +test From fe43e5eab04ba2b815c8ac775fd1482e7aceeab2 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 19:05:54 -0400 Subject: [PATCH 04/14] Added documentation for the pipe operator | --- _commands/tutorials/io_redirection.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/_commands/tutorials/io_redirection.md b/_commands/tutorials/io_redirection.md index 1a9ff7fd4..aec42e96d 100644 --- a/_commands/tutorials/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -3,13 +3,14 @@ I/O Redirection --------------- -The I/O redirection operators `<`, `>`, and `>>` allow us to read and write input and output from files. +The I/O redirection operators `>`, `>>`, `<`, and `|` allow us to read and write input and output from files. ~~~ bash $ echo "I love bash!" > message.txt $ cowsay -w < message.txt $ echo "I love bash so much!" >> message.txt $ cosway -w < message.txt +$ cat message | grep "bash" ~~~ @@ -37,12 +38,11 @@ More text in a file ~~~ #### `<` operator -Feeds the contents of the file on the right into the standard input stream of the program on the left. Will not modify any of the contents of the file, and will throw an error if it doesnt already exits. +Feeds the contents of the file on the right into the input of the program on the left. Will not modify any of the contents of the file, and will throw an error if it doesnt already exist. ~~~ bash $ echo "I love bash" > message.txt $ cowsay -w < message.txt - _____________ < I love bash > ------------- @@ -53,9 +53,13 @@ $ cowsay -w < message.txt || || ~~~ -### Advanced Usage: Standard Streams +#### `|` operator +Feeds the output of the command on the left into the input of the command on the right. +### Advanced Usage: Standard Streams +At their core, the I/O redirection operators are interacting with the standard input and output streams, `stdin`, `stdout`, and `stderr`. +### `stdin` From f5bb5b667d769814397ecf6aa0acbff765c65749 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 19:16:59 -0400 Subject: [PATCH 05/14] Setting up section on standard streams --- _commands/tutorials/io_redirection.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/_commands/tutorials/io_redirection.md b/_commands/tutorials/io_redirection.md index aec42e96d..786a613fa 100644 --- a/_commands/tutorials/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -3,7 +3,7 @@ I/O Redirection --------------- -The I/O redirection operators `>`, `>>`, `<`, and `|` allow us to read and write input and output from files. +The I/O redirection operators `>`, `>>`, `<`, and `|` allow us to read and write input and output from files, and pass output from one program to another. ~~~ bash $ echo "I love bash!" > message.txt @@ -57,7 +57,7 @@ $ cowsay -w < message.txt Feeds the output of the command on the left into the input of the command on the right. -### Advanced Usage: Standard Streams +### Standard Streams At their core, the I/O redirection operators are interacting with the standard input and output streams, `stdin`, `stdout`, and `stderr`. ### `stdin` From c5ac6c531b22d211bdb50d5fd67f9aec5730f252 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 21:10:42 -0400 Subject: [PATCH 06/14] Completed basic pipe example --- _commands/tutorials/io_redirection.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/_commands/tutorials/io_redirection.md b/_commands/tutorials/io_redirection.md index 786a613fa..ed3653e56 100644 --- a/_commands/tutorials/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -56,6 +56,10 @@ $ cowsay -w < message.txt #### `|` operator Feeds the output of the command on the left into the input of the command on the right. +~~~ bash +$ echo "Pipes are fun!" | grep -c "Pipe" +1 +~~~ ### Standard Streams At their core, the I/O redirection operators are interacting with the standard input and output streams, `stdin`, `stdout`, and `stderr`. From e55b50936a6679ed6a10601c9e739a988ff17402 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 22:09:41 -0400 Subject: [PATCH 07/14] Added first two stream selection examples --- _commands/tutorials/io_redirection.md | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/_commands/tutorials/io_redirection.md b/_commands/tutorials/io_redirection.md index ed3653e56..99a3e5a27 100644 --- a/_commands/tutorials/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -61,9 +61,27 @@ $ echo "Pipes are fun!" | grep -c "Pipe" 1 ~~~ -### Standard Streams -At their core, the I/O redirection operators are interacting with the standard input and output streams, `stdin`, `stdout`, and `stderr`. +### Useful Examples -### `stdin` +#### Stream Selection +By default, the `>`, `>>`, and `|` operators all operate on the standard output stream `stdout`, and leave the standard error stream `stderr` untouched. If we want to redirect just the error messages from a program, we can do so by selecting stream 2: +~~~ bash +$ git commit +fatal: Not a git repository (or any of the parent directories): .git +$ git commit > git_errors.txt +$ cat git_errors.txt # will generate no output +$ git commit 2> git_errors.txt +$ cat git_errors.txt +fatal: Not a git repository (or any of the parent directories): .git +~~~ + +We can't directly change the input source of the `|` operator in this fashion, it always reads from `stdout`. If we want to pipe `stderr` only, we can first point `stderr` at `stdout` and then point `stdout` at nothing: + +~~~ bash +$ git commit +fatal: Not a git repository (or any of the parent directories): .git +$ git commit 2>&1 1>/dev/null | grep -c "fatal" +1 +~~~ From d462cd4556cad2c7ecd1914e07a8953702cff50b Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 22:24:47 -0400 Subject: [PATCH 08/14] Completed stream selection example --- _commands/tutorials/io_redirection.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/_commands/tutorials/io_redirection.md b/_commands/tutorials/io_redirection.md index 99a3e5a27..bf8700571 100644 --- a/_commands/tutorials/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -61,7 +61,7 @@ $ echo "Pipes are fun!" | grep -c "Pipe" 1 ~~~ -### Useful Examples +### Advanced Examples #### Stream Selection By default, the `>`, `>>`, and `|` operators all operate on the standard output stream `stdout`, and leave the standard error stream `stderr` untouched. If we want to redirect just the error messages from a program, we can do so by selecting stream 2: @@ -85,3 +85,27 @@ $ git commit 2>&1 1>/dev/null | grep -c "fatal" 1 ~~~ +If we want the output of both streams to be redirected into a single file, we can do that with `&>` + +~~~ bash +$ echo '#include +> int main() { +> std::cout << "This is stdout text\n"; +> std::cerr << "This is stderr text\n"; +> }' > prog.cpp # a simple program that prints to both streams +$ g++ prog.cpp -o prog +$ ./prog 2>/dev/null 1> output.txt +$ cat output.txt +This is stdout text +$ ./prog 1>/dev/null 2> output.txt +$ cat output.txt +This is stderr text +$ ./prog &> output.txt +$ cat output.txt +This is stdout text +This is stderr text +~~~ + +#### Pipe Chaining + + From 0f124d174c32f8f04d0b19f7bbe995f4d5c10d10 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 22:54:33 -0400 Subject: [PATCH 09/14] Added pipe chaining example --- _commands/tutorials/io_redirection.md | 41 +++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/_commands/tutorials/io_redirection.md b/_commands/tutorials/io_redirection.md index bf8700571..39ef3e35e 100644 --- a/_commands/tutorials/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -107,5 +107,46 @@ This is stderr text ~~~ #### Pipe Chaining +The `|` operator allows us to solve complex problems in a simple and expressive manner by piping output between many different programs. To demonstrate this, we'll build up a pipeline to solve the following problem: given the text of a `git log`, how many unique authors have commited to this branch? For the purposes of this exercise, we'll define a unique author to be a unique email address. We'll build our pipeline one command at a time, starting with generating the `git log`: +~~~ bash +$ git log +~~~ + +First, we need to isolate the line of each commit that contains the email address. `grep` is great at this: + +~~~ bash +$ git log | grep "^Author" +~~~ + +Next, we need to remove the portion of the line before each email address. We can separate the line into two pieces using `cut`, specifying the delimeter as the < character and telling `cut` to output the second field: + +~~~ bash +$ git log | grep "^Author" | cut -d "<" -f 2 +~~~ + +Now we nearly have each email address isolated. We just need to remove the trailing > character from each line. `tr` can do this with the -d flag: + +~~~ bash +$ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" +~~~ + +With each email now isolated, we need to group all identical emails together. `sort` does this easily: + +~~~ bash +$ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" | sort +~~~ + +We're almost there. `uniq` will allow us to remove all the duplicate emails: + +~~~ bash +$ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" | sort | uniq +~~~ + +Now we have a list of every unique contributor to the project. All that's left to do is count them up with `wc`: + +~~~ bash +$ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" | sort | uniq | wc -l +~~~ +#### Redirecting To and From Loops From 3446a8b3170cc90f5095a15003220f7a8fc6d430 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 23:12:47 -0400 Subject: [PATCH 10/14] Cleaning up advanced examples --- _commands/tutorials/io_redirection.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/_commands/tutorials/io_redirection.md b/_commands/tutorials/io_redirection.md index 39ef3e35e..22f798bfb 100644 --- a/_commands/tutorials/io_redirection.md +++ b/_commands/tutorials/io_redirection.md @@ -148,5 +148,3 @@ Now we have a list of every unique contributor to the project. All that's left t ~~~ bash $ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" | sort | uniq | wc -l ~~~ - -#### Redirecting To and From Loops From 8382e645ed5335787c2d10a95d5cf390423612f7 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 23:15:31 -0400 Subject: [PATCH 11/14] Move IO redirection into scripting --- _commands/{tutorials => scripting}/io_redirection.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename _commands/{tutorials => scripting}/io_redirection.md (100%) diff --git a/_commands/tutorials/io_redirection.md b/_commands/scripting/io_redirection.md similarity index 100% rename from _commands/tutorials/io_redirection.md rename to _commands/scripting/io_redirection.md From 6e3420af22ae3631f5b502456ab21aa9d6263413 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Mon, 22 Oct 2018 23:20:04 -0400 Subject: [PATCH 12/14] Removed unneccesary test.txt --- test.txt | 1 - 1 file changed, 1 deletion(-) delete mode 100644 test.txt diff --git a/test.txt b/test.txt deleted file mode 100644 index 9daeafb98..000000000 --- a/test.txt +++ /dev/null @@ -1 +0,0 @@ -test From 8535146a45419123ccb1e42a27b8840aaa75fea2 Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Tue, 23 Oct 2018 11:25:51 -0400 Subject: [PATCH 13/14] Improved pipe chaining example with better use of regex --- _commands/scripting/io_redirection.md | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-) diff --git a/_commands/scripting/io_redirection.md b/_commands/scripting/io_redirection.md index 22f798bfb..609c1201d 100644 --- a/_commands/scripting/io_redirection.md +++ b/_commands/scripting/io_redirection.md @@ -113,38 +113,27 @@ The `|` operator allows us to solve complex problems in a simple and expressive $ git log ~~~ -First, we need to isolate the line of each commit that contains the email address. `grep` is great at this: +First, we need to isolate all the emails in this file. `grep` makes this pretty straightword. Every email +contains an @ sign and is bracketed by the < and > characters. This gives us the regular expression `<.*@.*>` (note that we're assuming no one types an email in this format into a commit message, which is true of the repository for this website). Combine this with the `-o` flag to print the matching strings only: ~~~ bash -$ git log | grep "^Author" +$ git log | grep -o "<.*@.*>" ~~~ -Next, we need to remove the portion of the line before each email address. We can separate the line into two pieces using `cut`, specifying the delimeter as the < character and telling `cut` to output the second field: - -~~~ bash -$ git log | grep "^Author" | cut -d "<" -f 2 -~~~ - -Now we nearly have each email address isolated. We just need to remove the trailing > character from each line. `tr` can do this with the -d flag: - -~~~ bash -$ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" -~~~ - With each email now isolated, we need to group all identical emails together. `sort` does this easily: ~~~ bash -$ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" | sort +$ git log | grep -o "<.*@.*>" | sort ~~~ We're almost there. `uniq` will allow us to remove all the duplicate emails: ~~~ bash -$ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" | sort | uniq +$ git log | grep -o "<.*@.*>" | sort | uniq ~~~ -Now we have a list of every unique contributor to the project. All that's left to do is count them up with `wc`: +Now we have a list of every unique contributor to the project. All that's left to do is count them up with `wc` (the `-l` flag counts the number of lines only): ~~~ bash -$ git log | grep "^Author" | cut -d "<" -f 2 | tr -d ">" | sort | uniq | wc -l +$ git log | grep -o "<.*@.*>" | sort | uniq | wc -l ~~~ From 1d8a9ed7bc6b17a3ebbb816162aafec1196843aa Mon Sep 17 00:00:00 2001 From: Noah Tutt Date: Tue, 23 Oct 2018 12:19:39 -0400 Subject: [PATCH 14/14] Added headings to the Advanced Examples section --- _commands/scripting/io_redirection.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/_commands/scripting/io_redirection.md b/_commands/scripting/io_redirection.md index 609c1201d..38a82ec29 100644 --- a/_commands/scripting/io_redirection.md +++ b/_commands/scripting/io_redirection.md @@ -64,7 +64,7 @@ $ echo "Pipes are fun!" | grep -c "Pipe" ### Advanced Examples #### Stream Selection -By default, the `>`, `>>`, and `|` operators all operate on the standard output stream `stdout`, and leave the standard error stream `stderr` untouched. If we want to redirect just the error messages from a program, we can do so by selecting stream 2: +By default, the `>` and `>>` operators all operate on the standard output stream `stdout`, and leave the standard error stream `stderr` untouched. If we want to redirect just the error messages from a program, we can do so by selecting stream 2: ~~~ bash $ git commit @@ -76,7 +76,8 @@ $ cat git_errors.txt fatal: Not a git repository (or any of the parent directories): .git ~~~ -We can't directly change the input source of the `|` operator in this fashion, it always reads from `stdout`. If we want to pipe `stderr` only, we can first point `stderr` at `stdout` and then point `stdout` at nothing: +#### Changing the Input of `|`: Crossing the Streams +We can't directly change the input source of the `|` operator as we did for `>` and `>>`, it always reads from the default source of `stdout`. If we want to redirect `stderr` only, we can first point `stderr` at `stdout` and then point `stdout` at nothing: ~~~ bash $ git commit @@ -85,6 +86,7 @@ $ git commit 2>&1 1>/dev/null | grep -c "fatal" 1 ~~~ +#### Combining the Streams If we want the output of both streams to be redirected into a single file, we can do that with `&>` ~~~ bash