first pass of last section of first tutorial

RobJY · s-sajid-ali · commit 043f5d0cae2a · 2025-05-30T18:49:51.000-04:00
diff --git a/docs/hpc/13_tutorial_intro_shell_hpc/06_scripts_variables_loops.mdx b/docs/hpc/13_tutorial_intro_shell_hpc/06_scripts_variables_loops.mdx
@@ -0,0 +1,393 @@
+import { Smile } from "lucide-react";
+
+# Scripts, variables, and loops
+
+:::note[Overview]
+Questions
+-   How do I turn a set of commands into a program?
+
+Objectives
+-   Write a shell script
+-   Understand and manipulate UNIX permissions
+-   Understand shell variables and how to use them
+-   Write a simple `for` loop.
+:::
+
+We now know a lot of UNIX commands! Wouldn’t it be great if we could save certain commands so that we could run them later or not have to type them out again? As it turns out, this is straightforward to do. A “shell script” is essentially a text file containing a list of UNIX commands to be executed in a sequential manner. These shell scripts can be run whenever we want, and are a great way to automate our work.
+
+## Writing a Script
+So how do we write a shell script, exactly? It turns out we can do this with a text editor. Start editing a file called `demo.sh` (to recap, we can do this with `nano demo.sh`). The `.sh` is the standard file extension for shell scripts that most people use (you may also see `.bash` used).
+
+Our shell script will have two parts:
+
+-   On the very first line, add `#!/bin/bash`. The `#!` (pronounced “hash-bang”) tells our computer what program to run our script with. In this case, we are telling it to run our script with our command-line shell (what we’ve been doing everything in so far). If we wanted our script to be run with something else, like Perl, we could use `#!/usr/bin/perl` instead.
+-   Now, anywhere below the first line, add `echo "Our script worked!"`. When our script runs, `echo` will happily print out "Our script worked!".
+
+Our file should now look like this:
+```bash
+#!/bin/bash
+
+echo "Our script worked!"
+```
+
+Ready to run our program? Let’s try running it:
+```bash
+$ demo.sh 
+bash: demo.sh: command not found...
+```
+
+Strangely enough, Bash can’t find our script. As it turns out, Bash will only look in certain directories for scripts to run. To run anything else, we need to tell Bash exactly where to look. To run a script that we wrote ourselves, we need to specify the full path to the file, followed by the filename. We could do this one of two ways:
+-   with our absolute path /home/yourUserName/demo.sh
+-   with the relative path ./demo.sh
+
+```bash
+$ ./demo.sh
+bash: ./demo.sh: Permission denied
+```
+There’s one last thing we need to do. Before a file can be run, it needs 'permission' to run. We'll get a better understanding of Linux file permissions in the next section that will allow us to finally run our script.
+
+## Permissions
+Let’s look at our file’s permissions with `ls -l`:
+
+```bash
+$ ls -l
+-rw-rw-r-- 1 yourUsername tc001 12534006 Jan 16 18:50 bash-lesson.tar.gz
+-rw-rw-r-- 1 yourUsername tc001       40 Jan 16 19:41 demo.sh
+-rw-rw-r-- 1 yourUsername tc001 77426528 Jan 16 18:50 dmel-all-r6.19.gtf
+-rw-r--r-- 1 yourUsername tc001   721242 Jan 25  2016 dmel_unique_protein_is...
+drwxrwxr-x 2 yourUsername tc001     4096 Jan 16 19:16 fastq
+-rw-r--r-- 1 yourUsername tc001  1830516 Jan 25  2016 gene_association.fb.gz
+-rw-rw-r-- 1 yourUsername tc001       15 Jan 16 19:17 test.txt
+-rw-rw-r-- 1 yourUsername tc001      245 Jan 16 19:24 word_counts.txt
+```
+
+That’s a huge amount of output: a full listing of everything in the directory. Let’s see if we can understand what each field of a given row represents, working from the left to right.
+
+### Column 1: File/Directory Permissions
+This column contains a block of subcolumns that define the permissions for a file or directory given in each row.  The permissions are shown for three user types to perform three actions each.
+
+The user types are:
+-   user (`u`): This refers to your permissions for this file/directory.
+-   group (`g`): This refers to the permissions for people in the same group as this file/directory.  You will see the group in the 4th column.
+-   other (`o`): This refers to the permissions for all other users.
+
+The actions are:
+-   read (`r`): This refers to the permission to read this file.
+-   write (`w`): This refers to the permission to write to this file.
+-   execute (`x`): This refers to the permission to execute this file.
+
+The following table show what each of the subcolumns refer to and their possible values:
+
+| directory  | user read  | user write | user execute | group read | group write | group execute | other read | other write | other execute |
+| ---------- | ---------- | ---------- | ------------ | ---------- | ----------- | ------------- | ---------- | ----------- | ------------- |
+| `d` or `-` | `r` or `-` | `w` or `-` | `x` or `-`   | `r` or `-` | `w` or `-`  | `x` or `-`    | `r` or `-` | `w` or `-`  | `x` or `-`    |
+
+If there is a `-` in the directory column, the row refers to a file.  If it contains a `d`, the row refers to a directory.  The following columns behave in a similar manner. If they contain a `-`, the associated action is not allowed for the associated user type.
+
+### Column 2: References
+This counts the number of references (hard links) to the item (file, folder, symbolic link or “shortcut”).
+
+### Column 3: Owner
+This is the username of the user who owns the file.
+:::note
+Their permissions are indicated in the first permissions block of three after the directory column
+:::
+
+### Column 4: Group
+Each user has a primary group and is optionally a member of other groups. When a user creates a file, it is normally associated with the user’s primary group. At NYU HPC, all users are in a group named ‘users’, so group permission has little meaning.
+:::note
+Other members of this group have the permissions in the second block of three after the directory column
+:::
+
+### Column 5: Size of item
+This is the number of bytes in a file, or the number of filesystem blocks occupied by the contents of a folder.
+:::note
+We can use the `-h` option here to get a human-readable file size in megabytes, gigabytes, etc.
+:::
+
+### Column 6: Time last modified
+This is the last time the file was modified.
+
+### Column 7: Filename
+This is the name of the file/directory.
+
+## Changing Permissions
+As previously mentioned, in Unix a file has three basic permissions, each of which can be set for three types of user. Those three permission also have a numeric value:
+
+-   Read permission (“r”) - numeric value 4.
+-   Write permission (“w”) - numeric value 2.
+-   Execute permission (“x”) - numeric value 1.
+:::note
+When applied to a directory, execute permission refers to whether the directory can be entered with `cd`.
+:::
+
+You'll need to use the `chmod` command to modify permissions.  You grant permissions with `chmod who+what file` and revoke them with `chmod who-what file`. (Notice that the first has `+` and the second `-`). Here, “who” is some combination of “u”, “g”, and “o”, and “what” is some combination of “r”, “w”, and “x”.  Leaving out the `who` part of the command applies it to all user types.
+
+So, to set execute permission we use:
+```bash
+$ chmod +x demo.sh
+$ ls -l
+-rw-rw-r-- 1 yourUsername tc001 12534006 Jan 16 18:50 bash-lesson.tar.gz
+-rwxrwxr-x 1 yourUsername tc001       40 Jan 16 19:41 demo.sh
+-rw-rw-r-- 1 yourUsername tc001 77426528 Jan 16 18:50 dmel-all-r6.19.gtf
+-rw-r--r-- 1 yourUsername tc001   721242 Jan 25  2016 dmel_unique_protein_is...
+drwxrwxr-x 2 yourUsername tc001     4096 Jan 16 19:16 fastq
+-rw-r--r-- 1 yourUsername tc001  1830516 Jan 25  2016 gene_association.fb.gz
+-rw-rw-r-- 1 yourUsername tc001       15 Jan 16 19:17 test.txt
+-rw-rw-r-- 1 yourUsername tc001      245 Jan 16 19:24 word_counts.txt
+```
+
+## Executing Script
+Now that we have executable permissions for that file, we can run it.
+```bash
+$ ./demo.sh
+```
+
+Our script worked!  Fantastic, we’ve written our first program!
+
+## Comments
+Before we go any further, let’s learn how to take notes inside our program using comments. A comment is indicated by the `#` character, followed by whatever we want. Comments do not get run. Let’s try out some comments in the console, then add one to our script!
+
+```bash
+# This won't show anything.
+```
+
+Now let's try adding this to our script with nano. Edit your script to look something like this:
+```bash
+#!/bin/bash
+
+# This is a comment... they are nice for making notes!
+echo "Our script worked!"
+```
+
+When we run our script, the output should be unchanged from before!
+
+## Shell variables
+One important concept that we’ll need to cover are shell variables. Variables are a great way of saving information under a name you can access later. In programming languages like Python and R, variables can store pretty much anything you can think of. In the shell, they usually just store text. The best way to understand how they work is to see them in action.
+
+To set a variable, simply type in a name containing only letters, numbers, and underscores, followed by an `=` and whatever you want to put in the variable. Shell variable names are often uppercase by convention (but do not have to be).
+```bash
+$ VAR="This is our variable"
+```
+To use a variable, prefix its name with a `$` sign. Note that if we want to simply check what a variable is, we should use `echo` (or else the shell will try to run the contents of a variable).
+```bash
+$ echo $VAR
+This is our variable
+```
+Let’s try setting a variable in our script and then recalling its value as part of a command. We’re going to make it so our script runs `wc -l` on whichever file we specify with `FILE`.
+
+Our script:
+```bash
+#!/bin/bash
+
+# set our variable to the name of our GTF file
+FILE=dmel-all-r6.19.gtf
+
+# call wc -l on our file
+wc -l $FILE
+```
+```bash
+$ ./demo.sh
+542048 dmel-all-r6.19.gtf
+```
+
+What if we wanted to do our little `wc -l` script on other files without having to change `$FILE` every time we want to use it? There is actually a special shell variable we can use in scripts that allows us to use arguments in our scripts (arguments are extra information that we can pass to our script, like the `-l` in `wc -l`).
+
+To use the first argument to a script, use `$1` (the second argument is `$2`, and so on). Let’s change our script to run `wc -l` on `$1` instead of `$FILE`. Note that we can also pass all of the arguments using `$@` (not going to use it in this lesson, but it’s something to be aware of).
+
+Our script:
+```bash
+#!/bin/bash
+
+# call wc -l on our first argument
+wc -l $1
+```
+```bash
+$ ./demo.sh dmel_unique_protein_isoforms_fb_2016_01.tsv
+22129 dmel_unique_protein_isoforms_fb_2016_01.tsv
+```
+
+Nice! One thing to be aware of when using variables: they are all treated as pure text. How do we save the output of an actual command like `ls -l`?
+
+First, a demonstration of what doesn’t work:
+```bash
+$ TEST=ls -l
+-bash: -l: command not found
+```
+
+What does work? We need to surround any command with `$(command)`:
+```bash
+$ TEST=$(ls -l)
+$ echo $TEST
+total 90372 -rw-rw-r-- 1 jeff jeff 12534006 Jan 16 18:50 bash-lesson.tar.gz -rwxrwxr-x. 1 jeff jeff 40 Jan 1619:41 demo.sh -rw-rw-r-- 1 jeff jeff 77426528 Jan 16 18:50 dmel-all-r6.19.gtf -rw-r--r-- 1 jeff jeff 721242 Jan 25 2016 dmel_unique_protein_isoforms_fb_2016_01.tsv drwxrwxr-x. 2 jeff jeff 4096 Jan 16 19:16 fastq -rw-r--r-- 1 jeff jeff 1830516 Jan 25 2016 gene_association.fb.gz -rw-rw-r-- 1 jeff jeff 15 Jan 16 19:17 test.txt -rw-rw-r-- 1 jeff jeff 245 Jan 16 19:24 word_counts.txt
+```
+:::note
+Everything got printed on the same line. This is a feature, not a bug, as it allows us to use $(commands) inside lines of script without triggering line breaks (which would end our line of code and execute it prematurely).
+:::
+
+## Loops
+To end our lesson on scripts, we are going to learn how to write a for-loop to execute a lot of commands at once. This will let us do the same string of commands on every file in a directory (or other stuff of that nature).
+
+for-loops generally have the following syntax:
+```bash
+#!/bin/bash
+
+for VAR in first second third
+do
+    echo $VAR
+done
+```
+
+When a for-loop gets run, the loop will run once for everything following the word `in`. In each iteration, the variable `$VAR` is set to a particular value for that iteration. In this case it will be set to `first` during the first iteration, `second` on the second, and so on. During each iteration, the code between `do` and `done` is performed.
+
+Let’s run the script we just wrote (I saved mine as `loop.sh`).
+```bash
+$ chmod +x loop.sh
+$ ./loop.sh
+first
+second
+third
+```
+
+What if we wanted to loop over a shell variable, such as every file in the current directory? Shell variables work perfectly in for-loops. In this example, we’ll save the result of `ls` and loop over each file:
+```bash
+#!/bin/bash
+
+FILES=$(ls)
+for VAR in $FILES
+do
+        echo $VAR
+done
+```
+```bash
+$ ./loop.sh
+bash-lesson.tar.gz
+demo.sh
+dmel_unique_protein_isoforms_fb_2016_01.tsv
+dmel-all-r6.19.gtf
+fastq
+gene_association.fb.gz
+loop.sh
+test.txt
+word_counts.txt
+```
+
+There’s a shortcut to run on all files of a particular type, say all `.gz` files:
+```bash
+#!/bin/bash
+
+for VAR in *.gz
+do
+    echo $VAR
+done
+```
+```bash
+bash-lesson.tar.gz
+gene_association.fb.gz
+```
+<details>
+<summary>
+:::info[Writing our own scripts and loops]
+`cd` to our `fastq` directory from earlier and write a loop to print off the name and top 4 lines of every fastq file in that directory.
+
+Is there a way to only run the loop on fastq files ending in _1.fastq? <br />
+**[Click for Solution]**
+:::
+</summary>
+:::tip[Solution]
+Create the following script in a file called `head_all.sh`
+```bash
+#!/bin/bash
+
+for FILE in *.fasatq
+do
+    echo $FILE
+    head -n 4 $FILE
+done
+```
+The `for` line could be modified to be `for FILE in *_1.fastq` to achieve the second aim.
+:::
+</details>
+
+<details>
+<summary>
+:::info[Concatenating variables]
+Concatenating (i.e. mashing together) variables is quite easy to do. Add whatever you want to concatenate to the beginning or end of the shell variable after enclosing it in `{}` characters.
+```bash
+$ FILE=stuff.txt
+$ echo ${FILE}.example
+stuff.txt.example
+```
+Can you write a script that prints off the name of every file in a directory with `.processed` added to it? <br />
+**[Click for Solution]**
+:::
+</summary>
+::::tip[Solution]
+Create the following script in a file called `process.sh`:
+```bash
+#!/bin/bash
+
+for FILE in *
+do
+    echo ${FILE}.processed
+done
+```
+:::note
+This will also print directories appended with `.processed`. 
+:::
+To truly only get files and not directories, we need to modify this to use the `find` command to give us only files in the current directory:
+```bash
+#!/bin/bash
+
+for FILE in $(find . -maxdepth 1 -type f)
+do
+    echo ${FILE}.processed
+done
+```
+but this will have the side effect of listing hidden files too.  We can fix this by making a small change to the find command:
+```bash
+#!/bin/bash
+
+for FILE in $(find . -maxdepth 1 -type f ! -name ".*")
+do
+    echo ${FILE}.processed
+done
+```
+We've added `! -name ".*"` to the `find` command.  It means `not` (`!`) a name that starts with `.`.<br />
+As you can see, programming is often iterative in more ways than one. <Smile />
+::::
+</details>
+
+<details>
+<summary>
+:::info[Special permissions]
+What if we want to give different sets of users different permissions. `chmod` actually accepts special numeric codes instead of stuff like `chmod +x`, as we mentioned above. Again, the numeric codes are as follows: read = 4, write = 2, execute = 1. For each user we will assign permissions based on the sum of these permissions (must be between 7 and 0).
+
+Let’s make an example file and give everyone permission to do everything with it.
+```bash
+$ touch example
+$ ls -l example
+-rw-r--r--  1 yourUsername  users  0 May 30 14:50 example
+$ chmod 777 example
+$ ls -l example
+-rwxrwxrwx  1 yourUsername  users  0 May 30 14:50 example
+```
+
+How might we give ourselves permission to do everything with a file, but allow no one else to do anything with it.
+**[Click for Solution]**
+:::
+</summary>
+:::tip[Solution]
+```bash
+$ chmod 700 example
+$ ls -l example
+-rwx------  1 yourUsername  users  0 May 30 14:50 example
+```
+We want all permissions, so: 4 (read) + 2 (write) + 1 (execute) = 7 for user (first position), no permissions, i.e. 0, for group (second position) and other (third position).
+:::
+</details>
+
+:::tip[Key Points]
+-   A shell script is just a list of bash commands in a text file.
+-   To make a shell script file executable, run `chmod +x script.sh`.
+:::
