Navigating the Caviness Filesystem
Overview
Teaching: 30 min
Exercises: 5 minQuestions
How do I navigate and start interacting with files and directories on Caviness?
How do I keep track of where I am in the filesystem?
Objectives
Learn how to navigate around directories and look at their contents.
Explain the difference between a file and a directory.
Translate an absolute path into a relative path and vice versa.
Identify the actual command, flags, and filenames in a command-line call.
Demonstrate the use of tab completion, and explain its advantages.
At this point in the lesson, we’ve just logged into the system. Nothing has happened yet, and we’re
not going to be able to do anything until we learn a few basic commands. In this lesson we will talk
about ls
, cd
, and a few other commands. These commands will help you navigate around not only the
Caviness filesystem structure but nearly any Linux/Unix filesystem using the CLI.
Right now, all we see is something that looks like this:
[traine@login00 ~]$
The dollar sign is a prompt, which shows us that the shell is waiting for input. If you do not
see the dollar sign, then the system is not ready for your next input. This applies for the duration
of your session. Later, we will talk about what happens if a command is entered and a $
prompt does
not return to your screen. When typing commands, either from these lessons or from other sources,
do not type the prompt $
, only the commands that follow it.
Type the command whoami
, then press the enter key (sometimes marked Return) to send the
command to the shell. The command’s output is the ID of the current user, i.e., it shows us who the
shell
thinks we are:
$ whoami
traine
More specifically, when we type whoami
the shell:
- finds a program called
whoami
, - runs that program,
- displays that program’s output, then
- displays a new prompt to tell us that it’s ready for more commands.
Next, let’s find out where we are by running a command called pwd
(which stands for “print working
directory”). At any moment, our current working directory (where we are) is the directory that
the computer assumes we want to run commands in unless we explicitly specify something else. Here,
the computer’s response is /home/1201
, which is user
traine
’s home directory.
Note that the location of your home directory may differ from system to system.
$ pwd
/home/1201
So, we know where we are. How do we look and see what’s in our current directory?
$ ls
ls
prints the names of the files and directories in the current directory in alphabetical order,
arranged neatly into columns.
Differences between remote and local system
Open a second terminal window on your local computer and run the
ls
command without logging in remotely. What differences do you see? Note: Window users will not be able to do this with PuTTY, since PuTTY only opens remote connections.Solution
You would likely see something more like this:
Applications Documents Library Music Public Desktop Downloads Movies Pictures
In addition, you should also note that the text before the prompt (
$
) is different. This is very important for making sure you know what system you are issuing commands on when in the shell.
If nothing shows up when you run ls
, it means that nothing’s there. Let’s make a directory for us
to play with.
mkdir <new directory name>
makes a new directory with that name in your current location. Notice
that this command required two pieces of input: the actual name of the command (mkdir
) and an
argument that specifies the name of the directory you wish to create.
$ mkdir training
Let’s type ls
again. What do we see?
training
Our folder (directory) is there, awesome. What if we wanted to go inside it and do stuff there? We will use the
cd
(change directory) command to move around. Let’s cd
into our new training folder.
$ cd training
$ pwd
/home/1201/training
Now that we know how to use cd
, we can go anywhere. That’s a lot of responsibility. What happens
if we get “lost” and want to get back to where we started?
To go back to your home directory, the following two commands will work:
$ cd /home/1201
$ cd ~
What is the ~
character? When using the shell, ~
is a shortcut that represents
/home/1201
.
A quick note on the structure of a UNIX (Linux/Mac/Android/Solaris/etc) filesystem. Directories and
absolute paths (i.e. exact position in the system) are always prefixed with a /
. /
is the “root”
or base directory.
Let’s go there now, look around, and then return to our home directory.
$ cd /
$ ls
bin dev home lib64 media opt root sbin sys ttt var
boot etc lib lustre mnt proc run srv tmp usr work
The “home” directory is the one where we generally want to keep all of our files. Other folders on a UNIX OS contain system files, and get modified and changed as you install new software or upgrade your OS.
Returning to your home directory
From the root directory please provide three different ways of getting back to your home directory.
Solution
cd
- Shortcut
cd ~
- Shorthand
cd /home/1201
- This is the absolute path
Using HPC filesystems
On HPC systems, you have a number of places where you can store your files. These differ in both the amount of space allocated and whether they are backed up or not. Caviness has three different types of storage where you can store files that serve different purposes. The amount of storage available and how they are backed up differs as well.
File storage locations:
Home Directory Storage - Each user has 20 GB of disk storage reserved for personal use on the home filesystem. Users’ home directories are in /home (e.g., /home/1201), and the directory name is put in the environment variable
$HOME
at login. The permanent filesystem is configured to allow nearly instantaneous, consistent snapshots. The snapshot contains the original version of the filesystem, and the live filesystem contains any changes made since the snapshot was taken. In addition, all your files are regularly replicated at UD’s off-campus disaster recovery site. You can use read-only snapshots to revert a previous version, or request to have your files restored from the disaster recovery site.You can check to see the size and usage of your home directory with the command
df -h $HOME
Work Group Storage - Each research group has at least 1000 GB of shared group (workgroup) storage in the
/work
directory identified by the«investing_entity»
(e.g.,/work/it_css
) and is referred to as your workgroup directory. This is used for input files, supporting data files, work files, and output files, source code and executables that need to be shared with your research group. Just as your home directory, read-only snapshots of workgroup’s files are made several times for the passed week. In addition, the filesystem is replicated on UD’s off-campus disaster recovery site. Snapshots are user-accessible, and older files may be retrieved by special request.You can check the size and usage of your workgroup directory by using the workgroup command to spawn a new workgroup shell, which sets the environment variable
$WORKDIR
df -h $WORKDIR
High-performance filesystem:
Lustre Storage - User storage is available on a high-performance Lustre-based filesystem having 210 TB of usable space. This is used for temporary input files, supporting data files, work files, and output files associated with computational tasks ran on the cluster. The filesystem is accessible to all of the processor cores via Omni-path Infiniband.
The Lustre filesystem is not backed up nor are there snapshots to recover deleted files. However, it is a robust RAID-6 system. Thus, the filesystem can survive a concurrent disk failure of two independent hard drives and still rebuild its contents automatically.
All users have access the public scratch directory (
/lustre/scratch
).A full system inhibits use for everyone potentially preventing jobs from running. Therefore IT staff may run cleanup procedures as needed to purge aged files or directories in Lustre if old files are degrading system performance.
There are several other useful shortcuts you should be aware of.
.
represents your current directory..
represents the “parent” directory of your current location- While typing nearly anything, you can have bash try to autocomplete what you are typing by
pressing the
tab
key.
Let’s try these out now:
$ cd ./training
$ pwd
$ cd ..
$ pwd
/home/1201/training
/home/1201
Many commands also have multiple behaviours that you can invoke with command line ‘flags.’ What is a
flag? It’s generally just your command followed by a -
or --
and the name of the flag.
You follow the flag(s) with any additional arguments you
might need.
We’re going to demonstrate a couple of these “flags” using ls
.
Show hidden files with -a
. Hidden files are files that begin with .
, these files will not appear
otherwise, but that doesn’t mean they aren’t there! “Hidden” files are not hidden for security
purposes, they are usually just config files and other tempfiles that the user doesn’t necessarily
need to see all the time.
$ ls -a
. .. .bash_history .bash_logout .bash_profile .bashrc .bash_udit training .ssh
Notice how both .
and ..
are visible as hidden files.
To show files, their size in bytes, date last modified, permissions, and other
things with -l
.
$ ls -l
drwxr-xr-x 2 traine everyone 2 Jul 13 15:48 training
This is a lot of information to take in at once, but we will explain this later! ls -l
is
extremely useful, and tells you almost everything you need to know about your files without
actually looking at them.
We can also use multiple flags at the same time!
$ ls -l -a
[traine@login00 ~]$ ls -la
total 36
drwx--x--x 17 traine everyone 29 Jul 21 14:22 .
drwxr-xr-x 79 root root 0 Jul 21 14:23 ..
-rw------- 1 traine it_css 20442 Jul 21 15:24 .bash_history
-rw-r--r-- 1 traine everyone 17 Jul 25 2018 .bash_logout
-rw-r--r-- 1 traine everyone 200 Jul 25 2018 .bash_profile
-rw-r--r-- 1 traine everyone 384 Mar 19 15:21 .bashrc
-rw-r--r-- 1 traine everyone 1154 Mar 24 10:16 .bash_udit
drwxr-xr-x 2 traine everyone 2 Jul 13 15:48 training
drwx------ 2 traine everyone 6 Mar 18 2020 .ssh
-rw------- 1 traine everyone 1007 Oct 15 11:29 .Xauthority
Flags generally precede any arguments passed to a UNIX command. ls
actually takes an extra
argument that specifies a directory to look into. When you use flags and arguments together, they
syntax (how it’s supposed to be typed) generally looks something like this:
$ command <flags/options> <arguments>
So using ls -l -a
on a different directory than the one we’re in would look something like:
$ ls -l -a ~/training
drwxr-sr-x 2 yourUsername tc001 4096 Nov 28 09:58 .
drwx--S--- 5 yourUsername tc001 4096 Nov 28 09:58 ..
Another useful flag is the -F
flag. The -F
appends indicators to entries which helps you
identify files verus directories.
ls -F -la
drwx--x--x 17 traine everyone 29 Jul 21 14:22 ./
drwxr-xr-x 79 root root 0 Jul 21 14:23 ../
-rw------- 1 traine it_css 20442 Jul 21 15:24 .bash_history
-rw-r--r-- 1 traine everyone 17 Jul 25 2018 .bash_logout
-rw-r--r-- 1 traine everyone 200 Jul 25 2018 .bash_profile
-rw-r--r-- 1 traine everyone 384 Mar 19 15:21 .bashrc
-rw-r--r-- 1 traine everyone 1154 Mar 24 10:16 .bash_udit
drwxr-xr-x 2 traine everyone 2 Jul 13 15:48 training/
drwx------ 2 traine everyone 6 Mar 18 2020 .ssh/
-rw------- 1 traine everyone 1007 Oct 15 11:29 .Xauthority
As you can see in the above output rows that end with a /
are directories. The rows that don’t
end with a /
are files.
Where to go for help?
How did I know about the -l
and -a
options? Is there a manual we can look at for help when we
need help? There is a very helpful manual for most UNIX commands: man
(if you’ve ever heard of a
“man page” for something, this is what it is).
$ man ls
LS(1) User Commands LS(1)
NAME
ls - list directory contents
SYNOPSIS
ls [OPTION]... [FILE]...
DESCRIPTION
List information about the FILEs (the current directory by default). Sort entries alphabetically if none of -cftu‐
vSUX nor --sort is specified.
Mandatory arguments to long options are mandatory for short options too.
Manual page ls(1) line 1 (press h for help or q to quit)
To navigate through the man
pages, you may use the up and down arrow keys to move line-by-line, or
try the spacebar and “b” keys to skip up and down by full page. Quit the man
pages by typing “q”.
Alternatively, most commands you run will have a --help
option that displays addition information
For instance, with ls
:
$ ls --help
Usage: ls [OPTION]... [FILE]...
List information about the FILEs (the current directory by default).
Sort entries alphabetically if none of -cftuvSUX nor --sort is specified.
Mandatory arguments to long options are mandatory for short options too.
-a, --all do not ignore entries starting with .
-A, --almost-all do not list implied . and ..
--author with -l, print the author of each file
-b, --escape print C-style escapes for nongraphic characters
--block-size=SIZE scale sizes by SIZE before printing them; e.g.,
'--block-size=M' prints sizes in units of
1,048,576 bytes; see SIZE format below
-B, --ignore-backups do not list implied entries ending with ~
# further output omitted for clarity
Unsupported command-line options
If you try to use an option that is not supported,
ls
and other programs will print an error message similar to this:[traine@login00 ~]$ $ ls -j
ls: invalid option -- 'j' Try 'ls --help' for more information.
Looking at documentation
Looking at the man page for
ls
or usingls --help
, what does the-h
(--human-readable
) option do?Solution
-h, --human-readable: with -l, print sizes in human readable format (e.g., 1K 234M 2G)
Absolute vs Relative Paths
Starting from
/Users/amanda/data/
, which of the following commands could Amanda use to navigate to her home directory, which is/Users/amanda
?
cd .
cd /
cd /home/amanda
cd ../..
cd ~
cd home
cd ~/data/..
cd
cd ..
Solution
- No:
.
stands for the current directory.- No:
/
stands for the root directory.- No: Amanda’s home directory is
/Users/amanda
.- No: this goes up two levels, i.e. ends in
/Users
.- Yes:
~
stands for the user’s home directory, in this case/Users/amanda
.- No: this would navigate into a directory
home
in the current directory if it exists.- Yes: unnecessarily complicated, but correct.
- Yes: shortcut to go back to the user’s home directory.
- Yes: goes up one level.
Relative Path Resolution
Using the filesystem diagram below, if
pwd
displays/home/1201/development
, what willls -F ../back_up
display?
../back_up: No such file or directory
beta_v4/
2019-10-28/ 2020-01-25/ 2020-07-05/
beta_v5/ back_up/
Solution
- No: there is a directory
back_up
in/home/1201
.- No: this is the content of
/home/1201/development/back_up
.- Yes: this is the contents of
/home/1201/back_up
.- No: this would be the contents of
/home/1201/development
or./
.
ls
Reading ComprehensionAssuming a directory structure as in the above diagram, if
pwd
displays/home/1201/development
, and-r
tellsls
to display things in reverse order, what command will display:beta_v5/ back_up/
ls pwd
ls -r -F
ls -r -F /home/1201/development
- Either #2 or #3 above, but not #1.
Solution
- No:
pwd
is not the name of a directory.- Yes:
ls
without directory argument lists files and directories in the current directory.- Yes: uses the absolute path explicitly.
- Correct: see explanations above.
Exploring More
ls
ArgumentsWhat does the command
ls
do when used with the-l
and-h
arguments?Some of its output is about properties that we do not cover in this lesson (such as file permissions and ownership), but the rest should be useful nevertheless.
Solution
The
-l
arguments makesls
use a long listing format, showing not only the file/directory names but also additional information such as the file size and the time of its last modification. The-h
argument makes the file size “human readable”, i.e. display something like5.3K
instead of5369
.
Listing Recursively and By Time
The command
ls -R
lists the contents of directories recursively, i.e., lists their sub-directories, sub-sub-directories, and so on in alphabetical order at each level. The commandls -t
lists things by time of last change, with most recently changed files or directories first. In what order doesls -R -t
display things? Hint:ls -l
uses a long listing format to view timestamps.Solution
The directories are listed alphabetical at each level, the files/directories in each directory are sorted by time of last change.
Key Points
Your current directory is referred to as the working directory.
To change directories, use
cd
.To view files, use
ls
.You can view help for a command with
man command
orcommand --help
.Hit the TAB key to autocomplete whatever you’re currently typing.