As silly as it sounds, when I was a new Unix SysAdmin, I read the entirety of "man 1 bash", which includes all bash builtins. I found that it improved by bash-foo 100x simply by knowing about so many of the utilities. I also took some cliff notes for things that seemed generally useful.
I did it for an hour or so a night for a week or so.
That being said, a few of my personal favorites to memorize:
* Keyboard shortcuts and how they are useful. A few example: CTRL-l (no need to ever use /usr/bin/clear), CTRL-k, CTRL-u, CTRL-e, CTRL-a, CTRL-w, CTRL-arrow left, CTRL-arrow right, CTRL-r (history reverse search with find as you type autocomplete)
The best way you can learn the shell is by using Linux as your primary desktop for at least a few months. You'll get very proficient very quickly by doing that.
One thing I have found that less people seem to know, is that the Unix metacharacters are expanded by the shell (bash etc.) not by individual commands. What this implies is that any command, whether built-in or written by you (in C, bash, Python or any other language), has metacharacter support automatically. That is, things like the file globbing/wildcard characters like *, ?, and [ ranges ].
This was not originally true on DOS (as a counterexample) and not sure whether it is true on Windows today (haven't checked), though I did notice that more commands seem to support wildcards in Windows nowadays.
Also, for some years now, Windows too has had redirections like:
command >file.txt 2>&1
(redirect stderr (2) to the same destination that stdout (1) is pointing to, i.e. file.txt), which Unix had from the start.
Yes, the man page is very good. The online gnu documentation is good, too. Some of the information will not be categorized as bash.[1] Also check out sed and awk.
Most of the responses here so far that do not include some sort of a guide are not the responses you're looking for (imho).
Mind your pipes and quotes. Guard your variables with braces. Do not export everything, test for and (try to) handle return codes and conditions and keep it simple (emphasis simple) but most of all just write it.
BASH (or Bourne) is ubiquitous when dealing with systems (vs programs). You don't need to be on the fashionable lang of the day by any measure. BASH, for most cases, will always be there, always ready and, in most cases, is the default human interface for deployed systems. As scripting languages go you don't need "better", you need dependability, zero dependencies with no requirement for modules or any other whizbangwoohoo plug-in. Language Fashionistas and personal preferences aside at least some level of fluency with BASH should be mandatory for anyone interfacing with a system.
You're going to get a lot of snark from people saying things like "don't", or "learn python instead".
This epitomizes "a little knowledge is a dangerous thing".
Bash has many cringeworthy aspects, but we have to deal with the world as it is, not the world as we would like it to be, and the reality is that bash is the default shell on 99.9% of unix boxes you encounter — even if you use an alt shell on your machine.
Coworkers machine? Bash. Default AWS AMI? Bash. init script to bootstrap $DAEMON? Bash. ssh to a server at your workplace? Bash. Random O'Reilly Linux tutorial? Assumes bash.
My advice?
Take some time.
Sit down.
and read "man bash"
cover-to-cover.
at least once.
A lot of the illogical things in bash make a lot more sense once you understand its parsing/expansion rules. And once you understand what is in the language vs an external terminal program in your PATH.
Since that sounds unappealing (and I scoffed at that very advice for many years), I've also found the wooledge bash guide to be very helpful.
I've been using linux since 1992 and if there's one thing I can't stress enough is to use full directories and not anything abbreviated. After 25 years, I still find myself slipping up, overlooking some minute detail, causing data loss.
I have written a simple tool called mann (https://github.com/soheilpro/mann) to help me remember little things that I learn when working in Bash/Zsh.
Basically, every time I learn something useful about a command, I add it to its mann page and then whenever I need it in the future, I simply run 'mann <command>' to find it.
Here's the current output of my 'mann sed', for example:
# Add char to beginning of each line
sed 's/^/#/'
# Replace with newline
sed 's/<oldvalue>/\'$'\n''/g'
# Replace newline
sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/<newvalue>/g'
# Plus sign
sed -E 's/foo+/bar'
# Digit
sed -E 's/[[:digit:]]/bar'
# Inplace
sed -i'.bak' -e <pattern> <file>
This looks great. I've always kept those types of little snippets in the README of my dotfiles repo - and always keep a printed out copy on my desk. But this seems way more practical.
If you're not already familiar with it, I would suggest learning about the basic Unix process model -- fork, execve, wait, open, pipe, dup2 and friends.
Bash is essentially a DSL for these. A lot of the weirdness you see in the language is due to these abstractions leaking through. For example:
* Quoting is building execve's argv parameter. It's hard to quote correctly if you don't know what exactly you're working towards.
* Redirections are opening and copying file descriptors. It explains their scope, order and nesting behavior.
* Variables are modifying and passing the environment, and their weird scope is due to forks imposed by the process model.
Once you know how you can do whatever you want in C through the basic syscalls, Bash is an extremely efficient and far less surprising shortcut to do it.
can you expand on this? does this elucidate, e.g., variable substitution and the difference between single- and double-quotes? or does it just help demonstrate when you need quotes for an argument that may contain whitespace?
Besides the obvious answers of just reading the manual, looking up howtos, and stackoverflow I can recommend some habits that might increase your uptake of bash.
1. If you are not running a unix as your default OS switch to one (ie Linux or Mac).
2. Create a bin (~/bin) directory in your home directory of all your shells scripts and source control it. Any script you ever write put in that directory. Even if its not bash (ie python, perl). I find that it is critical to look at how you did things previously to help you learn as well as it saves time.
3. Any command that is complicated one liner that you create or see on the internet... create script and put in the bin directory mentioned above.
4. Optimize your personal bin directory and review frequently.
5. If you run Linux build your system from scratch (ie read Linux from scratch).
6. Bonus to the above: Automate the creation of your personal system through Packer and Bash!
7. Find where things are not automated.
8. Bash is more than just the shell. A knowledge of GNU coreutils as well as tmux/screen is worthwhile and highly recommended.
9. Learn the readline shortcuts. Particularly "ctrl-r".
Train yourself to take the 20 minutes required to learn to "do it the right way" every time you need to. Its so easy not to bother because you are busy but in the long run you will save time.
I think this is the right answer, assuming OP actually wants to get better at bash and not route around his original question by learning stuff that isn't bash.
I use bash/shell scripts frequently and have many running 'in production' as cron jobs th at run various jobs or manipulate data for other jobs to run on.
One thing I really like about pure shell is that it's extremely portable and transparent about what it's doing.
I still have to re-learn control structures almost everytime I write a new script, I don't try and memorize [[]] vs [] and all the weird ways equality can work, I just google each time and answers are always on top (once you know what you're looking for).
and keep a set of text files for yourself where after learning to do it the right way you write it down. I find that especially with bash/command line things I don't have to solve the same problem often (which would help with remembering) but often enough to be annoying having to re-learn it all the time, so it really pays off to have some documentation to refer to. org-mode and vimwiki are quite good for that.
Read the manual front to back and install shellcheck. Doing both things has paid off for me a thousand times over. The rest is practice. Complete the bash exercises on Hackerrank. Bash is fantastic in it's domain but it does require serious study in my experience
I can second the shellcheck recommendation. Shellcheck makes great recommendations, and every suggestion has a corresponding code you can google to get to a detailed explanation of why that is considered a warning or error. Hell, even if I just considered the times I forgot to put quotes around a variable and got warned by shellcheck I would be happy that I use it.
The Tcl programming language is what shell scripting should be, basically. It is not just a language with all the features you need, it has explicit Unix shell alike scripting capabilities and strong DSL abilities. An example two-liner:
set files [glob /etc/*.conf]
foreach f $files {file lstat $f file_info; puts "$f: $file_info(size)"}
/etc/asl.conf: 1051
/etc/autofs.conf: 1935
/etc/dnsextd.conf: 2378
... and so forth ...
Also there is an `exec` command that supports pipes, redirections, and so forth:
set result [exec cat /etc/passwd | grep Directory]
The pipe has no special meaning in Tcl, but because of its DSL capabilities you can do things like that. Exec is a DSL basically.
For scripting, I recommend the rc shell from plan9, which is the one I use for my shell scripts. It is only when I want to share a script with other people that I consider using /bin/sh, and even then more often than not I've gone for rc.
From there, move on to using the shell as your IDE. How? First, understand the Unix philosophy. I think Ted Dzubia describes this pretty well in his Taco Bell Programming blog posting:
Great, so now you understand that there are a bunch of useful tools out there and you can string them together to do great things. Now you need to discover the tools themselves.
If you're a "read the dictionary" kind of person, go ahead and start off w/ the Gnu Coreutils documentation. https://www.gnu.org/doc/doc.html
However, if you're like me you'll learn fastest by watching other people work. In this case, I have to point back to Gary Bernhardt again. Specifically, his "Composing a Unix Command Line" screencast will open your eyes wide and very quickly introduce you to a range of incredibly useful coreutils programs in the context of solving a very specific problem. This content is $29/mo, but I'd argue it's money well spent. https://www.destroyallsoftware.com/screencasts/catalog/compo...
Flame war between bash/fish/zsh/powershell is almost meaningless to beginners, because the basic skills are common to all shells. (That said, you will love zsh once you use it)
I learned to use shell, about 7 years ago, by reading O'Reilly "Classic Shell Scripting". It is well written, and teach you something that you can hardly learn from google. But don't try to remember everything, especially those advanced string manipulation syntax, because one would usually use a scripting language such as ruby for advanced job.
Helping you how? If you actually have a problem you are trying to solve, then do just that. My experience with command line came from solving problems I had. Today I do a lot in the command line and I am learning new things all the time. However if I just wanted to get "better" at it, then I don't even know where to start because there is no clear goal.
Well said, it's an ongoing process and there is always something new to learn - some use case which we may not have come across. So, it's an ongoing process.
I wanted to get better at bash too, but instead I ended up getting everything[1] done at fish, which is cool, much better as a language, but no environment has fish pre-installed nowadays.
At the risk of being pedantic, grep sed awk and find are not bash, they are their own programs. You can run find from any shell, or without a shell at all.
I used to lean on python for much of my scripting needs, mainly because the more advanced bash syntax was pretty daunting. Getting better at bash has a trickle-down effect, especially in this container age. ENV var scoping + loops and various var expansion methods really made it click for me. Shelling out to various tasks (and grabbing the results) is effortless via bash scripts. With bash on windows now it's pretty much ubiquitous. My advice is to consider why the task at hand can't be done in bash, because often times it can, with much more portability.
In both my personal and professional life, I have the opposite conclusion. Bash is only for the simplest scripts and pipelines, and everything else (assuming you're going to use it more than once) gets written in Python or Go.
The Bash syntax is not daunting, or if it is that's never been the problem with Bash. The problem is that Bash or shell programming in general gives you a million ways to shoot yourself in the foot and maybe one or two obscure, funny-looking ways to do what you want. Like iterating over the files in a directory, for example. If you think that's easy in Bash you have either have a funny definition of easy or you're forgetting a few corner cases. Or getting the output from a program—did you know that $(my_prog) or `my_prog` modifies the program output?
Get a very brief reference book of every common UNIX command. Read all the commands, what they do, what options they take. Start using them.
Shells are most useful when they are used to tie together other programs. In order to do this, you have to know what all the command-line tools you have at your disposal are. Learn the tools, then start writing examples using them. Keep the examples and the docs somewhere to reference them later.
For quick reference, the command 'whatis' will give a blurb from the top of the command's man page. `whatis ls' "ls (1) - list directory contents". View many at once with "(cd /usr/bin; whatis * | grep -v noth)". Many often-used commands come in "util-linux" and "coreutils" packages. Read man pages completely when convenient.
It may also help to have a VM or desktop which has no GUI, where you will be forced to use the command-line. When I was starting out I used a desktop with no X server for months. You can get a lot more done than you think (and 'links -g' provides a graphical browser if you need images)
To learn more about Bash itself, you can look for server installation software packages made with Bash, or in the "init" tools distributed with big distros like RedHat, SuSE, etc before they used systemd. But it's better to get used to more shell-agnostic scripting using UNIX commands than it is to use Shell-specific language/syntax.
[+] [-] SEJeff|8 years ago|reply
I did it for an hour or so a night for a week or so.
That being said, a few of my personal favorites to memorize:
* Parameter expansion: https://www.gnu.org/software/bash/manual/html_node/Shell-Par...
* All of test(1) as you can use them in any if statement (/usr/bin/[ is a real command!): https://linux.die.net/man/1/test
* Knowing most of the bash internal variables: http://tldp.org/LDP/abs/html/internalvariables.html
* Keyboard shortcuts and how they are useful. A few example: CTRL-l (no need to ever use /usr/bin/clear), CTRL-k, CTRL-u, CTRL-e, CTRL-a, CTRL-w, CTRL-arrow left, CTRL-arrow right, CTRL-r (history reverse search with find as you type autocomplete)
The best way you can learn the shell is by using Linux as your primary desktop for at least a few months. You'll get very proficient very quickly by doing that.
[+] [-] _ncxu|8 years ago|reply
[+] [-] vram22|8 years ago|reply
One thing I have found that less people seem to know, is that the Unix metacharacters are expanded by the shell (bash etc.) not by individual commands. What this implies is that any command, whether built-in or written by you (in C, bash, Python or any other language), has metacharacter support automatically. That is, things like the file globbing/wildcard characters like *, ?, and [ ranges ].
This was not originally true on DOS (as a counterexample) and not sure whether it is true on Windows today (haven't checked), though I did notice that more commands seem to support wildcards in Windows nowadays.
Also, for some years now, Windows too has had redirections like:
command >file.txt 2>&1
(redirect stderr (2) to the same destination that stdout (1) is pointing to, i.e. file.txt), which Unix had from the start.
[+] [-] keithpeter|8 years ago|reply
[+] [-] mtdewcmu|8 years ago|reply
[1] http://www.gnu.org/software/coreutils/manual/html_node/index...
[+] [-] vram22|8 years ago|reply
Yes, and not only in an if statement. You can also use test or [ command in commands of the form:
test condition && commmand2
or
test condition || command2
which will only work if the condition is true or false, respectively (IIRC, need to check this).
[+] [-] lohengramm|8 years ago|reply
Also one should take a look at rlwrap after becoming comfortable with the keyboard shortcuts.
[+] [-] seorphates|8 years ago|reply
Mind your pipes and quotes. Guard your variables with braces. Do not export everything, test for and (try to) handle return codes and conditions and keep it simple (emphasis simple) but most of all just write it.
BASH (or Bourne) is ubiquitous when dealing with systems (vs programs). You don't need to be on the fashionable lang of the day by any measure. BASH, for most cases, will always be there, always ready and, in most cases, is the default human interface for deployed systems. As scripting languages go you don't need "better", you need dependability, zero dependencies with no requirement for modules or any other whizbangwoohoo plug-in. Language Fashionistas and personal preferences aside at least some level of fluency with BASH should be mandatory for anyone interfacing with a system.
[+] [-] curiousgal|8 years ago|reply
[+] [-] joshstaiger|8 years ago|reply
You're going to get a lot of snark from people saying things like "don't", or "learn python instead".
This epitomizes "a little knowledge is a dangerous thing".
Bash has many cringeworthy aspects, but we have to deal with the world as it is, not the world as we would like it to be, and the reality is that bash is the default shell on 99.9% of unix boxes you encounter — even if you use an alt shell on your machine.
Coworkers machine? Bash. Default AWS AMI? Bash. init script to bootstrap $DAEMON? Bash. ssh to a server at your workplace? Bash. Random O'Reilly Linux tutorial? Assumes bash.
My advice?
Take some time.
Sit down.
and read "man bash"
cover-to-cover.
at least once.
A lot of the illogical things in bash make a lot more sense once you understand its parsing/expansion rules. And once you understand what is in the language vs an external terminal program in your PATH.
Since that sounds unappealing (and I scoffed at that very advice for many years), I've also found the wooledge bash guide to be very helpful.
[+] [-] qrbLPHiKpiux|8 years ago|reply
[+] [-] soheilpro|8 years ago|reply
Basically, every time I learn something useful about a command, I add it to its mann page and then whenever I need it in the future, I simply run 'mann <command>' to find it.
Here's the current output of my 'mann sed', for example:
[+] [-] ben174|8 years ago|reply
https://github.com/ben174/dotfiles/blob/master/README.md
[+] [-] koala_man|8 years ago|reply
Bash is essentially a DSL for these. A lot of the weirdness you see in the language is due to these abstractions leaking through. For example:
* Quoting is building execve's argv parameter. It's hard to quote correctly if you don't know what exactly you're working towards.
* Redirections are opening and copying file descriptors. It explains their scope, order and nesting behavior.
* Variables are modifying and passing the environment, and their weird scope is due to forks imposed by the process model.
Once you know how you can do whatever you want in C through the basic syscalls, Bash is an extremely efficient and far less surprising shortcut to do it.
[+] [-] woodrowbarlow|8 years ago|reply
can you expand on this? does this elucidate, e.g., variable substitution and the difference between single- and double-quotes? or does it just help demonstrate when you need quotes for an argument that may contain whitespace?
[+] [-] racl101|8 years ago|reply
Do you recommend a book or something like that? And preferably for a beginner?
[+] [-] nunull|8 years ago|reply
[1]: https://github.com/koalaman/shellcheck
[+] [-] mafro|8 years ago|reply
It will highlight common mistakes, and their wiki explains each one detail and how you should use an alternate, better implementation.
[+] [-] 12s12m|8 years ago|reply
[+] [-] aMayn|8 years ago|reply
[+] [-] wyclif|8 years ago|reply
[+] [-] agentgt|8 years ago|reply
1. If you are not running a unix as your default OS switch to one (ie Linux or Mac).
2. Create a bin (~/bin) directory in your home directory of all your shells scripts and source control it. Any script you ever write put in that directory. Even if its not bash (ie python, perl). I find that it is critical to look at how you did things previously to help you learn as well as it saves time.
3. Any command that is complicated one liner that you create or see on the internet... create script and put in the bin directory mentioned above.
4. Optimize your personal bin directory and review frequently.
5. If you run Linux build your system from scratch (ie read Linux from scratch).
6. Bonus to the above: Automate the creation of your personal system through Packer and Bash!
7. Find where things are not automated.
8. Bash is more than just the shell. A knowledge of GNU coreutils as well as tmux/screen is worthwhile and highly recommended.
9. Learn the readline shortcuts. Particularly "ctrl-r".
[+] [-] bhaak|8 years ago|reply
Anything that is not simple in bash gets hard to read and debug and probably is wrong on some subtle levels.
I have a rule of thumb that any shell script that grows beyond a screenful of lines gets redone in a proper scripting language.
[+] [-] rajeshmr|8 years ago|reply
http://www.tldp.org/LDP/Bash-Beginners-Guide/html/
EDIT : Additional links -
Advanced - http://tldp.org/LDP/abs/html/
Bash programming - http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
[+] [-] mapcars|8 years ago|reply
[+] [-] fergie|8 years ago|reply
[+] [-] eof|8 years ago|reply
I use bash/shell scripts frequently and have many running 'in production' as cron jobs th at run various jobs or manipulate data for other jobs to run on.
One thing I really like about pure shell is that it's extremely portable and transparent about what it's doing.
I still have to re-learn control structures almost everytime I write a new script, I don't try and memorize [[]] vs [] and all the weird ways equality can work, I just google each time and answers are always on top (once you know what you're looking for).
[+] [-] tetraodonpuffer|8 years ago|reply
[+] [-] felixschl|8 years ago|reply
[+] [-] jdreaver|8 years ago|reply
[+] [-] antirez|8 years ago|reply
[+] [-] soveran|8 years ago|reply
I invite you to read about it: http://doc.cat-v.org/plan_9/4th_edition/papers/rc.
I find the control structures simpler and more elegant, and overall its design feels more consistent. For example, consider an if statement in bash:
And now in rc: Or a case statement in bash: And expressed in rc: In the past, I've used it as my shell too, but now I use it only for scripting. I think you can install it in most platforms.[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] benjamincburns|8 years ago|reply
https://web.archive.org/web/20161227222637/http://blog.extra...
From there, move on to using the shell as your IDE. How? First, understand the Unix philosophy. I think Ted Dzubia describes this pretty well in his Taco Bell Programming blog posting:
http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra...
Great, so now you understand that there are a bunch of useful tools out there and you can string them together to do great things. Now you need to discover the tools themselves.
If you're a "read the dictionary" kind of person, go ahead and start off w/ the Gnu Coreutils documentation. https://www.gnu.org/doc/doc.html
However, if you're like me you'll learn fastest by watching other people work. In this case, I have to point back to Gary Bernhardt again. Specifically, his "Composing a Unix Command Line" screencast will open your eyes wide and very quickly introduce you to a range of incredibly useful coreutils programs in the context of solving a very specific problem. This content is $29/mo, but I'd argue it's money well spent. https://www.destroyallsoftware.com/screencasts/catalog/compo...
[+] [-] brianon99|8 years ago|reply
I learned to use shell, about 7 years ago, by reading O'Reilly "Classic Shell Scripting". It is well written, and teach you something that you can hardly learn from google. But don't try to remember everything, especially those advanced string manipulation syntax, because one would usually use a scripting language such as ruby for advanced job.
[+] [-] lamby|8 years ago|reply
I can second this recommendation. :)
[+] [-] wingerlang|8 years ago|reply
[+] [-] rajeshmr|8 years ago|reply
[+] [-] fiatjaf|8 years ago|reply
[1]: https://github.com/fiatjaf/react-site
[+] [-] brad0|8 years ago|reply
I'd say learn the following topics:
pipe grep sed awk find
Once you feel comfortable using and combining these tools you should be able to find out the rest by yourself.
[+] [-] fenwick67|8 years ago|reply
[+] [-] TheGrassyKnoll|8 years ago|reply
[+] [-] moondev|8 years ago|reply
[+] [-] klodolph|8 years ago|reply
The Bash syntax is not daunting, or if it is that's never been the problem with Bash. The problem is that Bash or shell programming in general gives you a million ways to shoot yourself in the foot and maybe one or two obscure, funny-looking ways to do what you want. Like iterating over the files in a directory, for example. If you think that's easy in Bash you have either have a funny definition of easy or you're forgetting a few corner cases. Or getting the output from a program—did you know that $(my_prog) or `my_prog` modifies the program output?
For containers we do everything declaratively.
[+] [-] peterwwillis|8 years ago|reply
Get a very brief reference book of every common UNIX command. Read all the commands, what they do, what options they take. Start using them.
Shells are most useful when they are used to tie together other programs. In order to do this, you have to know what all the command-line tools you have at your disposal are. Learn the tools, then start writing examples using them. Keep the examples and the docs somewhere to reference them later.
For quick reference, the command 'whatis' will give a blurb from the top of the command's man page. `whatis ls' "ls (1) - list directory contents". View many at once with "(cd /usr/bin; whatis * | grep -v noth)". Many often-used commands come in "util-linux" and "coreutils" packages. Read man pages completely when convenient.
It may also help to have a VM or desktop which has no GUI, where you will be forced to use the command-line. When I was starting out I used a desktop with no X server for months. You can get a lot more done than you think (and 'links -g' provides a graphical browser if you need images)
To learn more about Bash itself, you can look for server installation software packages made with Bash, or in the "init" tools distributed with big distros like RedHat, SuSE, etc before they used systemd. But it's better to get used to more shell-agnostic scripting using UNIX commands than it is to use Shell-specific language/syntax.
[+] [-] keithpeter|8 years ago|reply
Without X presumably needs framebuffer or something?