This post describes a rather specific use of Git’s rebase command. Rebase is a powerful and complex topic which I am not fully covering here. I will attempt a more thorough look at rebase in general in an upcoming post. If you are newer to Git, as I am, you might want to refer to a few of my previous posts covering some of the basics:
A handy feature of Git is the ability to use the interactive rebase command to combine, re-organize, and otherwise clean up a messy commit history once development of a feature or bug fix is complete. I am a firm believer in the practice of “commit early, commit often” approach. However, this can leave untold number of commits which, when merged back into mainline development, clutter up a repository.
How often do you end up creating a commit which consists of some modified whitespace after cleaning up a code file, or a commit arising from updating some comments or renaming variables?
Oftentimes, when we complete a feature or fix, we can clearly see how (hindsight being, more often than not, 20/20)we would have liked the code to come together. In fact, for myself, I often wish I could just combine all of the relevant commits into one, which I could then merge back into master. For more complex features, or those which may have undergone substantial evolution during development, it can sometimes become clear how the commit history should have occurred, and not necessarily in the order of manner in which it actually did.
For reasons of maintenance, or particularly if we plan to push our code out to a public repository, it would be nice if we could straighten things up a little, and present a commit history to the world which will:
- Allow us to rollback changes in a manner which does not cripple our codebase.
- Make sense to those who might need to maintain or perhaps branch from our repository.
- Not leave us looking like disorganized idiots (“Did you really commit the deletion of a single empty line?”).
Interactive rebase gives us a way to write that history.
A Simple Example
I will start with an overly simple example. Assume a straightforward git repository consisting of a master branch. In order to add a new feature, you create a feature branch, names (appropriately, feature). On the new feature branch, you work up the basic functionality of the new feature, knowing you will be fine-tuning things shortly. However, you commit the work to this point.
While developing the primary feature code, you noticed a bug in another part of the codebase, perhaps impacting the development of your new feature. You quickly spot a solution, fix the bug, and commit.
Once that is done, you fine-tune the new feature code until it is right where you want it. After committing those changes, you decide to do some cleanup. You remove unnecessary whitespace, rename some methods for clarity, and other housekeeping tasks. After committing this last batch of changes, you are ready to merge your work into master.
Except . . . the new feature, in this case, is not overly complex, and you sure wish you could merge it into master as a single commit. Also, that bug fix should remain separate, as that should be applied to master separately from the new feature. That way, if for whatever reason the new feature needs to be rolled back, it can be done so independently of the bug fix. As of right now, the state of your repo is something like this:
Repo After Feature Development
If we were to perform a standard merge at this point, our repo would look like this:
Repo State After Hypothetical Standard Merge:
As I stated, this is a rather simplified example. But you can see where we might want to combine our feature development (commits C, D, and E) into a single commit before merging to our master branch, while leaving our bug fix (commit C) on its own.
Use Git’s Interactive Rebase Feature to Re-Write History
I created a (very contrived) sample project with a main application on master, and a feature branch. After writing some trivial code to illustrate the commit history we are using in our example, I ran Gitk, which now looks like this:
The Feature Branch Shown in Gitk:
Similarly, the git log command, run from within our feature branch, shows us this:
Git Log Contents for Feature Branch:
If we want to re-write the commit history here such that commits B, D, and E are combined into a single new commit, we can tell git to run an interactive rebase which will allow us to provide direction to git with respect to how we would like it to handle each of the commits involved in the rebase.
What we are actually going to do here, consistent with how rebase works, is rewrite one or more commits in the same branch onto the parent commit we specify in the command below. Because we will be using the interactive flag, we will be allowed to reorder the commits we have selected for the operation, as well as provide some other direction as to how we want git to handle the process.
The Command Syntax is like this:
$ git rebase -i HEAD~n
In the above, n represents the number of commits we would like to edit. However, it is important to note that this actually points to the parent of commit n as the rebase target. So, having examined the state of my feature branch, I can see that I would like clean up commits B thru E. I want to combine Commits B, D and E into s single commit, and leave commit C as a stand-alone (the bug fix).
Here is what I will type into the command shell:
$ git rebase -i HEAD~4
After I type that in and hit enter, my editor widow opens, and looks like this:
The Editor Window After Entering the Interactive Rebase Command:
In this case, the first four lines which start with “pick” represent the commits I have chosen to edit. Note the order in which they appear. When we are finished editing, git will process our revised commits from top to bottom. This is significant because we can re-order them if we so choose. Also note the comments below. We are given some options here which we can apply to the editing process. If we replace the pick keyword with one of these flags for any specific commit, that operation will be performed on that commit.
Of particular interest to us in this case is the squash or s flag. When we replace pick with squash, we tell git to combine that commit with the previous one (remember the processing order, top to bottom). Remember what we want to do in this rather contrived example – we want to combine commits B, D, and E into one. However, the current processing order doesn’t allow us to do this, because our bug fix is sandwiched between commits B and D.
So first we will need to change the processing order by moving some commits around. In the editor, we can easily Cut/Paste the top two lines to look like this instead:
Changing the Commit Processing Order in the Editor:
Now, git will process our edited commits as ordered above, starting with C, then B, then D, then E. Next, we tell git to squash commits D and E by replacing the pick flag with s or squash, so that they both roll up into the previous commit (Commit B, in this case):
Change the Commit Processing Flags in the Editor:
Now, having made the changes above, SAVE THE EDITOR CONTENTS (I often forget this step!) and close the editor window. Your command line window will return while Git processes the commits, and then the editor reappears. Git now wants you to sort out the commit messages, so that the newly created commit history will make sense.
In our example case, the first thing that we see is this:
Making Sense of the Commit History and Messages in the Editor:
We can see that the three commit messages in question are from our original commits B, D, and E, related to our new feature. We can clear all of these (and the comments included with them) and type a new, more meaningful commit message. Note that any line prefixed with a hash symbol (“#”) will be ignored:
Modifying the Commit Message for the Combined Feature Commit in the Editor:
Now, SAVE THE EDITOR CONTENT AGAIN and close the editor window. The command line window now looks like this:
Command Window After Successful Interactive Rebase:
If we run git log now, we see the following:
Git Log After Successful Interactive Rebase:
If we run Gitk, we see this:
As we can see, in the above, we now have a single commit representing the implementation of our new feature, and the original bug fix (previously, commit C) where previously there were four distinct commits. We have re-written history, in preparation for a sensible merge into our master branch.
Notice that our bug fix commit (previously commit C) now has a different hash identifier than previously, and the combined commit is brand-new as well. This is important, because again, HISTORY HAS BEEN CHANGED. The implication here is that if the previous history has been pushed to a public repository (or otherwise cloned, perhaps locally) the history of the feature branch will no longer be the same. In other words, we might as well think of Commit C as a new commit F, and the combined commit as a new commit G.
IMPORTANT: This should not be performed on commits which have been pushed to a public repository, if other committers are working with your code!!! It is generally best to reserve this technique for code in your local working tree, before publishing.
Graphically, our repository now looks like THIS:
Repo After Interactive Rebase:
But, What About Merge Conflicts During the Interactive Rebase?
As often as not, you will run into merge conflicts during this process. I contrived the example above specifically to illustrate the ability to combine commits, and studiously made sure conflicts would not occur here.
In cases where there are conflicts, Git will stop the rebase process, and demand that you fix any conflicts before resuming. Once you have resolved any conflicts, type git rebase –continue in the command window, and the process will resume. We’ll take a look at conflict resolution in another post.
Additional Resources
John on Google CodeProject
NOTE: This post is mainly targeted at newer git users who may want a more detailed look at what is happening. If you consider yourself comfortable with git, I have also created a “reference” version that skips most of the narrative:
The Quick Reference
- Link to condensed version HERE
Also, some navigation aids, because this is a long-ish post:
Problems introduced by “Nested” library projects in a Git Repository
Often it is the case I find myself building out a project and incorporating another library I have been working on (such as a custom control) as a dependency. The library in question may, in fact, be consumed by other projects as well (that IS the purpose of creating such libraries, right?). Equally as often, the project I am referencing as a dependency in my main project will itself be in a state of evolution. A problem arises related to either or both of the following:
- The newly introduced dependency (the “sub-project” e.g. custom control or such) is also being actively developed, and I will want the option to pull in any changes as that library evolves.
- I may, in the course of working on my “main” project, make some modifications to the sub-project, some or all of which I decide should be incorporated into the main line of development of the shared library.
John on Google CodeProject 
Set a reference to the binaries of the shared library from within my main project, perform all modifications to the shared library from within that project, and re-build so that changes are reflected in the binary outputs. - Use Git “Sub modules” to refer to a specific commit within the library project, and manage the updating of these commit pointers as the library project evolves.
- Copy the dependency project source into the main project. Yuck. Isn’t the whole idea of libraries to avoid this type of thing?
I have not used Git sub modules much, but my limited experience with them indicates to me that outside of certain circumstances, they are a pain. Equally unsatisfactory under the conditions I described in the first paragraph is the managing of a standard binary reference (especially in cases where I may need to customize the shared library within my main project, while keeping it up to date with the development repo. The third option is not even worth considering.
In searching for an optimal way to handle the “nested libraries” problem, I came across a few articles on the Sub-tree Merge workflow in GIT. In this workflow, we accomplish the following (this represents my understanding of what is going on – if you know differently, please let me know in the comments!):
- Maintain a tracking branch which tracks the remote shared library project, from which we can pull in changes made in mainline development of the shared library. This remote tracking branch tracks and represents the history of the shared library, distinctly form the history of the main project.
- From this remote, create a “subtree” as a subdirectory within the master branch (or whichever branch we designate) of the main project. The new sub-directory contains a copy of the shared library source code. This subtree does not bring with it the history of commits in the actual shared library. Instead, it joins and shares history with the main project.
- Once this is done, we can pull new change sets down from the shared library remote as needed, and merge them into our sub tree (there’s a trick to this, though, so keep reading). Likewise, we can also merge changes we make within our subtree directory out to our remote tracking branch, and push them up to the shared library remote.
Essentially, we will have told Git to read the contents of the remote tracking branch into a sub-directory on branch master, and remember that this sub-directory represents a subtree based on the remote tracking branch. Graphically, I think of it a little like this:
This is How I Picture the Git Subtree Merge Workflow:
Walk Through:
Let’s walk through the process of creating a subtree within an example project, and then we’ll discuss some things to pay attention to when it comes to merging changes to and from the subtree remote.
I have created an example project as the “main” project for use in the following examples. I will be using a Pdf Merge utility project created for another post on this blog as the “shared library.” The main project is called, unsurprisingly, “MainProject.” The shared library is my oh-so-useful iTextToolsExample. We will assume I have already created MainProject, created a local repository on my development machine, and a remote repo on Github.
From the Git command line, we set up a remote that points at the repository for the shared library project. I will be using an example project as the main project, and adding a shared library I created for another post as a subtree.
Git Add Remote Syntax:
$ git add <local_remote_name> <remote_location>
e.g:
$ git remote add itext_remote https://github.com/xivSolutions/iTextToolsExample.git
Next we need to fetch down the shared library into our working directory using the remote we just created. Note, we want to use “fetch” here, as we don’t want to commit what we add to branch master:
Git Fetch Syntax:
$ git fetch <remote_name>
e.g:
$ git fetch itext_remote
Note the warning after we fetch about “no common commits.” That’s ok – we wouldn’t expect there to be an common history here, we are importing a library for that very reason!
Now, in keeping with the illustration above, we want to get the contents of the shared library tracked in their own remote tracking branch, so we can keep things in sync separate from master:
Checkout to New Branch Syntax:
$ git checkout -b <new_branch_name> remote_name/branch_name
e.g:
$ git checkout –b itext_branch itext_remote/master
Now, the new tracking branch points to the root of the shared library project. We can check this by examining the directory contents here in itext_branch:
Directory Listing in iText Branch:
And then checkout master and compare:
Directory Listing in Master:
Now is when we perform the magic. We will use git read-tree to read the contents of itext_branch into a subdirectory of master. This will be essentially the same as copying it all in, with on small exception: Git knows and remembers that this is a subtree related to the remote tracking branch itext_branch.
Git Read-Tree Syntax:
$ git read-tree --prefix=<subdirectory_name>/ –u <shared_library_branch>
e.g:
$ git read-tree –-prefix=itextTools/ –u itext_branch
In the above, <subdirectory_name> can be anything you like – it is going to be the name of a new “folder” or subdirectory within your mater branch. The --prefix flag tells git that this will be the name of a directory. I usually use the same name as the original shared library project. Note it is important to include the forward-slash after this. The –s flag tells git to go ahead and update the working tree with the new changes. Also note, we execute this command from within branch master:
Running Git Read-Tree From Branch Master:
Now, if we run git status, we can see that a bevy of new files have been added to our master branch, in a directory named (not surprisingly) “iTextTools:”
Status of Branch Master After Read-Tree Operation:
After you have performed the read-tree operation, the files have been read into our master branch as described above. The important thing here is that the history they share with the remote source repository (as referenced by the itext_branch) has NOT. We can commit this as a single commit:
Commit the New Directory and Files:
We have now added the shared iTextTools library as a subtree in our master branch. If we check the contents of our project directory on branch master now, we see the new iTextTools subdirectory:
Now, what if there are changes introduced to the main shared library source project? How can keep our sub-project in sync with mainline development on the shared library?
If we need to pull changes down from the original shared library source into our sub-project, we can simply checkout the remote tracking branch itext_branch and use git pull, which will pull down and merge any changes made since we originally fetched the project. Say we (or whoever maintains the itextTools project) added a new class named SomeNewClass and solution file to the main development repo for the itextTools library:
Pulling Changes down from the Shared Library Remote:
While the above pulls the changes down, we need to be careful with this next step – merging the new change set into our subtree in master. The changes we just pulled down bring with them all of the history from the iText remote repository. In almost all cases, we do not want to merge this history with that of our main project, so we can’t simply do a merge or rebase to get the new changes into master. Instead, we want to use the following:
Squash and No-Commit Change Sets into the Subtree (Syntax):
$ git merge --squash -s subtree --no-commit <source_branch>
e.g:
$ git merge --squash –s subtree –-no-commit itext_branch
In the above, the --s flag indicates that you are going to specify which merge strategy git should use in merging changes. This is followed by the strategy itself, which in this case is the subtree strategy. We are also telling git to squash the commits from <source_branch> into a single change set, and to not automatically commit when the merge is complete.
So, now that we have pulled the changes down into our remote tracking branch, we checkout master, and execute the above command:
Merging Changes from Remote Shared Library into Subtree (from branch master):
If we check the status of master again, we see the changes from the remote iText library repo have been merged into the appropriate subdirectory, and staged for commit:
Status of Master After Subtree Merge Strategy with --Squash and --No-Commit:
From here, simply commit (with a descriptive commit message!).
If you made some changes to the shared library code within your subproject and decide you want to push them out to the main development repo, the process is essentially the same, in reverse. You merge from the subtree directory within the main project (again, explicitly specifying the Subtree Merge strategy), out to the tracking branch that points to the shared library remote. Then push the changes up to the remote.
Say we made some changes to SomeNewClass while working in MainProject and we decided they should be incorporated into mainline development for the itextTools project main repo:
- Once you have made changes to the shared library from within branch master of your main project, commit the changes as normal.
- Then, checkout the tracking branch that points to the shared library remote.
Now, we use the same basic merge command from before, in the opposite direction:
Syntax to Merge from Subtree to the Remote Tracking Branch:
$ git merge --squash -s subtree --no-commit <source_branch>
e.g:
$ git merge --squash –s subtree --no-commit master
Merging the itextTools Subtree into the Remote Tracking Branch:
As in the previous example, we can check Git status and see that the changes we made in our sub-tree project have been merged into the remote tracking branch and are now staged for commit:
Now commit the changes. I usually state clearly in the commit message that these are changes from <project>/subtree so that, when added to the shared library, I can clearly identify that this set of changes came from the sub-project. Now it’s time to push the changes back up to the main shared library repo.
Pushing from Tracking Branch to Shared Library Remote Syntax:
$ git push <shared_library_remote> <library_tracking_branch>
e.g:
$ git push itext_remote itext_branch
Pushing to the Remote Repository:
Note that using the syntax above will push the changes into a new branch in my remote repo, named (not surprisingly) itext_branch. This will allow me to apply the changes to the master or production branch of the shared library using a merge or pull request (in the case where I am not the maintainer of the shared library).
There is More To It
I have brushed through the basics here. There are numerous ways to condense some of the commands I used here, and like most things, there are probably dozens of different viewpoints on how to best manage project dependencies in source control. The Subtree Strategy has been working best for me, in my working context, but I welcome your feedback, and especially, corrections where I might have it wrong, or have missed seeing a better way.
Thanks for reading.
Additional Resources
I am not going to pretend there aren’t any number of good resources from which I learned the above. Most of the information here I assimilated from elsewhere. Here I have striven to simply present my own understanding, in a way which might have helped *ME* when I was trying to figure it out. Here are some of those sources:
John on Google CodeProject
The Subtree Merge Workflow offers a relatively painless mechanism for managing shared library source code as a component of a larger project. If you have ever had a shared library, custom control, or other component under development that you also wanted to use within one or more dependent projects, you no doubt understand what I mean.
This post is a condensed version of a longer post with more examples, and aimed at those less familiar with Git. If, like me, you have a little trouble following what is going on here, check out the other post first. This one is more of a reference.
In searching for an optimal way to handle the “nested libraries” problem, I came across a few articles on the Sub-tree Merge workflow in GIT. In this workflow, we accomplish the following (this represents my understanding of what is going on – if you know differently, please let me know in the comments!):
- Maintain a tracking branch which tracks the remote shared library project, from which we can pull in changes made in mainline development of the shared library. This remote tracking branch tracks and represents the history of the shared library, distinctly form the history of the main project.
- From this remote, create a “subtree” as a subdirectory within the master branch (or whichever branch we designate) of the main project. The new sub-directory contains a copy of the shared library source code. This subtree does not bring with it the history of commits in the actual shared library. Instead, it joins and shares history with the main project.
- Once this is done, we can pull new change sets down from the shared library remote as needed, and merge them into our sub tree (there’s a trick to this, though, so keep reading). Likewise, we can also merge changes we make within our subtree directory out to our remote tracking branch, and push them up to the shared library remote.
My Mental Picture of the Subtree Workflow:
In my head, I picture the subtree relationships something like THIS:
Create a Subtree in your Main Project
In the examples below, I will use MainProject as the name of the main project, in which I want to utilize a shared library which is also under development. I will use the iTextTools library from a previous post as the example shared library.
Step 1: Point a Remote in the Main Project at the Shared Library Repository:
$ git remote add <library_remote_name> <library_remote_location>
e.g.
$ git remote add itext_remote https://github.com/xivSolutions/iTextToolsExample.git
Step 2: Fetch Shared Library into the Working Tree:
$ git fetch <library_remote_name>
e.g:
$ git fetch itext_remote
Step 3: Checkout a New Tracking Branch Based on the Shared Library Remote:
$ git checkout -b <new_branch_name> remote_name/branch_name
e.g:
git checkout –b itext_branch itext_remote/master
Step 4: Read the Library Project Into Master as a Subtree Directory:
Before performing this step, switch back to branch master (or whatever your chosen target branch is). The read-tree command is going to read the contents of the remote library tracking branch into the branch you execute the read-tree command from.
- The --prefix option establishes the name of the subdirectory into which the contents of the read will be placed.
- The --u flag tells git to update the contents of the working tree once the read is complete. Files will be staged for commit when the read is complete.
$ git read-tree --prefix=<subdirectory_name>/ -u <shared_library_branch>
e.g:
$ git read-tree -–prefix=iTextTools/ –u itext_branch
Step 5: Commit the New Files:
Commit as normal. From this point forward, the files added in the new subdirectory will be tracked as part of this branch, and share history with MainProject. However, Git also knows about the subtree directory. We can leverage this information to merge changes between the local version of our shared library and the remote library repository, without merging the separate project histories (which have nothing in common other than the use of the library project source code).
As we will see, there are a few things to keep in mind when doing so, but overall it is pretty simple.
Update the Library Sub-Project from the Shared Library Remote
If changes are made to the shared library at the source repository, we will likely want to bring them into our sub-project so that the subproject is up-to-date with the latest code. The process is a little different than the standard pull, fetch/merge, or rebase process we are accustomed to, because most of the time, we don’t want to bring the commit history from the shared library remote into our main project (as they are unrelated and share no common commits).
Before performing this update, I recommend committing any other local changes you may have introduced in MainProject so that, when you commit the imported changes retrieved for the shared library code, it will stand as its own unique commit.
Step 1: Checkout the Shared Library Remote Tracking Branch:
$ git checkout <library_branch_name>
e.g:
$ git checkout itext_branch
Step 2: Pull from Remote Library Repository:
When you set up the remote tracking branch originally, you also pointed it at the the shared library remote repo, so Git understands that git pull from this branch means to fetch/merge from the proper location:
$ git pull
Step 3: Checkout Target Branch
Before we perform any merging, we obviously need to make sure we are back on the branch we want to merge the changes into:
$ git checkout <target_branch>
e.g:
$ git checkout master
Step 4: Merge Changesets without Merging History
This is where things differ a little from a normal merge or rebase. We want to merge all of the changes, but we don’t want to bring any commit history with it. Also, because we are targeting our subtree directory, we need to explicitly specify the subtree merge strategy. In the below, the –s flag tells git we will be explicitly prescribing a merge strategy, followed by the strategy to use (obviously, subtree in this case):
$ git merge --squash -s subtree --no-commit <library_branch>
e.g:
$ git merge --squash -s subtree --no-commit itext_branch
Step 5: Commit:
Now the sub-project version of the shared library is up-to-date with the remote. Commit as normal. I recommend using a descriptive commit message indicating that this commit is specifically related to updating the subtree project.
Update the Shared Library Remote with Changes from the Library Sub-Project
It may happen that while working in your main project, you introduce modifications or additional features to the shared library code that you decide should be integrated into the Shared Library source repository. Doing so is essentially the reverse of the preceding. Simply merge the changes from your working branch out to the shared library remote tracking branch (again, without bringing the local project history along), then push to the shared library remote (and/or submit a pull request, depending upon who maintains the shared library repository).
Obviously, prior to performing the following steps, commit your changes in the working tree on branch master (or whatever your working branch is) in your main project.
Step 1: Checkout the Shared Library Remote Tracking Branch
$ git checkout <library_branch>
e.g:
$ git checkout itext_branch
Step 2: Merge Changes from the Library Sub-Project Without Merging History
As with the previous situation, in all but some very odd circumstance we don’t want to merge the two projects histories together, so we use the --squash option when we merge. Also as before, we explicitly specify the subtree merge strategy:
$ git merge --squash -s subtree --no-commit <source_branch>
e.g:
$ git merge –squash –s subtree –no-commit master
Step 3: Commit the Merged Changes:
Now, assuming there were no merge conflicts, commit as normal, again using a descriptive commit message identifying the commit as a merge from a subtree in an outside project.
Step 4: Push Changes to the Shared Library Repo
When we go to push changes from our remote tracking branch to the shared library remote, the command below will actually push our changes on a new branch (named, not surprisingly, “itext_branch”). For me, this is the behavior I want. This way, I (or whoever is the maintainer of the remote library) can integrate the changes into the mainline of development from within the library remote. In other words, we are not merging directly back into master (rarely how we would want to do things under these circumstances). Obviously, if we are NOT the maintainer of the shared library, this is where a pull request would come in instead of a direct merge.
$ git push <library_remote> <library_branch>
e.g:
$ git push itext_remote itext_branch
Final Thoughts
I do not claim to have the authoritative post on this topic by any means. I have written this, and the longer-form post, as much as a learning exercise and reference for myself as anything else. I have sought to present the information in a way which would have helped me understand ti when I was first learning some of Git’s more advanced features and techniques.
I am aware that there are more condensed version of some of the commands used, and quite possibly better ways of doing this. If you have read this far, and have noted some incredible stupidity on my part herein, please do point it out in the comments.
Additional Resources
As I learned this, I referred frequently to the following, among other sources:
John on Google CodeProject
CodeProject
Up to this point . . .
This is the third part in a series focused on getting familiar with Git for Windows developers. The series assumes little or no experience using the command line. If you missed the first few posts, here is what we have covered so far:
In the previous post, we learned how to navigate directories within the Windows file system, create new directories, and delete directories. Now let’s look at creating files, and performing rudimentary (VERY rudimentary) text editing from the Bash command prompt.
Before We Proceed . . .
Let’s create a directory to use for the following examples. Open your Bash Command Line Interface (CLI) and make sure you are in your default Windows user folder. Create a new directory within your user folder called BashExcercises, and then move into that directory (Review the previous post if you are unsure how to do this):
Create a Directory for the Following Exercises
Also remember, when we are explaining syntax, we will often use square brackets to denote optional items, and angle brackets to denote use-provided content. In the following:
mkdir [options] <FileName>
We do not type the square brackets or angle brackets. The [options] are, well, optional, meaning there may be none or there may be many, and the <Filename> is required and provided by the user.
Bash: Create/Edit Files (touch/echo)
There are a couple of ways to create a new file using the Bash command prompt. The most basic is the touch command. The purpose of the touch command is ostensibly to change/modify the date/time stamps of files. However, it is commonly used to create new files as well, since, when used without any optional arguments, it does just that: creates a new file, with a date stamp.
Using the Touch Command to Create a New Text File
The syntax for the touch command is
Create a new file with a date stamp:
touch [options] <filename>
There are a number of options for use with the touch command, but for now, we are going to focus on simple file creation, and we won’t be using any of the options. If we want to create a simple text file, we can type:
Create a new Text File Named MyNewTextFile:
$ touch MyNewTextFile.txt
Type the above into your Bash window and hit enter:
Remember from the previous post that in many cases, when a Bash command executes successfully, we are rewarded with simply a new command prompt. However, we can use the ls command we learned from the last post to examine the contents of our directory:
What do you know? There is is. Now, you could open the new file using Notepad (or the text editor of your choosing), but you won’t see much. We have created an empty text file. To open the file using Notepad from the Bash prompt, simply type:
Open MyNewTextFile.txt in Notepad using Bash:
$ notepad MyNewTextFile.txt
Or, of course, you could open the file the old-fashioned way - by double clicking on it with your mouse!
Using the echo Command to Write Strings as Output
Before we look at using echo to create a new file, we should understand a little about what echo DOES.
The primary design purpose of echo is apparently to write strings to the standard output (usually, the Bash CLI).
The syntax for echo is:
Echo Command Syntax:
$ echo [options] <string>
Lets try that now. Type the following into Bash:
Output a line of Text to the Bash Screen:
$ echo show this line on the screen
We can also tell the echo command to write a line of text to a file, using the following syntax:
Syntax: Append text to the end of a file:
$ echo [options] <Text string> >> <FileName>
By using the >> (double right-hand angle-brackets) operator, we tell Bash to append the text to the end of the file. Type the following into the Bash window and hit enter:
$ echo This line was added to the end of the file >> MyNewTextFile.txt
Now, if we use our $ notepad MyNewTextFile.txt Command, we see the following:
Close that window, and let’s add another line. Type the following and hit enter:
Adding Another Line of Text:
$ echo This is another line of text >> MyNewTextFile.txt
The use $ notepad MyNewTextFile.txt again:
Huh. Looks like it worked, kind of. Apparently, Bash isn’t kidding about appending the new text to the end of the file. Happily, we can use the > (a single right-hand angle bracket) operator to replace the text in the current file and get rid of that mess. Type this and hit enter. Then open the file in Notepad again:
Replace the Text in a File with a New Text String:
$ echo This line replaced the old mess > MyNewTextFile.txt
Your Bash window should look like THIS:
And you should see this in your Notepad window:
Use the Echo Command to Create a New File with Text Content:
Ok, back to the original subject - creating a new text file. We can use echo to create a new file, and include the specified text string as content. In fact, it works the same as when we appended or replaced the text in an existing file, because, lacking an option argument telling it not to, Bash will assume that if the file specified in our echo command does not exist, that we want to create one. Type the following and hit enter:
$ echo This is a new line in a new file >> DifferentTextFile.txt
Then use ls -1 to display the directory contents. This is what your Bash window should look like:
Note the presence of our new file. Let’s open it in Notepad, and see what there is to see:
Well, well well. Whadd’ya know!
Bash: Remove Files (rm)
Ok, now lets get rid of our first file. To remove (delete) files using Bash, we use the rm command. The syntax for removing files is as follows;
rm [options] <FileName>
SO now, let’s delete that first file, MyNewTextFile.txt. Type the following, hit enter, then use the ls -1 command to display the directory contents:
rm MyNewTextFile.txt
Your Bash window should look like this:
Remove a Directory and all Contents (rm -rf)
In the previous post, we discussed how to remove an empty directory. But what if we want to remove a direcory and all of its contents? We use the rm command with the -r and -f options:
Syntax for Removing a Directory and All of its Contents:
rm -rf <FolderName>
So now we are going to remove the BashExcercises directory we created at the beginning of this post. Of course, we can’t remove a directory if we are currently IN that directory. So first, let’s return to the directory above (in this case, our home folder) using the cd .. Comand (the cd Command, followed by a single space, followed by two periods in a row. This one means “move up one level”).
Next, type the following into your Bash window, hit enter, and then use the ls -d */ command to view the directories within the current directory (which should be, at this point, your Windows User folder):
$ rm -rf BashExcercises
When you are done, your Bash window should resemble this (note that the directories are organized alphabetically in columns, and the BashExcercises directory is no longer there. Also note that for obvious reasons, your user folder will contain different folders than mine!).
Now, let’s use everything we just learned in this post, plus we’ll add one final tidbit at the end which you may find useful. We’re going to step through this right quick. We will:
- Add one more new directory
- Add a file containing a line of text to the new directory
- Remove the directory and its contents
- Have Bash tell us what was removed.
1. Add a New Directory Named OneLastFolder to our current directory:
$ mkdir OneLastFolder
2. Add a New File Named OneLastFile.txt, containing the text “One more text line” to the new folder:
$ echo One more text line >> OneLastFolder/OneMoreFile.txt
Notice in the above, I did not navigate into the new folder to create the new file, but simply used the relative path from my current location.
3. Remove the directory and all its contents, and have Bash tell us what was done:
rm -rfv OneLastFolder
Notice in the above, I added one more optional argument, the -v option (for “verbose”). This tells Bash to print the actions it took to the output window.
When you have done all of that, your window should look like this:
There you have it. You now have the most basic set of tools for using the Bash Command Line Interface. From here, we are ready to examine the basic command line operations required to start using Git to manage our code. We’ll pick that up in the next post in this series.
Additional Resources:
John on Google
This is the second installment of a multi-part series about getting your feet wet with Git for Windows Developers. If this is your first time here, you may want to review Part I of the series.
If you have been following along so far, you likely have by now installed msysgit on your development machine, and performed the initial configuration necessary to begin using git to track changes in your source code. So, what next?
Well, before we can do much with Git, we need to make sure we have at least a token familiarity with using the Bash command prompt. To do this, we will walk through some basic commands related to navigating in bash, getting information about the file system and environment, and working with directories and files. All of these are necessary in order to work with Git using the Bash command line.
Since these posts can get long (due in large part to the large number of screen shots), this may span a few posts. It’s not as bad as it looks, really. Lots of pictures.
If you are already familiar with the concepts here, and mainly just need help with the proper syntax for performing these actions in Bash, refer to my Basic Git Command Line Reference Post which contains a listing of the most common git and Bash commands without all the narrative.
Otherwise, read on, and let’s get comfortable using the Bash command line in a Windows environment.
Get Acquainted with Bash
The Bash command line comes from a Unix heritage, and even if you are familiar with the Windows command line, or Powershell, using Bash is different.
First, we should get a little comfortable moving around our file system and performing basic tasks with files and directories.
When you first open the Bash command window, you should see something like this:
In the above, the text in green is the User Name and Computer Name, displayed using the convention UserName@ComputerName. The curlicue character following means that we are in our default directory (or folder), usually our Windows User Folder.
A line which begins with the default prompt (in this case, a $ symbol) is a line on which you will enter a command. Lines without the $ symbol will represent feedback from the system. Each time you type in a command and hit enter, Bash will execute the command, return whatever output (if any) and the leave you with a new prompt.
Note that many commands simply execute, and there is no output. It is a principle amongst Linux programming that a good program executes silently, and only provides feedback if requested, or if something has gone wrong. More on this momentarily.
List the Directories within the Current Folder
We can list other directories within our current folder using the ls command. Type ls into your Bash window and hit enter:
Use Bash to list the contents of the current directory:
Huh. That’s a lot of stuff, and except for those items with file extensions, it is hard to tell what is a directory (folder) and what is a file of some sort. Wouldn’t it be nice to show just the folders, without the other stuff?
In a clear case of stating the obvious, the following will clear your Bash window. Type this and hit enter:
Clear the Bash window:
$ clear
Now, let’s try to examine the folders in our current directory again, and see if we can get a more useful view of the subfolders we are working with. Type the following command into the Bash window, and hit enter again:
Print a listing of all directories within the current directory:
$ ls -d */
The result should look like this:
That’s a little more like it. Ok, so now we can see what folders exist within our current directory.
For a cleaner listing of the variations for the ls command, see the Show Directory Contents section of my Basic Git Command Line Reference Post.
Create a New Directory in the Current Directory
Next, let’s create a new folder. The command for creating a new directory is as follows:
Create a new directory within the current directory:
$ mkdir NewFolderName
Type that into your Bash window, substituting MyNewFolder for NewFolderName and hit enter. This is what you should see when you are done:
But hey - nothing happened, right? Wrong.
As mentioned above, when the mkdir command executed properly, the result is “silence.” In other words, Bash assumes YOU know that, lacking additional feedback, everything went fine. Let’s check and see. Type your new “Show me all the folders, but only the folders” command we discussed previously, and hit enter:
Well, what do you know. There is our new folder. But wait. What if we want spaces in the name of our folder? Let’s try THAT with a another new folder. Type the following and hit enter:
Create a folder named My Other New Folder:
$ mkdir My Other New Folder
When you hit enter, you should see something like this:
Hey, looks like everything worked! No complaints from Bash. Let’s see, using our “Show directory contents” command again:
Uh-oh. Looks like things didn’t go quite the way we expected. As it turns out, we confused Bash, because the actual syntax of the mkdir command is:
The full syntax for the mkdir command:
$ mkdir Folder_1 Folder_2 . . . Folder_n
So when we typed in a folder name containing spaces, Bash thought we were creating a separate new folder for each word. As it turns out, most of the time when using Bash, we need to enclose text with spaces in quotes, so that Bash will know that the quoted text represents a single object or idea. Let’s try creating our new folder with spaces in the name again. This time, type the following into the Bash window:
Create a new folder with spaces in the name:
$ mkdir "My Other New Folder"
When you are done, you should see something like this:
Again, we assume that nothing went wrong from a technical standpoint, because Bash executed the command and returned without complaint. If we check to see again, we find the following:
Yay! So now we can not only create folders, but with spaces in the names too, dammit!
For more on creating directories, see the Creating New Directories Section in my Basic Git Command Line Reference Post
Move to a Folder Within the Current Directory
Ok, so now let’s navigate ourselves into the first folder we created, MyNewFolder. First, though, let’s clear the bash window again, so we can get rid of all the clutter we have accumulated so far.
When we want to move to a new location using Bash, we use the cd command:
The correct syntax for this command is:
Change Directory Command Syntax:
$ cd [options] [<DirectoryName>]
In the above, the square brackets denote optional inputs, and the angle brackets denote user-supplied values. In both cases, we do not type the actual brackets into the command window.
Since Bash already knows the current directory we are in, we can specify a folder name which exists in the current directory, without the rest of the directory path. Type the following into the Bash window, and hit enter. You should see something similar to this:
Note in the above that the line above our prompt has changed, and appears to indicate our current path as:
~/MyNewFolder
Remember that Bash uses the ~ symbol to denote our default (User) directory, so this is a relative reference within the user directory. From where we are right now, there are a number of ways we could return to the user directory.
The first is to simply enter the cd command with no additional input. Doing this will return us to the default directory from wherever we happen to be in the file system:
Return to the Default Directory (“Take Me Home”)
$ cd
Another option is to type cd - (that’s the cd command followed by a space, followed by a single dash). This tells bash to go back to the previous directory. In this particular case, the previous directory also happens to be our home user folder, but it really doesn’t matter what directory we came here from, cd - returns us there.
Return to the Previous Directory (“Take Me Back”)
$ cd -
The third option is to type cd .. (that is the cd command, followed by a space, followed by two periods in a row). This command tells bash to go up one level in the folder hierarchy.
Move Up One Level (“Take Me Up”)
$ cd ..
Considering that we are currently in a folder within our home directory, you can see why in the current situation, these three are all about the same in terms of returning to the home directory. At the moment, it is not only the last place we navigated from, but is also one level above our current location. And it is, indeed the home folder. So all three options work for us in this case. Try the “Take Me Up” command. Type it into your Bash window and hit enter. You should see something like this:
Note that we now appear to back in the “directory known as ~” or, our user folder.
Now type the “Take Me Back” Command (cd -) and hit enter:
Now, here we are right back in the MyNewFolder directory. Note that Bash decided to tell us where it took us in long form:
A Linux-Style Directory Path
/c/Users/xivSolutions/MyNewFolder
But wait - what is with that funny directory syntax? That is the Linux directory style. In fact, when typing explicit directory paths into Bash, you will need to use that format, because Bash will not recognize the Windows directory style. It’s easy enough to follow though:
- Instead of using the forward-slash (\) character as a path delimiter, Bash (and Linux) uses the backslash (/)
- /c/ is the Linux way of expressing the familiar C:\
Move to a Specific Directory Using the Complete Directory Path
Knowing this, we can also use explicit paths for navigation. For example, let’s say I wish to move from the current directory to my Documents folder. I can type the following:
Move to a specific directory using the directory path:
$ cd /c/Users/CurrentUserFolder/Documents
Try that now, substituting your user folder, assuming that you have a standard Windows file structure, which includes a Documents folder in your user folder by default:
Ok. If all went well, you should now find yourself in your Documents folder. Note that even though you are within a subfolder of your user folder, Bash is now displaying the long form of the directory path, instead of the shorthand relative path.
Now try using the “Take Me Home” version of the cd command (simply type cd and hit enter):
Presto - back in our home directory.
For more on directory navigation, see the Navigating the File System Section of my Basic Git Command Line Reference Post
Remove Directories
Ok, now what about those folders we “accidentally” created when we entered a multi-part directory name? You know, the folders named “My” and “Other” and “New” and “Folder”? Remember THIS:
Since these folders are all empty, we can simply use the rmdir command:
Remove One or More Empty Directories:
$ rmdir Directory_1 Directory_2 . . . Directory_n
Type the rmdir command as follows and hit enter. When you are done, you should see something like this:
Then, if we check the directory contents again, we see that those extra directories no longer exist:
Removing an empty directory or directories is a simple undertaking. The process is a little different if the directory to be removed contains files or other directories. We’ll look at this in the next post.
Next: Creating and Working with Files using Bash
Additional Resources
Related Posts
John on Google