sglavoie.com – Tag: Terminal

Git worktrees for a better parallel workflow

Sébastien Lavoie — Mon, 02 Jan 2023 09:52:00 -0600

Introduction

Why are Git worktrees needed in the first place? Well, there are certainly a couple of different use cases where they come in handy, but the reason they exist is to allow a developer to check out multiple branches at once without having to do any kind of cleanup when switching branches. Introduced back in 2015 in Git 2.5, it’s certainly not a new feature anymore, yet it seems like its adoption really took off more recently. I was an avid user of git stash but I often got myself cornered in some specific situations where it was not as convenient as I thought it was… and git worktree was for me a pretty good solution without much overhead.

Why is `git stash` not enough?

It can be enough, but there are edge cases where it falls short of its promise of keeping stuff neatly around without too much fuss.

Switching context

One particular use case in favor of git worktree is when you have to quickly switch context and you have a dirty branch checked out – which is basically the same use case advertised in the Git documentation. Committing changes away might be a bit risky if those aren’t ready to be pushed to the remote server and stashing files might be annoying because there could be newer files to include with git stash --include-untracked or some files already staged in the middle of the work where a git stash --keep-index is also appropriate, or maybe even a case where you have created temporary files that match ignored files where git stash --all might do. It gets even messier if you want to leave things in a clean state with only the changes you need with git stash --patch where it would be needed to select interactively all your hunks. And then, you might already have multiple stashes or you might have forgotten to give one or more stash(es) a name, which makes it harder to git stash pop or git stash apply your changes later without having to inspect the content of your stashes with something like git stash show -p.

With git worktree, this is no longer an issue! If you don’t already have a new worktree you can switch to, creating one with git worktree add is quick and easy, as shown in the docs:

bash

git worktree add -b emergency-fix ../temp master

This will create a branch named emergency-fix, creating a new worktree at ../temp checking the branch emergency-fix that will be based off the master branch. Then, you would switch to that branch which is managed with a new worktree by changing directory into it (or pushd ../temp to put the directory into a stack so you can come back later to your current branch with popd). You would add changes and commit them, likely pushing them to a remote, then you could resume your work by going back to the original directory for your master branch. Now, you could leave behind the emergency-fix branch but if that was meant as a temporary one, you could just delete the worktree with git worktree remove ../temp. Besides being removed by path, worktrees can also be removed by the name they are associated with, which is the branch name shown in square brackets when issuing the command git worktree list.

Arguably, it’s not too hard to see that dealing with temporary changes in this way is a lot more straightforward since the current changes can be left intact without having to commit or stash them.

Running tasks in the background

Whether that be running a test suite that takes a while to execute or leaving the build of a system untouched while it is happening (like building a Docker image), it can be really useful to go work on something else while leaving the original task in the background. More than this, it allows one to go on another branch and make changes to the repository while files from the other branch are being accessed. In the case of building a Docker image for instance, it could cause hard-to-find bugs if the build is stopped or failed and meanwhile files were modified in the repo on the same branch to then be copied back in the next build of the Docker image only to realize that the files being copied have changed prior to being copied.

More generally though, I find I just like to leave my terminal windows open into a specific path (branch) while having the freedom to go work elsewhere. This way, a virtual environment (for instance, in Python) can be left “activated” and ready to use when context is switched again. Often, different branches in a project might have slightly different requirements and so a distinct virtual environment must be activated. Having them entirely separated by path in separate worktrees makes this kind of workflow a lot simpler to manage.

How I operate worktrees

First, other approaches in the wild

I have seen some great programmers use them within a “bare repository”, notably ThePrimeagen in his YouTube video titled Git’s Best And Most Unknown Feature. This works by creating different directories inside the bare repository, effectively having all worktrees at the same level. There is a slight chance of colliding with a reserved directory name by Git, though, since all the metadata is stored there too.

Another interesting approach I’ve witnessed is the use of a .worktrees directory inside the directory containing the main branch of the project, which works by adding a match for .worktrees/ in a global .gitignore file (kudos to Redhwan Nacef in his Git Worktree Tutorial!). I like this approach because it can list all working trees anyways with git worktree list and there’s nothing really cluttering the directory structure. However, there might be some huge build files or environments and nesting everything that way might require creating a script to find all worktrees as it is not immediately obvious where they would be stored, if at all.

What I actually do these days

The approach I have adopted is very simple and requires almost no setup at all.

Starting with a project with existing branches

Assuming a project is already cloned and worked on with no additional worktrees, then it leads to a minor hiccup because the project will need to be renamed. So what I currently do is create a top-level directory for the project, say sglavoie.github.io for this website. Then, I will literally reuse the branch name to create some hierarchy inside that project. Let’s say that right now, I’m writing multiple articles in different branches and I keep my main branch as always, then I would first mkdir sglavoie.github.io/main and put the project’s content in there with its .git folder. Mind you, this is a one-time thing when not already using worktrees!

At this stage, the project structure is laid out nicely and ready to accommodate new worktrees. So for instance, if I want to work on a new branch to create a new article without touching the main branch, I would git worktree add ../articles/article-name-here article-name-here from the main branch and voilà, a new worktree will be ready to use. Once a couple of worktrees are created in this way, the directory structure might look as follows:

 
 
 
~/dev/sglavoie/sglavoie.github.io
├── articles
│ ├── adv-web-dev-social-network-app
│ ├── git-worktree
│ ├── react-native-app
│ └── text-based-diagrams
└── main

In this way, it’s just a matter of changing directory and opening the worktree in your editor of choice (e.g., nvim for neovim or code . with VS Code). From now on, it’s easy to switch between branches of a project right from the editor by searching for the “topic” (e.g., main or articles in this case, but that could be hotfix or feat depending on how your name your branches) or for a project name directory, which would show all the branches that have been opened previously to allow further filtering. This is what git worktree list would display in this example:

 
 
 
/some/path/sglavoie.github.io/main d67dd8d2 [main]
/some/path/sglavoie.github.io/articles/adv-web-dev-social-network-app 666a0b43 [articles/adv-web-dev-social-network-app]
/some/path/sglavoie.github.io/articles/git-worktree d67dd8d2 [git-worktree]
/some/path/sglavoie.github.io/articles/react-native-app bcfac34e [articles/react-native-app]
/some/path/sglavoie.github.io/articles/text-based-diagrams 4b25371d [articles/text-based-diagrams]

New worktrees from a project already using worktrees

If you’ve already embraced this approach, then the next time you want to create a worktree, it will just be a matter of switching to the “base” branch you want to create a worktree from and creating one at whatever location you fancy outside the current directory. For instance, for this article:

bash

git worktree add ../articles/git-worktree articles/git-worktree

This would take care of using the branch name “nesting” convention (e.g., feat/feature-name, hotfix/bug-name, etc.) and create sub-directories as needed too. Assuming a worktree has been merged in another branch and/or is no longer necessary, it can be removed with git worktree remove git-worktree in this example (where git-worktree is the name given to the worktree… a bit confusing here, you’re right).

Keeping worktrees around

Although the official Git documentation presents a useful case for a temporary fix where the worktree is almost immediately deleted upon the completion of a task, I have found myself in a situation where I’d rather have multiple longstanding worktrees. For example, I regularly need to boot a web application from a different branch, either because the currently checked out branch does not have the necessary requirements installed or because I want to leave a web application running in the background without affecting my current work. In my case, I use it to process some data when it arrives and come back to the work I was doing earlier, doing practically the same thing shown in the Git docs but keeping the worktree around for future use.

On a large project, I might have a couple of worktrees, but still I would keep branches around without them being part of worktrees when these are meant to be short-lived feature branches that share the same dependencies as other branches as long as they won’t need to be checked out at the same time. Another time I might have separate branches like this could be to avoid having to duplicate gigantic node_modules or .venv directories when not needed, because working with different worktrees is pretty much like cloning an entire repository in a separate folder and having to reinstall requirements. There is also a nice Git feature where worktrees are prefixed with a plus sign when issuing git branch, so that makes them distinctly different from regular branches:

 bash
 
 
 
$ git branch
+ articles/react-native-app # worktree
 gh-pages # other regular branch
+ git-worktree # worktree
* main # currently checked out

Conclusion

I have briefly touched upon a downside of worktrees, which is that it’s almost like cloning over and over a repository, which can take up a lot of disk space with large projects. Nevertheless, this is a tradeoff I am entirely willing to make given the advantages Git worktrees bring with them! If you’re not already using them, maybe this little post might have inspired you to give them a go!

Resources and references

External links

From this website

A retrospective on creating an impractical little tool just for fun

Sébastien Lavoie — Sun, 18 Sep 2022 20:06:00 -0500

Introduction

Building little projects to scratch one’s own needs is a great way to practice the craft of software development, or at least this is some consistent advice I have picked up from legendary figures in the field. Whether it be formulated by the famous John Carmack (an excellent interview with Lex Fridman was recently released) or a rising star like The Primeagen (he has a fantastic YouTube channel too), the core of the message remains the same: deliberate practice is required to make progress.

While I was in the middle of completing a BSc in computer science, I decided that a simple and reliable spreadsheet was not enough to keep track of my grades and decided to tackle the creation of a TUI – terminal user interface – to come to my rescue by providing, hopefully, correct answers. And so uol-grades-calculator (ugc for short) was born out of the desire to learn more about TDD in a practical way, the Hypothesis testing library written in Python, Sphinx and Read The Docs for publishing nice-looking documentation in reStructuredText format (the Markdown format being another option). Even though the goal of this tool has never been to gather a growing user base, it was a good learning experience because alternatives to it spawn up over time and, arguably, some of them were much friendlier and simpler to use (who would have thought that…).

Setting up the stage

To put things in perspective, this tool was never really under active development, but it did receive updates from August 2020 until September 2022, so the project slowly took shape in a time frame of a little over 2 years as can be shown in the following repurposed Gantt chart, where the length associated with each Git commit corresponds to the amount of time elapsed from one commit to the next.

Mermaid source of this diagram

Vim magic behind the scenes

While the chart itself is neither the prettiest nor the most interesting, the how of it has a possibly more entertaining story attached to it:

The list of commits was extracted with the command git log --pretty=format:'%s : %cs' > commits.txt to get a one-liner for each commit with the description followed by the date, piped/stored in the text file commits.txt.
Some Vim regex magic was done to substitute the first ‘:’ character on each line so that Mermaid wouldn’t struggle to render the graph: ‘%s/^$\w\+$: /\1 - /’. This pattern was actually predictable because the Conventional Commits specification was followed religiously right from the beginning.
The previous command left some ‘:’ characters in there that weren’t needed, so they were removed except where a digit was immediately following such that the last part of the line including the date would be left in place (here, \v is the “very magic” flag used so that we don’t need backslashes around the captured group, i.e. the digit, which looks a bit more readable): ‘%s/\v: (\d)@!/ -’.
To show the elapsed time between two commits, a Vim macro was concocted so that the date on the current line would be copied and pasted on the following line in a specific way, which works because the Git history appears in chronological order by default: ‘kf:wv9ly0jA,^[pj0’. It’s Ugly but it gets the job done ;).
- In essence, it goes up by one line (k), searches for the : character with f (which precedes the date), moves ahead by one word, highlights the date which is in the format YYYY-MM-DD with v9l, yanks/copies the date with y, goes back to the beginning of the line (0), goes down one line with j, Appends a , at the end of the line, goes out of insert mode with a Ctrl - c shortcut (which visually translates to the escape sequence ^[), pastes the date, goes down one line again with j to be ready to process the next date and goes to the beginning of the line again with 0 so we don’t miss a match. Vim macros are definitely quite expressive in a terse way!
- Then, it’s just a matter of running the macro for the number of lines below the cursor in the file. We can quickly get a sense of the number of lines in the file with Ctrl - g and run the macro on the required number of lines with 100@a, replacing 100 with the number of lines and a with the register where the macro was saved (in this case, in the register a with qa to start recording the macro there, which is to be stopped by pressing q once more in normal mode).
Once the macro is run, the output needs to be reversed so that the initial commit can be displayed where the graph starts in the top-left corner, which can be done with the clever command ‘g/^/m0’: it runs on every line of the file, successively putting each line at the very top of the file (line 0) to have the effect of reversing the whole document!
Finally, the graph is generated literally by copying and pasting that list of commits into a Mermaid textual chart, which shall be the topic of another post!

The whole process is done in a few minutes at most, making it a nice solution to deal with manageable outputs under a few thousand lines. Otherwise, it might be more convenient – albeit a bit more time consuming too – to write a script to do the work since Vim macros, when run at a relatively large scale, can be slow and error-prone.

What went well

With all that fluffy preamble, we might hope that something went well in the end. Well, there were a couple of noteworthy things indeed:

Even though I worked on this project alone, I created a bunch of GitHub issues that I assigned to myself, each with its own set of tasks, acceptance criteria, a short analysis section as required, linked pull requests and referenced commits. It is more work and arguably could be done more simply in a notes application, but it feels good to be publicly accountable for one’s work and that also encouraged (very few) people to submit their own issues. Additionally, managing tasks on a platform like GitHub makes it easy to review the state of a project and to manage a backlog of tasks, which was a nice plus when the project was linked to what was meant to be its successor, uol-grades-calculator-server, a backend using ugc as its API which was to be hooked to a React frontend to display the data (more on that in the list of things that went poorly).
The TDD approach was embraced right from the start and I consider that being one of the highlights of the project in hindsight. With over 250 tests (some of which are randomly generated to do property-based testing with Hypothesis), I felt confident it was giving somewhat sensible answers.
Best practices were adopted: Pylint, the king of complainers amongst the Python linters, was very verbose but set up to be slightly more quiet after a while. A pyproject.toml file was included, a clean pytest config file was used to automate the input of some parameters when running the test suite, the project was developed with the intention of being deployed as a package, only popular and stable libraries were used and just as importantly, only portable formats like JSON made it to the final version of the project.
Sub-modules were created to make everything a bit more… modular.
A GitHub workflow was added to automatically deploy new releases to PyPI.
Documentation was generated and hosted on Read The Docs, with clean outputs and images and an entire section dedicated to developers (i.e., myself :)) so that reproducibility wouldn’t become an issue for any step taken along the way.
Using pre-commit helped with catching some annoyingly formatted code before reaching production by taking advantage of Git pre-commit hooks.

With what seems like a glowing review so far, needless to say, a whole lot went wrong too.

Illustrating the summarize command.

What could have been done better

Independently of the expectations with regards to the user base, it would have been beneficial to do more research and planning initially before writing a single line of code. After working on other projects, I’ve found that what works well for me is to follow a specific set of steps, broadly speaking:
- Gather a list of requirements as precisely as possible to validate assumptions and define the correct scope for the work to be done (YAGNI!);
- Produce some early flowcharts to make sense of how the application might be structured and iterate on this until a clear winner emerges;
- Produce low-fidelity wireframes (even for a TUI, that would be informative before committing to doing the wrong thing!) – even if this is just quickly sketched on paper;
- Create high-fidely mockups (or at least reasonable-looking ones for a small project like this one) once things have been thought through carefully;
- Design a general plan of work before writing any code to have a better idea of how the different parts of the system should communicate together;
- Research existing tools to be leveraged so as to not re-invent the wheel all the time and see how they fit in together;
- Go with TDD and stick to it piously for anything of relative importance that should be tested;
- Go at it with atomic commits so that rewinding errors on the path will be a breeze – at least that ought to help significantly.
Related to the first point, issues could have been more substantial instead of having to often deal with useless refactoring due to not thinking things through fully.
With more mental processing done upfront, it would have been clear which features should have been part of the tool earlier and which ones could have waited longer. Glancing at the Gantt chart above, one can see that many commits happened at the beginning just to get the structure right, the project was not documented until much later and basic functionality such as being able to plot results (which was really a core feature to be expected) was not integrated until past the mid-life of the project.
On top of the above, it was clear that using YAML in the presence of the other technologies used for a backend and frontend to the CLI made things more complicated and it wasn’t until one of the latest versions that JSON was used instead.
The user experience could have been improved drastically at the beginning by relying on well-known libraries such as Rich to make the UI much more appealing and even a bit more interactive.
Some features were presented in a way that did not make much sense. For instance, the Dockerfile does allow one to use the tool from a Docker container, but not all functionality works out of the box that way (e.g., saving a plot to the filesystem or loading a configuration file).
It wasn’t productive at all to start scattering the few resources put into this project by creating a sister repository uol-grades-calculator-server. The CLI should have been more feature-rich before considering such an endeavor and it should have been made in a way that’s easier for a backend to consume.

Showing the image generated by running the plot modules (sub-)command.

Key lessons learnt and where to take it from here

Studying better the available libraries would have avoided some headaches. For instance, the display of what should have been tabular data started out as plain JSON (printing a dict object), then beautifultable was used to create some colored output, then pandas replaced the previous library because some limitations were found with it – reverting to a plain table output in the process – and finally rich was used to create a more decent-looking output.
Likewise, spending more time in the planning department before getting started would probably have resulted in a tool that’s more straightforward to use, focused on the most needed features and possessing a broader feature set by not having invested extra resources on building a backend and a frontend separately. It should have been either a more powerful TUI limited to be used within a terminal or a web application, but not both in the way it was designed.
Work on small, isolated parts of the system. Some issues were split into large chunks of work (this one for instance, implementing at once the plot command and its sub-commands). This wasn’t a problem per se given that I worked alone, but it would be harder for someone else to follow and to create a reasonably-sized pull request from such an issue. This feature could have been broken down into different sets of related flags.

A simple component diagram demonstrating how the CLI was to be used in a greater context.

Conclusion

“Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.”

--Norm Kerth, Project Retrospectives: A Handbook for Team Review

All in all, this has been a fun experiment to bring to fruition. It would have been much quicker to use a spreadsheet given what the final result ended up being, but the experience obtained was worth every drop of developer sweat! Using Python felt appropriate as iterations of the tool could be produced fast and the CLI was responsive enough to be useful and pleasant to interact with thanks to its intuitive commands. In the near future, using a library like Textual could be more rewarding as one could expect a much nicer visual experience in the terminal at a fraction of the current implementation efforts.

Resources and references

Git the gist of it: common commands for a working workflow

Sébastien Lavoie — Sun, 03 Apr 2022 11:21:00 -0500

Introduction

Git, the friend you wish you had met earlier, when organizing files like report 1.pdf, report 2.pdf and report final 1.pdf still made sense somehow. It’s undoubtedly complicated to wrap one’s head around it at first, although there exist some GUIs to make the journey smoother – the best examples that come to mind would include GitKraken and Tower at the top of the list. I have been advised to learn to use Git from the terminal and I would, without flinching, recommend anyone to do the same to really grok this tool and get an intuitive feel for what it does.

Beyond the extraordinary technical prowess that it is, Git can be useful to a broad audience. It’s not good at dealing with large binary files (and Git LFS is not exactly easy to use just yet to cover that case), but for anything involving text that a human can read, it shines – may it be a programming language, some reports written with LaTeX, some diagrams produced with Mermaid or PlantUML, personal notes taken in plain text or in Markdown or the fact that synchronizing changes made to a project is a breeze and allows ones to work from multiple locations and machines.

With platforms like GitHub and GitLab, it is easy to keep a “backup” of projects, which can be either public or private. And despite Git not being awesome with very large files, any smaller binary files like images and PDF files can easily be added to a Git repository, allowing users to keep different versions of these files very easily without using a cloud service such as Dropbox or Google Drive. So even if the original input is not written in a text-friendly manner, the output can still be saved under multiple versions with great details, backed up to different websites and fetched just as conveniently from anywhere else (assuming access has been set up to continue working on a project, or at least one can log in to the website and download files without issue).

Sure, it is not for everyone, but the use cases just described can go a long way. I have used it to practice solving programming challenges, to document all my project at the university, to keep track of preferences and configuration files for my different systems (Linux and macOS) which I’ve described in Managing dotfiles with a Git bare repository, to work on pet programming projects and coding experiments that I keep private, to store code snippets into a growing library for reference purposes, to take notes in the form of “tutorials” or “wiki” (platforms like GitHub and GitLab have a nice wiki feature that renders prose nicely!) for different tasks I need to do infrequently and of course to build this very website in the open (the source code is available on GitHub)! And that is just describing some of my personal use cases without touching on the fact that collaboration within a team is a huge reason to use it!

So with that said, let’s explore some of its functionality, starting with a few tips and tricks, then delving deeper into a real-world workflow and topping it off with more useful commands and configuration settings.

Global and local `.gitignore`

There are files we never care about adding to Git repositories, such as a thumbnail cache file Thumbs.db on Windows or a .DS_Store file storing custom attributes for folders on macOS. Instead of ignoring these kinds of entries in a per-repository .gitignore file which might not contain everything we would like to exclude, it can be simpler and more efficient to use a global .gitignore, which serves the same purpose but for any Git repository regardless of the presence of a .gitignore in that specific repository! In other words, this serves as a permanent list of things to ignore so that there is no need to remember about them later.

The setup is straightforward: create a file ~/.gitignore, fill it as usual with patterns you are sure to want to exclude globally and set a configuration option to use that file – for instance by adding the following to .gitconfig:

git

[core]
 excludesfile = ~/.gitignore

Toptal provides a nice tool on the command-line that can be used to easily exclude patterns. With it, typing gi python >> .gitignore would append a bunch of common patterns to a .gitignore file – in this case, for Python.

Meaningful commit messages with `.gitmessage`

Besides using the excellent Conventional Commits specification to write messages in a format that makes sense to other fellow human beings, using a template for Git commits can help with remembering why commits are made and how they should be structured. There is a configuration setting, commit.template, that can be used to set the default text being displayed when a text editor opens after typing git commit.

The following is the current template I use, which can be saved in a file like ~/.gitmessage:

 
 
 
# [Add/Fix/Remove/Update/Refactor/Document] [summary]


# Why is it necessary? (Bug fix, feature, improvements?)
#-
# How does the change address the issue?
#-
# What side effects does this change have?
#-
# Include a link to the ticket, if any.

Then, this can be added to the config file under the [commit] section like so:

git

[commit]
 template = ~/.gitmessage

All lines starting with a hash symbol (#) will be ignored, so there is no need to manually delete all of this verbose content when saving the commit message. Sweet!

My usual Git workflow

This section depends very much on team standards, if applicable. But in general terms and considering mostly a solo interaction with Git, I like to adhere to the following processes, without taking into account how one would plan, produce diagrams, brainstorm, etc. That shall be the topic of an upcoming post!

Starting a new project

There are different ways to proceed, although I like the simplicity of just creating a new empty repository on GitHub and cloning it on my machine using the SSH protocol, e.g. git clone git@github.com:some/project.git. This has the benefit of automatically setting the main branch and the remote URL, using origin as the default remote. Cloning an empty repository has the same effect as starting with git init, so this also becomes unnecessary.

If I have been experimenting locally first, I would do a git init, commit any changes and then push the new repository on GitHub, either using the great Git integration provided by VS Code or again creating a new repository on GitHub to set the remote URL in order to be able to push local changes, i.e. by doing git remote add origin git@github.com:some/project.git and then pushing to that new URL. To make sure that my default branch is always main (to match the new default on GitHub) and not something like dev or master, I have a Git template lying around that ensures that the first branch created when doing git init will be main. It is as simple as creating the file ~/.git-templates/HEAD with this content:

ref: refs/heads/main

Then, in ~/.gitconfig, the following section is used to read the templates:

git

[init]
 defaultBranch = main
 templateDir = ~/.git-templates

I have this template since before init.defaultBranch was introduced in Git 2.28.0 back in July 2020: now just having the option defaultBranch will do the trick. Still, it’s good to have a templateDir configured for other purposes! Although admittedly, I don’t use them at all and would rather create a custom command to more easily script what I need for each repository: more on this in the section Custom Git sub-commands.

Committing to an existing project

Now that there is some folder set up for Git to track files, it’s time to make changes! I’m a big fan of committing “small and often” because Git is super flexible and updating the history is usually a simple process – unless you have pushed to a remote repository used by other people, in which case being a “force push” type of person won’t make you many friends. May the --force be with you when you push to a private repository where you’re the only contributor, otherwise, it’s good practice to think twice before sending your final changes away as you should not mess up with public history (and depending on how your access is set up within a team, you might not even be able to use the --force flag when pushing on certain branches!).

So what’s the alternative to constantly rewriting history in a harmful way? Well, assuming you have a choice when working alone for instance, there are two main contenders: merge vs rebase. I think neither option is better than the other when used right, but it’s good to know that rebase might create possible headaches because it modifies the Git history while merge might create undesirable noise when used profusely by adding “merge commits”. I use both techniques, but for different purposes.

Being committed

The basic workflow might look like this, with a couple of different options thrown in to cover more scenarios:

 bash
 
 
 
# Check the current state of the repo
git status

# Inspect the changes to commit (before staging)
git diff

# Stage a file to be committed
git add SOME_FILE

# Stage all new/modified files, (i.e., --all)
git add -A

# Stage all modified files (not any new ones, i.e., --update)
git add -u

# Get fancy with adding patches only (part of files)
git add -p

# Undo changes done to a tracked file
git restore -- FILE_NAME

# Inspect staged changes to commit
git diff --staged

# Unstage files
git reset HEAD --

# or just 'git commit' to open a text editor
git commit -m "SOME_MESSAGE"

# See the changes done in the latest commit
git show

# or a specific commit hash from earlier
git show COMMIT_HASH # or ref like HEAD^

# Forgot to add something?
git add SOME_OTHER_FILE
git commit --amend --no-edit

# Want to rephrase the last commit?
git commit --amend

# Tag the latest commit
git tag TAG_NAME

# Tag earlier commit with annotation
git tag -a TAG_NAME COMMIT_HASH -m "MESSAGE"

# Undo a commit, keeping changes in the work tree
git reset HEAD^

# Undo a commit and DO NOT keep changes
git reset --hard HEAD^

# Oops, get back a commit that was "lost"
git reflog # find the relevant commit hash
git cherry-pick COMMIT_HASH # or git checkout COMMIT_HASH

# Explore the repo at an earlier commit
git checkout COMMIT_HASH

# Committed some changes while checking out an old commit?
# Save the changes to a new branch, then potentially merge them back
git switch -c NEW_BRANCH_NAME
git switch OTHER_BRANCH
git merge NEW_BRANCH_NAME

# Undo a merge
git reflog # find the commit hash before the merge
git reset --hard COMMIT_HASH

# Or undo to a known point from a remote branch
git reset --hard origin/main

# Stop tracking a file/directory (remove) but keep it on disk
git rm --cached FILE_OR_DIRECTORY

# Remove stuff that hasn't been committed yet (interactive mode is nice)
# Don't forget about -n for a dry run first!
git clean -i

# Want to undo a change that's already pushed?
# Add a new commit without rewriting history
git revert COMMIT_HASH

# See who did what to which file and when
git blame FILE_NAME

Arguably, basic editing doesn’t require stashing, but that can be helpful for a quick modification, like temporarily saving changes to quickly switch to another branch or pull some remote changes without having to commit first.

 bash
 
 
 
# git stash --help - "Stash the changes in a dirty working directory away"

# Maybe you want to keep changes around without committing
# --include-untracked and --keep-index might be needed!
git stash

# Stash changes with a clearer name/purpose
git stash save "SOME MESSAGE"

# Selectively stash changes (keep specific lines)
git stash save -p

# List existing stashes
git stash list

# See the summary of a stash
git stash show

# See all changes made in a stash (patch)
git stash show -p

# Apply changes from a stash (TAB completion after `apply` is useful)
git stash apply stash@\{SOME_ID\}

# Apply and remove the last stash at the same time
git stash pop

# Remove a single stash
git stash drop stash@\{SOME_ID\}

# Remove all stashes
git stash clear

`rebase`

When working locally with changes that are not yet part of the “permanent” Git history, I like to git rebase very much to clean things up before making a git push, which I like to view as an irrevocable decision. Because I commit small and often, it’s often the case that I end up with a couple of very simple commits which really belong together and should form a bigger commit. While committing a big chunk of code at once avoids having to do any rebase in the first place, it’s also inconvenient to undo a substantial amount of work when you’re experimenting and/or working on an unfamiliar codebase. So in the end, my preference goes towards smaller commits because rebasing is actually fun to do and lets you believe that you came up with brilliant solutions on your first attempt :). That makes for a cleaner Git history for sure, although you will lose some context if you never merge. Meaning, rebasing creates a “linear” history while merging from different branches shows a more complete picture of how a repository really took shape over time. My simple rule of thumb on the matter is this: use rebase to consistently ship meaningful commits (good descriptions, fewer typos, changes chunked logically and precisely, etc.) and use merge to incorporate somewhat larger pieces of history, like a new feature coming from a “feature branch”.

Some common commands I use:

 bash
 
 
 
# git rebase --help - "Reapply commits on top of another base tip"

# Include all commits up to COMMIT_HASH, included
git rebase -i COMMIT_HASH^

# Include most recent commit only
git rebase -i HEAD^

# Include 2 most recent commits
git rebase -i HEAD^^

# Include 3 most recent commits
git rebase -i HEAD~3

# make BRANCH have the history up to COMMIT_OR_BRANCH, included
git rebase COMMIT_OR_BRANCH BRANCH

# Used after resolving a conflict
git rebase --continue

# Reset to where you were before rebasing (move HEAD)
git rebase --abort

# To update the next actions to be taken on the remaining commits
git rebase --edit-todo

There are a bunch of obscure options just like almost any other Git command in existence, but for that there is help available for each command (such as git rebase --help) from the terminal and the trusty Stack Overflow ;).

Sample workflow

In the case of rebase, I enjoy working in this way:

Make small commits, maybe 5 to 10 depending on the complexity of the project – usually as long as a series of commits relates to a single topic whenever possible.
Have a peak at the recent history with git log to have a good idea of what just happened.
Realize that the history can be improved. Find the oldest commit hash (not pushed yet!), take note of it and run git rebase -i COMMIT_HASH^ (with the “hat” ^ at the end to reference the parent of that commit).
This opens up a text editor with some explanations on how to proceed (for more on this, have a look at this great article on Rewriting history). From there, I usually use r to reword, s to squash, e to edit and sometimes f to fixup, besides also re-ordering commits and from time to time deleting one by removing the line entirely.
- r is pretty harmless as it will just show up a new window where the commit message and description can be modified.
- s is more destructive because commits will be “merged” together! No actual change is lost, but if you squash a ton of commits by accident, it will create huge commits and you might want to undo that with reflog before it’s too late (Atlassian has more to say about this in git reflog). The s option is also nice because it keeps all commit messages, so you can edit the final log message as needed without discarding messages.
- e is used to modify the actual source code being committed, maybe to fix a typo or a minor bug after realizing that the test suite no longer passes (assuming you are not programming blindly without tests :)). This can cause some “conflicts” if you change some lines which are also part of other commits in the selection you made originally, so one has to be careful with that. This can also be used in conjunction with git reset HEAD^ to “cancel” the current commit when doing the rebase in order to split it into multiple smaller commits, which can then be “git added” back, followed by a git rebase --continue to keep going.
- f is a quick way of squashing commits together while discarding messages you don’t need. So maybe the last commit has a proper description (“Implemented something…”) and a few commits before that one were just temporary “savepoints” with non-sensical commit messages (things like “testing…” or “new commit”) that were done until you reached a final solution that’s worth committing. So of course in that case the history would be pretty ugly and hard to understand if all commits were kept intact, so these can be added to the latest change which encompasses a working solution as a whole. Just as well, maybe that last commit didn’t have a complete message just yet, so it could also be rewritten with the r prefix.
When individual commits make the rebase stop temporarily (such as when a commit is to be edited with e and waits for changes to be made), then it’s time to do git rebase --continue once the necessary changes are made.
If something goes wrong along the way, there’s always a way back to the place you were right before rebasing with git rebase --abort. It’s worth noting that Git itself will output some useful messages all along so it’s not necessary to remember all of this: it quickly becomes intuitive.

Now, rebase covers the cases where local changes are done in isolation, but for something more imposing or simply to keep track of what happened in which branch, merge remains a powerful ally.

`merge`

So I don’t use merge as much, especially when working solo, but it is undeniably useful and could replace the rebase workflow completely (it might also be demanded by your colleagues anyways!). There’s nothing like a good rebase session to keep things tidy, but a flat history line won’t look attractive on a huge project with many contributors, so merge is there to deal with cleanly integrating changes from different places into some main branch (it could be named differently based on the team workflow of course and there might be more than one “main” branch too). What makes merge incredibly cool is the fact that branches in Git are “cheap”, meaning they can be created very quickly and efficiently, they don’t take space at all, you could have dozens of branches or more and you can incorporate changes from one branch to another with a simple git merge, which is nicer than using git cherry-pick to get multiple commits from one branch into another (for one or two commits though, cherry-pick is handy).

I see rebase more like a cleaning step while merge really shines when dealing with multiple commits between different branches. Even on a simple project when working alone, it has its uses! For example, let’s say we have a repository full of small programming challenges (maybe from HackerRank or LeetCode). We might get stuck on a tricky problem and wouldn’t want to commit an unfinished solution. Also, maybe we would get fed up eventually working on a given challenge and would rather try our luck with a different one. Well, we can always leave changes uncommitted (not very safe as these might be lost!), but that can cause confusion and you might inadvertently end up committing files that have nothing to do with the latest challenge – effectively committing at the same time your unfinished challenge with the one you just completed. There’s always the possibility to git stash individual files to keep them somewhere, but I find working with stashes to be a bit more cumbersome than branches and you might literally forget that you have changes stashed. If you do that a few times, you might not even know which stash is what if you didn’t give them a good name…

Some common commands I use:

 bash
 
 
 
# git merge --help - "Join two or more development histories together"

# Get commits from BRANCH_NAME into the currently checked one
git merge BRANCH_NAME

# Merge one branch into another
git merge SOURCE_BRANCH_NAME DESTINATION_BRANCH_NAME

# Get out of conflicts...
git merge --abort

# Continue after resolving conflicts
git merge --continue

# Keep the "merge commit" in the history
git merge --no-ff BRANCH_NAME

# Set a custom commit message for a "merge commit"
git merge -m "MESSAGE" BRANCH

_{Tip: The command git merge --help describes some nice examples about the whole process of conflict resolution.}

Sample workflow (kinda)

So, instead of risking losing changes like this, creating a new branch with git switch -c BRANCH_NAME and working from there is much easier to confine changes to a specific matter, which we can see like a “feature” that can later be added to the main “trunk”. We can commit unfinished business, switch to a different branch, come back to it later, git reset any old commit to “revert” the history and keep working from a clean state. In any case, there’s always rebase to avoid committing useless stuff, so there’s really no reason to avoid branches.

The merge command will be involved when changes from a “feature branch” (or whatever other purpose you give a branch) are merged into the main branch with git merge BRANCH_NAME. When there are no “merge conflicts” (such as when you work on a totally new file that won’t clash with prior work), this is a seamless workflow where you can just merge as if changes were made in the branch you are on in the first place. There are other instances where bigger changes can be merged successfully in the same way, but it’s interesting to keep track of how you worked on a project, so in this scenario you can always use git merge --no-ff BRANCH_NAME (--no-ff meaning “no fast-forward”) to keep a “merge commit”, which is strictly speaking not needed (because there’s no conflict resolution in the case of a “successful” merge) but can be a helpful reference point when you look back at your Git history two years from now to understand how you worked on something, piecing things together bit by bit.

There’s also an indisputable advantage conferred by the use of branches, which is: you can have plenty of them! So while rebase is nice to use on a given branch, in no way can working on multiple things at once in a single branch and then rebasing all that effort be as straightforward as creating new branches for every new topic that’s being started and then merging it back into a single place of truth. Juggling different concerns in a single branch where you have a few things going on (maybe 2-3 partially implemented features) is far from being a cozy environment and rebasing that kind of work where commits are not even related to each other can become a nightmare in a short amount of time.

Managing branches

So with all that was written earlier, I believe a point was made in favour of using branches. Because branches are such a wonderful feature of Git, here are some more commands I rely on (surprisingly, there aren’t that many for git branch!):

 bash
 
 
 
# git branch --help - "List, create, or delete branches"

# List local branches
git branch

# List only remote branches
git branch -r

# List all branches
git branch -a

# Delete a branch
git branch -d BRANCH_NAME

# Delete a branch forcefully
git branch -D BRANCH_NAME

# Delete a remote branch (doesn't matter if the local branch was deleted)
git push origin --delete REMOTE_NAME/BRANCH_NAME

# Rename a branch
git branch -m BRANCH_NAME NEW_BRANCH_NAME

# Track a specific remote branch
git branch BRANCH_NAME -u UPSTREAM_NAME

# See which branches have been merged or not
git branch --merged
git branch --no-merged

# Switch to the branch last checked out
git checkout -

# Compare changes between branches
git diff BRANCH_ONE..BRANCH_TWO

# Compare a file across branches
git diff BRANCH_ONE:FILE_NAME BRANCH_TWO:FILE_NAME

# See a file from another branch
git show BRANCH_NAME:FILE_NAME

# Get a file from a different branch
git restore --source OTHER_BRANCH_NAME path/to/file.txt

Aliases

While I believe it’s good practice to type complete commands to remember them better, I also tend to use Git aliases for two reasons: because some deeply ingrained concepts don’t need as much reinforcement and because some obscure operations can be made so much more accessible with a pertinent mnemonic device.

Some commands like add just become unforgettable over time, while others – e.g., unstage – just don’t exist yet. Without further ado, here is a list of the aliases I currently rely on when using Git, which comes straight from ~/.gitconfig.

For instance, instead of typing git add, by setting the alias a = add (see below), it is now possible to type git a to get the same result.

git

[alias]
 a = add
 br = branch
 c = commit
 ca = commit --amend
 can = commit --amend --no-edit
 ch = checkout
 d = diff
 ds = diff --staged
 po = push origin
 pos = push --set-upstream origin main
 rc = rebase --continue
 ri = rebase -i
 rl = reflog
 sh = show
 s = status --short
 st = status
 sw = switch
 t = tag
 unstage = reset HEAD --

 ; show changes since last commit
 difflast = diff --cached HEAD^

 ; show last commit
 last = l -1 HEAD

 ; undo last commit
 undo = "!f() { git reset --hard $(git rev-parse \
--abbrev-ref HEAD)@{${1-1}}; }; f"

 l = log

 ; short log
 sl = shortlog

 ; log graph
 lg = l --color --graph --pretty=format:'%Cred%h%Creset \
-%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit

 ; log name
 ln = l --name-only

 ; log decorate all
 logda = l --oneline --decorate --graph --all

 ; log stat
 ls = l --stat

 ; log pretty
 lp = log --pretty='%C(yellow)%h %C(cyan)%ad %Cblue%aN%C(auto)%d \
%Creset%s' --date=relative --date-order --graph

At some point, using Git aliases will become second nature, but until then, it can be useful to have a separate terminal alias to list them all (or maybe you forgot the command an alias is using under the hood!). For this, you can set the following alias in a file like ~/.bash_aliases:

 bash
 
alias gitaliases='git config -l | grep alias | sed "s/^alias\.//g"'

# For a fancier version displaying aliases in two columns neatly aligned:
alias gitaliases='git config -l | grep alias | sed "s/^alias\.//g" \
 | sed "s/=/Ω/" | column -t -s "Ω"'

That command will look in the Git configuration, extract all the lines containing “alias” and remove the “alias.” prefix so that you can see all aliases in the form a=add in the case of git add being aliased to git a. If you have a ton of aliases (which is probably a red flag…), finding a specific one is just a matter of doing gitaliases | grep keyword, of course replacing keyword with something else ;).

_{Tip: One can go further to shorten Git commands by shortening git itself! You can put a line such as alias g='git' in ~/.bash_aliases for instance. Now, git add can become g a. Pretty efficient! For more on Bash aliases, you might like to read Aliases: Also Known as Terminal User’s Best Friends.}

This section is really just meant to give a taste of the possibilities that aliases offer. To learn about actual commands and their usage, there is nothing like the official Git docs (there is a short section on Git aliases too).

Custom Git sub-commands

There might be less of a need for custom-made commands to be created when so much can be done with Git alone, but there are cases where these custom additions can shine! As a concrete example, I have been updating this website with what I call my “learning progress” but committing always the same kind of stuff gets boring, so I have created a simple Bash script to automate the process of publishing and committing (the content of the script is of no importance, it is only there to demonstrate that commands other than Git can be called too):

 bash
 
 
 
#!/bin/bash
# Act as custom command `git addlearning` and perform all that follows
# Assumes that `pelican` command is available
cd ~/dev/sglavoie/sglavoie.github.io-source
git add .
git commit -m "Add learning progress"
git push origin main
pelican

cp -r \
 ~/dev/sglavoie/sglavoie.github.io-source/output/* \
 ~/dev/sglavoie/sglavoie.github.io/ && \
 cd ~/dev/sglavoie/sglavoie.github.io && git add .
git commit -m "Add learning progress"
git push origin main
cd ~/dev/sglavoie/sglavoie.github.io-source

It is noteworthy to know that any kind of executable programs can be used, such as Ruby, Python and so on, as long as they are made available in the $PATH. This will be somewhat similar on most UNIX-based system where the PATH variable needs to be exported (usually in a file like .bashrc or .zshrc), for instance:

 bash
 
# $HOME refers to a place like /home/user on Linux or /Users/user on macOS
export PATH="$HOME/dev/git-scripts:$PATH"

On Windows, there are separate instructions to be followed. With that done, there’s also a need to make sure the file is executable, which can be done on the command-line with chmod +x FILENAME_HERE.

Some lesser-known yet incredibly useful commands

`git bisect`

I try to commit working code as often as possible, but sometimes there are just bugs that have flown by and took residence in the codebase a long time ago. In this scenario, there is one tool in the Git tool belt for that very specific use case: git bisect. Working with the binary search algorithm, git bisect searches in a range of commits and by identifying “good” and “bad” commits, will pinpoint exactly where the issue came from in the minimum number of iterations possible. You’ll need sub-commands like start, bad, good and reset at the very least and for more advanced use cases where automatically running a script for each commit is needed, run will do the job. See the docs on git bisect for more.

`git rerere`

I almost never use this in practice, but it can be very useful when working in a large project where similar merge conflicts are likely to occur over time. Git can be “taught” to automatically solve merge conflicts! There are some official docs on this, but I’ve found this Stack Overflow answer to be a more approachable read.

A word on the configuration file

At the very minimum, setting a name and email will be necessary to use Git:

 bash
 
git config --global user.name "Your Name"
git config --global user.email "email@example.com"

Seeing the current configuration can come in handy, too:

bash

git config --global --list

And in the case of this particular command, git config, it can’t be emphasized enough how useful searching through the help page in the terminal will be: git config --help.

I’d be remiss if I didn’t include an example of my own configuration at this point, so here it goes! To make it so that it is convenient to commit this configuration file publicly, some private settings can be “included” from other paths inside the [include] block. For clarity, I’ll show what that file looks like below the main configuration file.

 bash
 
 
 
# This is the content of ~/.gitconfig

[include]
 ; This file can add more information to extend the main configuration.
 ; For example, when inside a directory that match a certain pattern,
 ; we can tell Git to change our name or email (e.g., when committing
 ; with different credentials from a work email)
 path = ~/Dropbox/.custom/.gitconfig
[color]
 branch = auto
 diff = auto
 grep = auto
 interactive = auto
 status = auto
 ui = true
[alias]
 # cut out for brevity, see the section on Aliases
[core]
 editor = nvim
 excludesfile = ~/.gitignore
 pager = diff-so-fancy | less --tabs=4 -RFX
[commit]
 gpgsign = true
 template = ~/.gitmessage
[mergetool "nvim"]
 cmd = nvim $MERGED
[difftool "vscode"]
 cmd = code --wait --diff $LOCAL $REMOTE
[mergetool]
 prompt = false
[color "diff-highlight"]
 oldNormal = red bold
 oldHighlight = red bold 52
 newNormal = green bold
 newHighlight = green bold 22
[color "diff"]
 commit = yellow bold
 frag = magenta bold
 meta = 11
 new = green bold
 old = red bold
 whitespace = red reverse
[push]
 recurseSubmodules = on-demand
[gpg]
 program = gpg2
[filter "lfs"]
 clean = git-lfs clean -- %f
 smudge = git-lfs smudge --skip -- %f
 process = git-lfs filter-process --skip
 required = true
[pull]
 rebase = false
[credential]
 helper = store
[init]
 defaultBranch = main
 templateDir = ~/.git-templates
[diff]
 tool = vscode

The file being referenced/included here, ~/Dropbox/.custom/.gitconfig, looks like this:

 bash
 
 
 
[user]
 email = email@example.com
 name = Your Name
 signingkey = 798034F11B2ADED2
[commit]
 gpgsign = true

# The following block could be used to update credentials when committing
# depending on what directory is opened. So, if you work for COMPANY_NAME
# and the pattern 'COMPANY_NAME' appears in the current working directory,
# then Git can be smart and update your configuration by including
# something like the previous block so that the original email, name,
# signingkey and possibly other config options are overwritten.

; when working at COMPANY_NAME
[includeIf "gitdir:**/COMPANY_NAME/**/.git"]
 path = ~/Dropbox/.custom/.gitconfig-COMPANY_NAME

So this section briefly points out how custom tools like diff-so-fancy can be used, how diff output colors can be customized, how specific code editors like Neovim and VS Code can be opened when committing or dealing with merge conflicts and generally just how the file is meant to be structured.

A lot more on configuring Git can be found in the official documentation on git config.

Conclusion

Being exhaustive when it comes to using Git is definitely not something this article strived to achieve: Git is such a complex piece of software that has been evolving since 2005, when it became a necessity to deal with incoming patches sent to Linus Torvalds to make Linux the fantastic project it has become! There is much, much more to learn, but hopefully this post gives you a slightly different perspective and points the way to a well-rounded journey with the list of references below. Git good!

More resources and references

Websites

Conventional Commits - “A specification for adding human and machine readable meaning to commit messages.”
Git Cheatsheet - “Interactive Git Cheatsheet, categorizing commands based on what they affect.”
Git Immersion - “A guided tour that walks through the fundamentals of Git, inspired by the premise that to know a thing is to do it.”
GitHowTo - “Git How To is a guided tour that walks through the fundamentals of Git, inspired by the premise that to know a thing is to do it.”
GitHub Learning Lab - “With GitHub Learning Lab, grow your skills by completing fun, realistic projects. Get advice and helpful feedback from our friendly Learning Lab bot.”
Learn Git Branching - “An interactive Git visualization tool to educate and challenge!”
Microsoft Learn GitHub modules - “Learn new skills and discover the power of Microsoft products with step-by-step guidance.”
Pro Git Book - “The entire Pro Git book, written by Scott Chacon and Ben Straub and published by Apress, is available here. All content is licensed under the Creative Commons Attribution Non Commercial Share Alike 3.0 license. Print versions of the book are available on Amazon.com.”
Productive Git for Developers - “You will walk through a series of scenarios which you’ll most commonly encounter in your daily work life as a developer.”.
Step-by-step guide to contributing on GitHub - “Have you thought about contributing to an open source project, but you’re too confused (or intimidated) by the process to even try? I’ve been there too!”
Try GitHub - Resources to learn Git: Handbook, cheat sheets, git commands in the browser, etc.

YouTube

CS50 Beyond 2019 - CS50
Git & GitHub Crash Course For Beginners - Traversy Media
Git and GitHub for Beginners - Crash Course - freeCodeCamp.org
Git Tutorial for Beginners: Command-Line Fundamentals (playlist) - Corey Schafer
GitHub Basics Tutorial - How to Use GitHub - freeCodeCamp.org
MIT Missing semester (Version Control) - Git data model - MIT
The Ultimate Git & GitHub Crash Course - Learn to Version Control for Beginners - Laith Harb

Online courses

Git Essential Training (free with free trial) - LinkedIn Learning. “Using a step-by-step approach, author Kevin Skoglund presents the commands that enable efficient code management and reveals the fundamental concepts behind version control systems and the Git architecture. Discover how to track changes to files in a repository, review previous edits, and compare versions of a file; create branches to test new ideas without altering the main project; and merge those changes into the project if they work out.”
Git Started with GitHub - Udacity. “This course is designed to jump right into showing how Git and GitHub work together, focusing on the Git basic workflow. Students can expect to learn the minimum needed to start using Git in about 30 minutes.”
How to Use Git and GitHub - Udacity. “This course, built with input from GitHub, will introduce the basics of using version control by focusing on a particular version control system called Git and a collaboration platform called GitHub.”
Short and Sweet: Get Started with Git and GitHub Right Now - Udemy. “In just 30 minutes, this “Short and Sweet” course covers the essential ideas you need to move forward, without a lot of filler. At the end of the course, you’ll be able to set up a GitHub account, install Git, create and configure new Git and GitHub repositories, create a change history for your software projects, and publish your software projects to GitHub.”
The Ultimate GIT 5-day Challenge - Udacity. “A quick way to determine if working with GIT is something you want to learn more about. This course takes us step-by-step through some basic GIT operations. The course will not dive too deep, and takes small steps on each of five days. As we continue through the course, we learn a basic, single-person workflow that could allow anyone to store files at GitHub or BitBucket.”
Version Control with Git - Coursera. “In this course, you will not learn everything there is to know about Git, but you will build a strong conceptual understanding of the technology, and afterward will be able to confidently dig deeper on any topic that interests you.”
Introduction to Git and Github - Coursera. Offered by Google. “In this course, you’ll learn how to keep track of the different versions of your code and configuration files using a popular version control system (VCS) called Git. We’ll also go through how to setup an account with a service called GitHub so that you can create your very own remote repositories to store your code and configuration.”

References for this article

git rerere - git-scm.
35+ Git Commands List Every Programmer Should Know - loginradius.
Binary search algorithm - Wikipedia.
Gitignore - Toptal.
HackerRank.
LeetCode.
Rewriting history - Atlassian.
The new Git default branch name - GitLab.
What is git-rerere and how does it work? - Stack Overflow.

Using Google Sheets as a database to extract data converted to Markdown

Sébastien Lavoie — Sun, 19 Dec 2021 21:00:00 -0600

Introduction

The purpose of this tool is to parse and convert the content of a Google Sheets spreadsheet to Markdown. In this specific case, I just wanted a simple script to reduce the friction in getting a valid output I could use to maintain a “learning log” on my website. So that gives me a quick way to input data in a spreadsheet and then I can take the time whenever needed to update the learning progress as desired by reformatting any of the entries and optionally adding notes to them.

General software requirements

This tool should:

Be able to read a private spreadsheet with a service account configured with the Google Sheets API.
Convert rows from a Google Sheets spreadsheet into Markdown (see Input and Output sections below for details).
Skip rendering rows that are missing a value for the Title or Date columns (it couldn’t be rendered in the correct place properly without this information).
- Other columns are optional and the corresponding level of nesting will be skipped if no value was provided (e.g. a category Articles may have an item with no sub-category and could be missing any of the other values except for Title and Date).
Separate entries by month by parsing the Date column.
- A command-line parameter should exist to specify a year from which the date should be extracted.
- The output should be in reverse chronological order (i.e. for a full year with entries for each month, December will appear at the top and January at the bottom).
Output a hierarchical format like the following: Category > Sub-category > Title > Activity & notes
Group entries by category and sub-category (e.g. if a category is named Articles and there is a sub-category named Python, then Python must be nested under Articles for the corresponding month).
Sort on the Date column before doing any parsing on other columns (rows in the spreadsheet can be in non-chronological order).
- Dates from a single year will be kept (either the year received as a parameter or the most recent year found).
Sort Category, Sub-category and Title alphabetically (ascending order from top to bottom).
Not sort the Activity column to preserve the order in which rows were added to the spreadsheet.
Render links only if the Link column is a valid URL.
Not validate the content of columns except for Date and Link (i.e. render the other columns as is).
Display items with no category above those having a category. The same logic would be true for sub-categories.
Format the output:
- Date should become headers (##).
- Category should be bolded.
- Sub-category should be emphasized.
- Title should become a link if the Link column contains a valid URL.
- Activity should be nested under Title, occupying a new line for each activity.
- Notes should appear (italicized and inside parentheses) next to the activity.
  - If there is a value for Notes but no value for Activity, the output would be next to the Title.

Input

A spreadsheet with the following columns (starting with Date):

[additional info]	Date	Category	Sub-category	Title	Activity	Link	Notes
November	13/11/2021	Articles	Python	Python slots, slots, and object layout		https://www.youtube.com/watch?v=Iwf17zsDAnY	understood that X does Y
November	14/11/2021	Books	Software engineering	The Pragmatic Programmer	ch. 1	https://pragprog.com/titles/tpp20/the-pragmatic-programmer-20th-anniversary-edition/	note 1 here
November	15/11/2021	Books	Software engineering	The Pragmatic Programmer	ch. 2		note 2 here too
November	16/11/2021	Books	Software engineering	Clean Code
December	1/12/2021	Documentation	Python	The Python Tutorial	Sections 1-4	https://docs.python.org/3/tutorial/index.html
December	2/12/2021	Books	Software engineering	Clean Code	ch. 2-10
Render available values	2/12/2021	Books		Clean Coder			book note
Render available values	2/12/2021			Python/C API Reference Manual		https://docs.python.org/3/c-api/index.html
Title missing: skip rendering	3/12/2021	Articles
Title missing: skip rendering	3/12/2021	Articles	Software engineering		ch. 3	https://docs.python.org/3/c-api/index.html	note
Date missing: skip rendering		Articles	Python	Python slots, slots, and object layout	Section 2	https://www.youtube.com/watch?v=Iwf17zsDAnY	note not rendered

Output

A Markdown output to the terminal in the following format:

 markdown
 
 
 
## December

- [Python/C API Reference Manual](https://docs.python.org/3/c-api/index.html)
- **Books**
 - Clean Coder (_book note_)
 - _Software engineering_
 - Clean Code
 - ch. 2-10
- **Documentation**
 - _Python_
 - [The Python Tutorial](https://docs.python.org/3/tutorial/index.html)
 - Sections 1-4

## November

- **Articles**
 - _Python_
 - [Python slots, slots, and object layout](https://www.youtube.com/watch?v=Iwf17zsDAnY) (_understood that X does Y_)
- **Books**
 - _Software engineering_
 - Clean Code
 - [The Pragmatic Programmer](https://pragprog.com/titles/tpp20/the-pragmatic-programmer-20th-anniversary-edition/)
 - ch. 1 (_note 1 here_)
 - ch. 2 (_note 2 here too_)

Input/Output example when using this tool.

Installing this tool

Tested only under Python 3.9.7, requires at least Python 3.7+.

Using `pip`

[recommended] Activate a virtual environment.
To install dependencies, run from this directory: pip install -r requirements.txt
Then execute the script to run the tool: python3 get_learning_logs

Using `pipenv`

Run from this directory: pipenv install.
Activate the newly created virtual environment with pipenv shell.
Then execute the script to run the tool: python3 get_learning_logs

How to use this tool

You will need to create an environment file (default path: ~/.learning-logs or edit the path for LEARNING_LOGS_ENV_PATH in get_learning_logs.py). It should look as follows:

 env
 
SERVICE_ACCOUNT_FILEPATH="/path/to/service_account_learning_logs.json"
SPREADSHEET_ID=SPREADSHEET_ID # Found in the URL of the document
WORKSHEET_ID=0 # First sheet is 0 by default, it comes after the URL parameter `gid`

To set up this project, you will have to:

Enable the Google Sheets API for your project (https://support.google.com/googleapi/answer/6158841?hl=en).
In the project, search for “service account” in the search bar and create new credentials.
Download the credentials and put them at the path SERVICE_ACCOUNT_FILEPATH referenced in the environment file.
Create a spreadsheet with the following columns (not necessarily in that order): Date, Category, Sub-category, Title, Activity, Link, Notes.
- The column Date Should match a date format: select all cells from that column, go to Format > Number > Date to apply the expected format (mm/dd/yyyy). You can then input a date (e.g. 11/1/2021) and you will see a calendar pop up when double-clicking on it. Dragging it down to a new cell will create a new date for the next day (in this example, 11/2/2021).
Get its SPREADSHEET_ID (https://docs.google.com/spreadsheets/d/SPREADSHEET_ID_HERE/edit#gid=0) to put it in the environment file.
Get the WORKSHEET_ID (the integer after gid= in the URL of the document).
From the spreadsheet document (or from the folder view in Google Drive), share with the email associated with the service account (e.g. learning-logs-something@project-name-12345.iam.gserviceaccount.com). Read access (“Viewer” permission) is enough.

By now, the script is ready to be called. I just created an alias to execute it more easily by putting the following in ~/.bash_aliases (you will have to adapt the paths of course):

 bash
 
alias learning-logs='~/.local/share/virtualenvs/learning-logs-to-markdown-XJLvhmzn/bin/python3.9 \
 ~/dev/sglavoie/dev-helpers/learning-logs-to-markdown/get_learning_logs.py'

This will output the converted data from the spreadsheet to Markdown as shown above.

Conclusion

This little tool scratched an itch and will be helpful in making the process of updating the learning progress of this website more straightforward, more convenient and less hands-on! You can find its source code on GitHub.

Convenient and lightning fast backups with rsync

Sébastien Lavoie — Sat, 31 Jul 2021 09:01:00 -0500

Introduction

Coming into its fourth year of existence and usage in production environments, this simple script has proven to be a fantastic ally. Despite the plethora of great options out there that have more bells and whistles, I have yet to find a single complaint about this solution as it hinges on the minimalist side, requiring only a working Python 3 installation along with the powerful rsync command-line tool.

It’s one of those pieces of code that perform some really basic tasks that can be scheduled with cron in the background, that one forgets about until a real need to access a backup comes up. This script is barely a Python wrapper calling rsync but still, I felt like sharing it for those who like uncomplicated setups. I’ve literally left this script alone for years some time after the release of Python 3.6 which was released back in December 2016 and it has been going strong since then mostly due to the fact it does not do much more than backing files up and leveraging rsync.

Prerequisites

As briefly mentioned above, this tool uses Python 3.6+ and rsync. That’s it, really.

Setting up the script

This script is being made available on GitHub within a set of other small tools. To get the source code for this particular tool, you can either clone the whole repository, get the content of the files separately or simply copy and paste the current version from below. You will need a settings.json file as well as the Python script that will run the necessary commands.

The configuration file

Right off the bat, you will most probably want to update the array of strings for the data_sources key as well as the string for the key data_destination in the following JSON file.

 json
 
 
 
{
 "data_sources": [
 "/home/sglavoie"
 ],
 "data_destination": "/media/sglavoie/Elements",
 "terminal_width": 40,
 "sep": "=-",
 "log_name": ".backup_log_",
 "log_format": "%y%m%d_%H_%M_%S",
 "rsync_options": [
 "-vaAHh",
 "--delete",
 "--ignore-errors",
 "--force",
 "--prune-empty-dirs",
 "--delete-excluded"
 ],
 "backup_exclude": ".backup_exclude"
}

By default, the following options are passed to rsync:

Option	Description
`-vaAH`	verbose, archive, ACLs, hard-links (preserve)
`--delete`	“delete extraneous files from destination dirs”
`--ignore-errors`	“delete even if there are I/O errors”
`--force`	“force deletion of directories even if not empty”
`--prune-empty-dirs`	“prune empty directory chains from the file-list”
`--delete-excluded`	“also delete excluded files from destination dirs”

Description of available settings

Name of setting	Description
`data_sources`	Directories to backup, supplied as a list of strings (no slash at the end).
`data_destination`	Single destination of the files to backup, supplied as a string. This can be overridden when passing option `-d` or `--dest` to the script
`terminal_width`	Line length in the terminal, used for printing separators.
`sep`	Separator to use along with `terminal_width`.
`log_name`	Sets the prefix of the log filename.
`log_format`	This goes right after `log_name` as a suffix.
`rsync_options`	Options to use with rsync as a list of strings.
`backup_exclude`	Default file in each source in `data_sources` where files/directories will be ignored.

The script

 python
 
 
 
"""
Script that uses `rsync` to make a simple and convenient backup.
Note: requires Python 3.6+. No other Python third-party libraries required.
"""
import argparse
import datetime
import glob
import json
import os
import pathlib
import subprocess


def run_backup():
 """This is where all the action happens!"""
 settings = get_settings()

 # initiate the parser
 parser = argparse.ArgumentParser("backup")
 parser.add_argument(
 "-c",
 "--clear",
 help="Delete all log files for current source in DATA_SOURCES.",
 action="store_true",
 )
 parser.add_argument(
 "-s",
 "--src",
 dest="source",
 default=None,
 help="Specify an alternative source to backup as a string.",
 action="store",
 )
 parser.add_argument(
 "-d",
 "--dest",
 dest="destination",
 default=None,
 help="Specify an alternative destination for backup as a string.",
 action="store",
 )

 # read arguments from the command line
 arguments = parser.parse_args()

 # check for --source or -s
 # Replace potential list of sources to backup with this one
 if arguments.source is not None:
 if os.path.isdir(arguments.source):
 settings["data_sources"] = [arguments.source]
 else:
 print("Please enter a valid source to backup.")
 exit(0)

 # check for --clear or -c
 if arguments.clear:
 clear_logs(
 data_sources=settings["data_sources"],
 log_name=settings["log_name"],
 )

 # check for --dest or -d
 # Replace destination to backup with this one
 if arguments.destination is not None:
 if os.path.isdir(arguments.destination):
 settings["data_destination"] = arguments.destination
 else:
 print("Please enter a valid destination.")
 exit(0)

 # don't run the script if the destination doesn't exist
 if not os.path.isdir(settings["data_destination"]):
 print(
 f"The destination doesn't exist.\n({settings['data_destination']})"
 )
 exit(0)

 backup_all_sources(settings)


def backup_all_sources(settings: dict) -> None:
 """Iterate over all sources to backup."""
 for source in settings["data_sources"]:
 date_now = datetime.datetime.now()
 log_format = datetime.datetime.strftime(
 date_now, settings["log_format"]
 )
 log_filename = f"{source}/{settings['log_name']}{log_format}"
 log_option = f"--log-file={log_filename}"

 backup_source = settings["backup_cmd"].copy()
 backup_source.extend([log_option])

 # files to ignore in backup
 exclude_file = f"{source}/{settings['backup_exclude']}"

 if os.path.exists(exclude_file):
 exclude_option = f"--exclude-from={exclude_file}"
 backup_source.extend(
 [exclude_option, source, settings["data_destination"]]
 )
 else:
 # skips '--exclude-from' option if no file is found
 backup_source.extend([source, settings["data_destination"]])

 settings["source"] = source
 settings["backup_source"] = backup_source
 settings["log_filename"] = log_filename

 try:
 backing_source(settings)
 except KeyboardInterrupt:
 print("\nKeyboardInterrupt: Exiting operations.")
 exit(0)


def backing_source(settings: dict) -> None:
 """Print information to STDOUT and to `log_filename` and executes the
 rsync command."""
 print(settings["sep"] * settings["terminal_width"])

 cmd_executed = " ".join(settings["backup_source"])
 msg_executed = f"Command executed:\n{cmd_executed}\n"
 print(msg_executed)

 with open(settings["log_filename"], mode="w") as log_file:
 log_file.write(f"{msg_executed}\n")

 try:
 child = subprocess.Popen(settings["backup_source"])
 _ = child.communicate()[0] # call communicate to get the return code
 rc = child.returncode
 except FileNotFoundError:
 print(f"FileNotFoundError: Is the `rsync` tool installed?")
 exit(1)

 print(f"\nBackup completed for: {settings['source']} (return code: {rc})")
 print(settings["sep"] * settings["terminal_width"])


def clear_logs(data_sources: list, log_name: str) -> None:
 """Clears log files for each source specified in SETTINGS."""
 for source in data_sources:
 # Retrieve a list of all matching log files in `source`
 log_files = glob.glob(f"{source}/{log_name}*")
 if log_files == []:
 print(f"\nThere is no log file to delete in {source}.")
 exit(0)
 else:
 print(f"Log files in {source}:")
 for log_file in log_files:
 print(log_file)
 if user_says_yes():
 for log_file in log_files:
 os.remove(log_file)
 print("Log files deleted.")
 print("Exiting script...")
 exit(0)


def user_says_yes(
 msg: str = "\nDo you want to delete log files for this source? (y/n) ",
) -> bool:
 """Asks the user to enter either "y" or "n" to confirm. Returns boolean."""
 choice = None
 while choice is None:
 user_input = input(msg)
 if user_input.lower() == "y":
 choice = True
 elif user_input.lower() == "n":
 choice = False
 else:
 print('Please enter either "y" or "n".')
 return choice


def get_settings() -> dict:
 """
 Get the settings from `settings.json`.

 Returns:
 dict: Containing all settings used by the tool.
 """
 directory = pathlib.Path(__file__).parent.absolute()
 with open(directory / "settings.json") as fp:
 settings = json.load(fp)

 backup_cmd = ["rsync"]
 backup_cmd.extend(settings["rsync_options"])
 settings["backup_cmd"] = backup_cmd

 return settings


if __name__ == "__main__":
 run_backup()

How to use

One-time setup

Set all values in settings.json to suit your needs.
Make sure that the backup destination is available/mounted. A simple warning will be echoed if the destination can’t be found.
Call the Python script. As an example, if it’s located at /home/user/backup/rsync_backup.py, then you could put the following alias in ~/.bash_aliases (or equivalent)*:

 bash
 
alias backup='python3 /home/user/backup/rsync_backup.py'

* Python may be called differently on your system, e.g. simply python instead of python3.

In this instance, the script can now be executed in a terminal with the keyword backup along with optional arguments.

Using as a daily driver

Make sure the destination to back files up is available/mounted and simply call the script (e.g. backup in this example).

The output will look something like this:

 bash
 
 
 
$ backup
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Command executed:
rsync -vaAHh --delete --ignore-errors --force --prune-empty-dirs \
 --delete-excluded --log-file=/home/sglavoie/.backup_log_210731_09_28_09 \
 --exclude-from=/home/sglavoie/.backup_exclude /home/sglavoie /tmp

building file list ... done
deleting sglavoie/.backup_log_210731_08_47_24
sglavoie/
sglavoie/.bash_aliases
...

Avoid deleting files at the destination

The rsync command first builds a list of the files that are part of the backup and will go on adding new files, updating existing files as well as removing files that are found in the destination but not in the source anymore according to the default list of flags passed to it in this implementation. It is therefore important to note that if you would like to back up files in the destination that no longer exist in the source, a convenient solution is simply to create a new folder at the root level inside the destination directory and manage those files separately.

For instance, I sometimes want to back up large video files but there is rarely a need to update them. In this case, it would be possible to add the directories to exclude in the file .backup_exclude (default configuration option) like so:

Videos/
Dropbox/other/folder/

Then, these directories will be ignored when backing up the source (if they are nested inside the source directory to start with!). If the destination is set to /media/user/backup and the source set to /home/user/my_folder, then the script would recreate the “root” directory of the source if it doesn’t exist on the destination:

bash

$ ls /media/user/backup
my_folder

You could keep track of those heavy files that do not need to be backed up frequently by putting them manually next to my_folder in /media/user/backup. This will lead to huge speed ups when running the script if you track many files in this way!

Conclusion

This was a quick overview of what rsync can offer: I neither touched upon its remote syncing capabilities nor the vast majority of its options, for which you can find a lot more information about by typing man rsync to display the manual page in the terminal.

Without making use of any fancy features of recent versions of Python or rsync, I have found this simple backup procedure to have worked flawlessly for a long time. Hardware inevitably fails at some point and after having heard about countless examples of people losing some or all of their most precious files (including my own story…), I have come to enjoy backing my system with this little script.

I wouldn’t trust my external drive to last forever either, hence I also rely on Dropbox and Google Drive for documents I find are a good fit. But overall, I just do not tend to back up every single system file in the cloud, so there is still a convincing use case where daily and complete backups become a possibility when using a fast approach like rsync to incrementally save snapshots.

More resources and references

rsync official website – samba.org.
Keeping Linux files and directories in sync with rsync – RedHat, going through the main options offered by the tool.
How To Use Rsync to Sync Local and Remote Directories – DigitalOcean, demoing how to use the tool and covering the case of remote file access.

Managing dotfiles with a Git bare repository

Sébastien Lavoie — Sun, 30 May 2021 13:13:00 -0500

Introduction

I have been managing my configuration files for the past year using a technique I originally came across in this YouTube video from DistroTube. This uses a Git bare repository instead of the “usual” way, which consists of using a normal Git repository where one would store the actual configuration files and create symbolic links to all those files to the expected path on the system.

Needless to say, symlinks are a pain to manage when potentially hundreds of files need to be tracked in this way. Fortunately, working with a bare repository is much simpler: there’s neither a need to move files around nor store any of them in the dotfiles bare repository either. Let’s dive in to see how this works! I’ll keep the instructions to the point: for more details, please refer to the article The best way to store your dotfiles: A bare Git repository from Atlassian, the source of knowledge behind this post.

1. Set up the system

Open a terminal and brace yourself to type a few commands. This will need to be done only once on any machine you use.

Set the following variable to the name of the directory where you want to store your dotfiles (e.g. .dotfiles, dotfiles, cfg, etc.):

 bash
 
DOTFILES=dotfiles

Then type the following commands:

 bash
 
git init --bare $HOME/$DOTFILES
alias config='/usr/bin/git --git-dir=$HOME/$DOTFILES/ --work-tree=$HOME'
config config --local status.showUntrackedFiles no

Depending on where you store your terminal aliases (e.g. .bashrc, .zshrc, .bash_aliases, etc.), set the following variable according to your needs:

 bash
 
ALIASES=.bash_aliases

Once that is done, type the following command (or open the file containing your aliases with a text editor to manually put the alias wherever you want):

 bash
 
echo "alias config='/usr/bin/git --git-dir=$HOME/$DOTFILES/ --work-tree=$HOME'" >> $HOME/$ALIASES

You can of course change the config alias to something else. I like to simply use c instead.

1.1 Create a repository to host your dotfiles

At this point, you will want to make sure you have created an empty Git repository on a website such as GitHub, GitLab or BitBucket. In the interface of that website, you should find a place that allows you to copy the link to that Git repository, which might look like something like this: git@github.com:username/dotfiles.git.

1.2 Set the remote to track your dotfiles

With your new Git repository created in the previous step, you will now want to add tracking information to be able to connect to the remote Git repository. You can do so as follows:

bash

config remote set-url origin git@github.com:username/dotfiles.git

For more information about this, you can read the article Setting up SSH and Git for Multiple Accounts if you’ll be using an SSH key to connect and more specifically the official documentation on the git remote command.

2. Use the new system

2.1 On your current machine

Tracking changes will now be a breeze! All we have to do is to add files with our new alias (assuming config was chosen) to Git, commit and push to a remote repository to keep everything neatly backed up.

A typical workflow will look like this:

 bash
 
 
 
config status
config add .gitconfig
config commit -m "Add .gitconfig"
config add .zshrc
config commit -m "Add .zshrc"
config push

2.2 On a different machine

Once yet have set up the system once with a remote repository, then the process becomes simpler as you can clone that repository and save a few steps. Here’s what you’d be expected to do to set up the same dotfiles management system on another machine:

 bash
 
 
 
# set to different values as desired
DOTFILES=dotfiles
REMOTE=git@github.com:username/dotfiles.git

alias config='/usr/bin/git --git-dir=$HOME/$DOTFILES/ --work-tree=$HOME'
echo "$DOTFILES" >> $HOME/.gitignore
git clone --bare $REMOTE $HOME/$DOTFILES
config checkout

At this point, you might see an error message if you have existing configuration files on your machine in locations that are already tracked by Git in your dotfiles on the remote repository. If that’s the case, you can move them to a secure place like so:

 bash
 
mkdir -p .config-backup && \
config checkout 2>&1 | egrep "\s+\." | awk {'print $1'} | \
xargs -I{} mv {} .config-backup/{}

Let’s make sure we don’t show all untracked files again:

bash

config config --local status.showUntrackedFiles no

From here, this will be the same workflow as described in the section 2.1 On your current machine.

Conclusion

No more symlinks: this is definitely a nicer approach! You will need to be a bit more familiar with Git if you need to edit your Git history such as with the rebase sub-command or when adding changes selectively such as with the sub-command add -p to add files interactively — which might be the case if you want to commit only part of the changes made to one file — but that’s a great tool to master anyways, and there’s an article to get the ball rolling: Git the gist of it: common commands for a working workflow ;).

More resources and references

Git Bare Repository - A Better Way To Manage Dotfiles — DistroTube, YouTube
The best way to store your dotfiles: A bare Git repository, the source of the idea for this post. — Atlassian

Ignoring Sync of Local Files to Dropbox on Linux

Sébastien Lavoie — Sat, 30 Nov 2019 13:22:00 -0600

Introduction

Dropbox is an awesome company which has been until recently the working home of none other than Guido van Rossum, the creator of the Python programming language. It works like a charm out of the box on Linux and can be installed easily enough through the package manager that comes with your chosen distribution.

Why Dropbox?

In my experience, it has been working flawlessly and syncs consistently faster than Google Drive. It’s also worth mentioning that there is still no official Google Drive Linux client to this day: some users of the GNOME desktop environment will connect to their Google account and mount Google Drive within the file manager Nautilus/Files; other users will use the paid software Insync or the free software alternative ODrive. No thanks.

On the other hand, Dropbox is easy to use with any flavor of Linux and also syncs seamlessly with mobile devices. I use Dropbox for its convenience, but other solutions exist like Syncthing that do not depend on proprietary software and third-party cloud services. I store mostly insensitive information in my Dropbox account and for the few files that happen to be a bit more sensitive (such as the password database I synchronize on both desktop and mobile with KeePassXC and Keepass2Android, respectively), they are of course encrypted (hopefully in a secure way).

Because I like to keep things simple and light on system resources, I do not let Dropbox run in the background (it only syncs once a day in the evening). Since all one needs to do in order to use it is to create and modify files inside a specific folder (such as ~/Dropbox by default), it can just synchronize at any moment and conflicts will be handled automatically if you make changes in more than one place that’s being synced.

So, what’s wrong with Dropbox?

With all that said, there’s one little feature that I find missing with Dropbox when using the official Linux client and that’s the ability to selectively sync local files. Yes, there is a selective sync feature available, but that will delete the local copy of your files and store them in the cloud only, not the other way around by keeping your local files intact and prevent uploading them to the cloud. There are some hacks you can do to avoid this, but this isn’t a particularly enticing work flow, because we can do better.

You may have heard of other alternatives like GoodSync that allow, through yet another third-party proprietary software, to ignore specific files at your request. That’s not what we are after here: instead, there’s something you can do from the terminal that works quite well, too. At the time of this writing, it is still a beta feature according to this page:

This feature is currently in beta and not available to all Dropbox users. It will be rolled out to more users in the future.

Yet, because you can work with this feature programmatically to ignore and un-ignore files from your Dropbox directory, I thought it was worth giving it a try. Here is how I’m using it to give you a few ideas.

Any simple example?

Getting started is quick and painless. To ignore a file within your Dropbox folder, you only need to use the attr (on Linux, else xattr on macOS) command to set (-s) (or -w write on macOS) the special attribute com.dropbox.ignored to give it a value of 1 (-V 1) so it ignores a file (file.txt in this example) as follow:

 bash
 
cd /path/to/file/in/Dropbox/folder

attr -s com.dropbox.ignored -V 1 file.txt # Linux
xattr -w com.dropbox.ignored 1 file.txt # macOS

Ignore all .git and node_modules folders

Things get really interesting when we can do that automatically by matching a specific pattern (it depends who you ask, anyways). How about exploring recursively the whole directory tree to exclude only hidden .git folders? Suppose you store your Git projects in ~/Dropbox/git/. You would then do something like this:

 bash
 
 
 
cd ~/Dropbox/git/

# Linux
find . -type d -name ".git" |
 xargs -I {} attr -s com.dropbox.ignored -V 1 "{}"

# macOS
find . -type d -name ".git" |
 xargs -I {} xattr -w com.dropbox.ignored 1 "{}"

In an instant, you will start getting some feedback printed to the terminal for each .git folder that is being found:

Attribute "com.dropbox.ignored" set to a 1 byte value for ./path/to/.git:
1

Dealing with node_modules folders is equally easy. Because a message will be printed to the console every time a file attribute is changed, you could want to delete the output (preferably once you are sure you are modifying only the targeted files!) by redirecting it to /dev/null, a null device that discards all data it receives. We can always split a long command onto multiple lines with a backslash character \ like so (it’s not required right after the pipe | symbol):

 bash
 
find . -type d -name "node_modules" |
 xargs -I {} attr -s com.dropbox.ignored -V 1 \
 "{}" > /dev/null

Can we make this simpler?

Yes we can! An alias might be a bit hard to manage, but we can definitely create a set of functions that will be accessible from anywhere in the terminal. It suffices to add them to your shell configuration file, which could be located in ~/.profile, ~/.bash_profile, ~/.zshrc or something else depending on your shell.

One solution that I see fit for this purpose is to have two distinct functions that have antagonistic effects so that we can revert our changes in a pinch. Basically, setting a value of 1 for the attribute com.dropbox.ignored ignores a file while setting it back to 0 tells Dropbox to sync it again. Our functions could be conveniently called dropbox-ignore and dropbox-sync.

Now, the following functions could become quite a bit more complicated to take into account various scenarios and custom behaviors, but let’s keep things simple and assume that the user knows what is going to happen. We could have our two functions as follow:

 bash
 
 
 
# Ignore specific files/directories in Dropbox
dropbox-ignore() {
 arg1=$1
 arg2=$2

 if [ -z "$arg1" ] || [ -z "$arg2" ]; then
 echo "Usage: dropbox-ignore <f|d> <name> (e.g. dropbox-ignore d .git)"
 return 1
 fi

 find ~/Dropbox -type $arg1 -name "$arg2" |
 xargs -I {} xattr -w com.dropbox.ignored 1 "{}"
}

# Sync specific files/directories in Dropbox
# that were previously ignored (or not)
dropbox-sync() {
 arg1=$1
 arg2=$2

 if [ -z "$arg1" ] || [ -z "$arg2" ]; then
 echo "Usage: dropbox-sync <f|d> <name> (e.g. dropbox-sync d .git)"
 return 1
 fi

 find ~/Dropbox -type $arg1 -name "$arg2" |
 xargs -I {} xattr -d com.dropbox.ignored "{}"
}

How do you use those functions?

Our commands will be accessible by typing dropbox-ignore and dropbox-sync the next time we open a terminal window. Both functions do exactly the same thing, but dropbox-ignore sets the attribute value to 1 to ignore files and dropbox-sync sets the attribute to 0 to allow syncing to happen.

_{Note: For simplicity and practicality, keep in mind that those commands will search recursively from the specified directory. It’s also good to know that if you set an attribute value to 1 then back to 0 again, Dropbox will need to re-sync the affected files and directories.}

We need to pass them two arguments: the first is the type of search to perform (pass f for files and d for directories) and the second is the pattern to match in the name (node_modules or .git would fit here, no need to use quotes). Concretely, that will look like this:

 bash
 
# To ignore a file matching the pattern `myfile`
dropbox-ignore f myfile

# To avoid syncing node_modules/ directory
dropbox-ignore d node_modules

# Te re-add a .git/ folder to Dropbox
dropbox-sync d .git

Conclusion

This tip may certainly not be the best approach to solving this particular issue with Dropbox on Unix, but I hope you find it useful nevertheless as it worked out very nicely on my end! One could store files and directories elsewhere, working with symbolic links or hard links as necessary to reference files or simply give up on making a backup with Dropbox in some circumstances. But I like to keep an extra copy of projects I’m working on in Dropbox and that can include potentially large files that don’t need to be backed up, so ignoring what’s not indispensable can speed up the syncing process tremendously while keeping disk usage in the cloud possibly much lower.

Dropbox only calculates the size of the files being synced in the cloud, so you can end up needing a lot more space locally if you ignore a number of big files or, conversely, selectively syncing many files will result in a cloud storage larger than what’s needed locally.

Black: A Python Formatter to Brighten Your Workflow

Sébastien Lavoie — Sat, 13 Jul 2019 13:10:00 -0500

What is Black?

From its GitHub repository:

Black is the uncompromising Python code formatter. By using it, you agree to cede control over minutiae of hand-formatting. In return, Black gives you speed, determinism, and freedom from pycodestyle nagging about formatting. You will save time and mental energy for more important matters.

Black will take less than ideal code and will automatically reformat it according to the PEP-8 style guide. It will take care of many things, including the following (taken in that order from README.md in Black’s repo):

Wrapping lines properly by shortening as much as possible single statements;
Break long lines;
Remove superfluous empty lines;
Take care of adding trailing commas where necessary;
Convert single quotes (') to double quotes (");
Convert numeric literals to lowercase (0XAB to 0xAB);
Superfluous parentheses are removed;
Split call chains on different lines;
Works from the terminal and support many code editor integrations (Emacs, Vim, VS Code, Sublime, etc.).

Now that it’s clear why it can be a good idea to use it, let’s see how it works with Vim and the terminal.

How to use it?

First, we need to install it using Python 3.6.0+:

bash

$ pip install black

That’s it!

Use it from the terminal

In its simplest invocation, we can use Black by specifying a file or a directory by typing:

bash

$ black path_to_file_or_directory

It doesn’t come with many options as its goal is to automatically format code consistently, but you can find out more about this tool in the following way:

bash

$ black --help

Use it with Vim

Using Black as part of your workflow in a code editor is where I believe it really shines, because you can see almost in real-time when you make a formatting mistake, which will help down the road in committing less sins.

Integrating Black with Vim or Neovim is quick and painless, adding a simple line in your .vimrc or init.vim file. You can also proceed to do a manual installation without a plugin manager as explained in the GitHub repository.

With vim-plug:

 vim
 
Plug 'python/black'

With Vundle:

 vim
 
Plugin 'python/black'

Once the plugin is installed, it may well be convenient to automatically format Python files when saving the buffer, which can be accomplished by adding the following line to the configuration file:

 vim
 
autocmd BufWritePre *.py execute ':Black'

By default, Black works surprisingly well on its own. I only went ahead and changed the default line length from 88 to 79 characters by also adding this line:

 vim
 
let g:black_linelength = 79 " default is 88

Finally, it’s worth noting you can update Black at any point from the terminal with pip and you can also do it directly from within Vim/Neovim with this command:

 vim
 
:BlackUpgrade

Vim will then output something along those lines:

Upgrading Black with pip…

DONE! You are all set, thanks for waiting

Any example?

Taking a random Python 3.6+ code sample, we could originally have something like this (hopefully not):

 python
 
 
 
 for source in data_sources :


 # Retrieve a list of all matching log files in `source`
 log_files=glob.glob( f'{source}/{LOG_NAME}*' )
 if log_files == [ ]:
 print(f'\nThere is no log file to delete in {source}.')
 continue
 else :
 print(f'Log files in {source}:')
 for log_file in log_files:
 if (ARGUMENTS.remind): REMINDER_IS_SET = True
 print(
 log_file
 )
 message = (
 '\nDo you want to delete log files ' 'for this source? (y/n) '
 )
 if (user_says_yes( message = message )) :
 for log_file in log_files: os.remove(log_file)
 print(
 'Log files deleted.'
 )
 continue
 else : continue
 print( 'Exiting script...' )
 sys. exit(0)

Running Black, we will get something that’s quite a bit more palatable:

 python
 
 
 
 for source in data_sources:

 # Retrieve a list of all matching log files in `source`
 log_files = glob.glob(f"{source}/{LOG_NAME}*")
 if log_files == []:
 print(f"\nThere is no log file to delete in {source}.")
 continue
 else:
 print(f"Log files in {source}:")
 for log_file in log_files:
 if ARGUMENTS.remind:
 REMINDER_IS_SET = True
 print(log_file)
 message = (
 "\nDo you want to delete log files " "for this source? (y/n) "
 )
 if user_says_yes(message=message):
 for log_file in log_files:
 os.remove(log_file)
 print("Log files deleted.")
 continue
 else:
 continue
 print("Exiting script...")
 sys.exit(0)

Conclusion

I think the results speak for themselves: enforcing a style with a tool like Black is definitely useful to avoid arguments about conflicting coding styles. It lets you focus on what matters instead: being productive while providing value to the world.

fzf - A Fuzzy Finder to Accomplish Anything

Sébastien Lavoie — Fri, 24 May 2019 10:56:00 -0500

Introduction

I know people who make such a mess when not organizing their files and directories appropriately on their system, they would benefit greatly from fzf.

What is fzf?

From the official GitHub page:

fzf is a general-purpose command-line fuzzy finder. It’s an interactive Unix filter for command-line that can be used with any list; files, command history, processes, hostnames, bookmarks, git commits, etc.

Terminal aliases

One of the main uses of fzf is from the terminal. Being so flexible to use, it can be combined with all kinds of commands with the help of pipes (|) to bend it to your desires. Here are some aliases I am currently using that undoubtedly improve my terminal workflow.

 bash
 
##### Functions

# Select a configuration file with fzf and open it with Neovim
conf() { du -a ~/.dotfiles/* ~/.config/* | awk '{print $2}' | fzf | xargs -r nvim ;}

# Select a file from current folder and recursively with fzf and open it with Neovim
se() { du -a ./* | awk '{print $2}' | fzf | xargs -r nvim ;}

# Select a file recursively from university folder with fzf and open it with default app
sc() { du -a ~/Dropbox/university/* | awk '{print $2}' | fzf | xargs -r xdg-open ;}

The first alias, conf (short for configuration), allows to search only within the two folders specified for configuration files, which makes it pop almost instantaneously since it doesn’t have to scan files scattered anywhere else. You can then type anything that partially matches a file path and even include slashes (/) in your match if you know in which directories to look. You can then type Enter to open the file with your favorite text editor (here set to nvim) or type CTRL + c to abort the command.

The other aliases work in a similar fashion. se (short for search) will simply search recursively for any kind of files in the current directory and open the selected file in a text editor. Of course, certain file types can be excluded and everything else can be tweaked with more piping power. In the case of the third alias, sc (short for school), it will also search recursively for any kind of file in the university folder and will open it with the default application set to open that kind of file (video, image, text, PDF, etc.).

Vim/Neovim integration

Using Vim-Plug from the same author as fzf is as easy as adding the following to Vim/Neovim’s configuration file. First, we make sure that fzf is available on the system and install it if it’s not. Then, we install the fzf.vim plugin to integrate fzf in Vim and optionally, we may configure more options so that fzf can be used more optimally, which can be found on the GitHub plugin page.

 vim
 
 
 
Plug 'junegunn/fzf', { 'dir': $HOME . '/.fzf', 'do': './install --all' }
Plug 'junegunn/fzf.vim'

""""" [ FZF ]
" Allows fzf to ignore patterns in .gitignore
let $FZF_DEFAULT_COMMAND = 'ag -g ""'

" Mapping selecting mappings
nmap <leader><tab> <plug>(fzf-maps-n)
xmap <leader><tab> <plug>(fzf-maps-x)
omap <leader><tab> <plug>(fzf-maps-o)

" Insert mode completion
imap <c-x><c-k> <plug>(fzf-complete-word)
imap <c-x><c-f> <plug>(fzf-complete-path)
imap <c-x><c-j> <plug>(fzf-complete-file-ag)
imap <c-x><c-l> <plug>(fzf-complete-line)

" Advanced customization using autoload functions
" (expand word completing window)
inoremap <expr> <c-x><c-k> fzf#vim#complete#word({'left': '20%'})

" Make use of fzf command instead of CtrlP
map <C-p> :FZF<cr>
""" [ / FZF ]

Working with Neovim

fzf really shines when used with Neovim as it is extremely fast, especially when configured with The Silver Searcher which deserves its own article. It integrates with core functionality of Vim and makes it easy to find what you are looking for.

From the above configuration, for instance:

 vim
 
nmap <leader><tab> <plug>(fzf-maps-n)

This allows to search for existing mappings and commands, which can be faster than diving in the help pages when looking for a quick reference.

This one in insert mode is very handy:

 vim
 
imap <c-x><c-k> <plug>(fzf-complete-word)

Without affecting keyword completion with CTRL + n and CTRL + p (unless you have set complete+=k in your configuration file), you can complete words from a custom dictionary of your choice with CTRL + x CTRL + k. The window that appears can be moved and resized, which is what is happening here:

 vim
 
inoremap <expr> <c-x><c-k> fzf#vim#complete#word({'left': '20%'})

In the same way, file paths can be completed from the current working directory CTRL + x CTRL + f and existing lines can be quickly inserted with CTRL + x CTRL + l.

Finally, I use fzf to open any file quickly from the working directory inside Neovim with the mapping CTRL + p, which replaces the CtrlP plugin and can work much faster on larger codebases.

Conclusion

There is much more that can be done with it and I am barely scratching the surface here. Luke Smith shared a great video that will complement the information from this post nicely with a more technical approach. Highly recommended to see more practical ways to use this superb piece of software!

Luke Smith also makes a strong point of combining fzf with dmenu, a tool that I described succinctly in a previous article.

Using Embedded Terminals Inside Neovim

Sébastien Lavoie — Wed, 16 Jan 2019 17:32:00 -0600

Terminals & Neovim

Terminal buffers are a built-in feature of Neovim. This means that terminals can be launched inside Neovim just like any other buffer, may it be a split window, a new tab or an independent buffer. Why is this so useful? Be prepared to be amazed.

(click to open)

Seamless integration with Vim commands

For a start, terminals are not just opened inside Neovim: they are truly integrated. When you type the letter i in a terminal buffer, it switches to a new mode, Terminal, where you have access to all underlying features of a regular Bash terminal, such as CTRL + r to search, tab completion of commands, CTRL + w to delete a word backwards or CTRL + u to delete the whole line backwards.

When you escape the Terminal mode, you switch back to Normal mode, where all Vim commands are available as usual, such as navigation, search, etc. Where this gets interesting is when you consider that the terminal is just like any other buffer (except it’s not modifiable) and so you can for instance copy its content or paste into it from different registers. You can split them, view the line number, get automatic scrolling when output appears in Insert mode, you can switch back to Normal mode at any time even if a command hasn’t completed… and of course all commands like gt to go to another tab work as always.

Terminals get supercharged with the power of Vim

This is getting exciting (at least for me). Now, having a terminal with more power is useful in many cases:

A whole terminal session can be saved to another file easily by copying the desired range (:%y to copy everything, for example).
Instead of having to scroll with the mouse, all Vim commands are available to search and navigate through the buffer: gg to go to the beginning of the session, G to go to the end, 12G to go to line number 12, / or ? to search forward and backward respectively. You can even set local or global marks to jump back and forth to precise lines consistently!
It’s even possible to send commands directly to a specific terminal or set up a REPL to execute any code, either a selection, a line or a range of lines. I don’t personally use that feature yet, but I see how useful that can be and you can too if you watch this [screencast presented by Drew Neil](https://thoughtbot.com/upcase/videos/neovim-sending-commands-to-a- terminal-buffer) on thoughtbot.com.
If the terminal is opened in Normal mode, the cursor and the current position are kept intact even though new output could appear, such as would be the case if you are running a server in the background or waiting for a system update to finish. This is especially nice for instance if you are trying to debug a web application: you can leave the cursor exactly where you would like to keep reading the output and never get lost… Plus you have the ability to easily move around, search and copy like a ninja!

The workflow is more compact and centralized

Because the terminals are running with one Neovim instance, you get everything in the same place, which is usually how you need to access your work anyway. If for some reasons you need to have multiple instances of Neovim running, this is possible too and in that situation you could simply split them into different workspaces.

Because everything is launched with Neovim, every terminal left open will also be exited when you close Neovim, shutting down any server or quitting any background interaction going on in the terminals. I have found that this behaviour is what I expect pretty much all the time because the idea of having a terminal inside a working session of Neovim is to work on something related, so it makes sense to close everything that’s to do with a single project or task when you exit the text editor. No need for CTRL + d or exit anymore!

Terminal settings to consider

I don’t use many settings or mappings related to terminals, but I guess the nicest one is to remap the Escape key because by default, this is a weird mapping that leads to twisted fingers (<C-\><C-n>). Here is what I currently have in my init.vim file (the equivalent of .vimrc for Neovim):

 vim
 
 
 
set termguicolors " Make colors look better in terminal

" Exit from terminal buffer (Neovim) more easily (remaps Esc key in
" terminal)
tnoremap <C-[> <C-\><C-n>

" Open terminal buffer (M stands for the Alt key)
nnoremap <M-t> :te<CR>

" Switch to terminal buffer automatically (when only one terminal is
" open)
nnoremap <M-0> :b term://<CR>

" Move between windows exactly the same way as usual
tnoremap <C-j><C-k> <C-\><C-N>
tnoremap <C-h> <C-\><C-N><C-w>h
tnoremap <C-j> <C-\><C-N><C-w>j
tnoremap <C-k> <C-\><C-N><C-w>k
tnoremap <C-l> <C-\><C-N><C-w>l

Conclusion

This was a quick overview of a feature that I have neglected for some time but recently realized how well it integrates with my workflow. You can find some more information by typing :h terminal. If you already use Neovim, it’s worth giving it a try! Otherwise, it’s worth giving Neovim a try.

Bash History Cleaner

Sébastien Lavoie — Sun, 23 Dec 2018 19:35:00 -0600

Introduction

This is a Python 3.6+ script that helps to clean the file containing the Bash history commands. It will remove any line matching a specified regular expression and can also remove any line starting with an alias.

The idea behind this small utility was simple:

The Bash history file (usually located in ~/.bash_history) contains much of the work one ends up doing in the terminal.
The history can grow large over time and it becomes more cumbersome to find interesting information in all that clutter, such as a rarely used command with specific flags.
By removing all superfluous commands that are repeated often and which give no real benefit in certain contexts (such as ls, cd, cat, etc.), the history is much cleaner and easier to navigate and actually becomes much more useful in my opinion.
True, it will be harder to follow the bread crumbs for everything you did, but I haven’t come across a situation where having access to yet another empty ls or cd has proven necessary and reading .bash_history doesn’t make for a great narrative story either.

Make history in a big way

I took advantage of the fact that the history can be cleaned with the script you are about to see and set up what is known as an eternal history which, as it sounds like, can grow infinitely big! All you have to do is append the following lines to the file ~/.bashrc:

 bash
 
 
 
# Eternal bash history.
# ---------------------
# Undocumented feature which sets the size to "unlimited".
# http://stackoverflow.com/questions/9457233/unlimited-bash-history
export HISTFILESIZE=-1
export HISTSIZE=-1
export HISTTIMEFORMAT="[%F %T] "
# Change the file location because certain bash sessions truncate
# .bash_history file upon close.
# http://superuser.com/questions/575479/bash-history-truncated-to-500-lines-on-each-login
export HISTFILE=~/.bash_eternal_history
# Force prompt to write history after every command.
# http://superuser.com/questions/20900/bash-history-loss
PROMPT_COMMAND="history -a; $PROMPT_COMMAND"

Bash History Cleaner

And here is the script in question *. It comes in two files that need to be in the same directory:

One is a Python file that needs to be launched from the terminal with Python 3.
The other file, settings.json, is a JSON file used to store the settings of the script, which will be detailed below.

* _{Improvements to the original script can be found on Github. To keep this
article a bit more readable, the original version is shown.}

`bash_history_cleaner.py`

 python
 
 
 
'''
Python script that helps to clean the file containing the Bash history
commands.

Note: Requires Python 3.6+

It will remove any line matching a specified regular expression and can
also remove any line starting with an alias.

Description of available settings in `settings.json`:

 "home_directory": Absolute path to user's home directory.

 "history_file": Name of file where the history will be cleaned up.

 "aliases_file": Name of file where Bash aliases are set up.

 "ignore_patterns": List of patterns to ignore in `history_file`.
 Each line where a pattern is found will be deleted.
 → Patterns are specified as regular expressions.

 "add_aliases": Boolean. If set to `true`, aliases from `aliases_file`
 will be added to `ignore_patterns`.

 "aliases_match_greedily":
 Boolean. If set to `true`, any line in `history_file`
 starting with an alias in `aliases_file` will be
 deleted. If set to `false`, delete line if the alias is
 the content of the whole line (with optional space at
 the end): `false` matches "^alias$" or "^alias $" only.

 "backup_history": Boolean. If set to `true`, `history_file` will be backed
 up in the same directory with a name ending in .bak
 based on the current date.

 "delete_logs_without_confirming":
 Boolean. If set to `true`, script with flag `-c` will
 automatically delete all the backup files found for
 `history_file`.
'''
from datetime import datetime
from pathlib import Path
import argparse
import fileinput
import glob
import json
import os
import re


def get_current_path():
 '''Returns the current working directory relative to where this script
 is being executed.'''
 return Path(__file__).parents[0]


def user_says_yes(message=""):
 '''Check if user input is either 'y' or 'n'. Returns a boolean.'''

 while True:
 choice = input(message).lower()
 if choice == 'y':
 choice = True
 break
 elif choice == 'n':
 choice = False
 break
 else:
 print("Please enter either 'y' or 'n'.")

 return choice


def delete_logs(settings: dict, history_file: str):
 '''Delete log files in `home_directory` based on `history_file`.'''

 # Retrieve a list of all matching log files
 log_files = glob.glob(f'{history_file}_*.bak')

 if log_files == []:
 print("There is no log file to delete.")
 else:
 print(f'Log files found in {settings["home_directory"]}:')

 for log_file in log_files:
 print(log_file)

 if settings['delete_logs_without_confirming']:
 for log_file in log_files:
 os.remove(log_file)
 print('Log files deleted.')
 return

 message = ("\nDo you want to delete those log files? [y/n] ")

 if user_says_yes(message=message):
 for log_file in log_files:
 os.remove(log_file)
 print('Log files deleted.')
 return

 print('Operation aborted.')
 return


def generate_date_string() -> str:
 '''Return date formatted string to backup a file.'''
 return datetime.strftime(datetime.today(), '_%Y%m%d_%H%M%S.bak')


def load_settings(settings_file: str) -> dict:
 '''Load settings in the script. Return them as a dictionary.'''
 with open(settings_file, "r") as read_file:
 settings = json.load(read_file)
 return settings


def get_list_aliases(bash_aliases_file: str, settings: dict) -> list:
 '''Retrieve the name of all the aliases specified in `bash_aliases_file`.

 Return aliases as a list of strings formatted as regular expressions.'''

 match_whole_line = bool(settings['aliases_match_greedily'])

 with open(bash_aliases_file) as file:
 content = file.read().splitlines() # one alias per line
 aliases_list = []
 for line in content:
 try:
 # Get the actual alias in each line
 # Use negative lookbehind to remove 'alias ' at the beginning.
 # Matches anything after that is a dot, digit, underscore
 # or letter (will stop at the equal sign: alias blah='...')
 alias = re.search(r'(?<!not )alias ([\.\w]*)', line).group(1)

 # If for some reason the alias cannot be extracted, skip it
 # If search doesn't match, it's of type None and won't work
 except AttributeError:
 continue

 if match_whole_line:
 # Match the whole line if it starts with the alias.
 alias = f'^{alias}( )?$|^{alias} .*'
 else:
 # Will match only when alias is the whole content of the line,
 # followed by optional space.
 alias = f'^{alias}( )?$'

 # Escape dots in alias
 alias = alias.translate(str.maketrans({".": r"\."}))

 aliases_list.append(alias)

 return aliases_list


def clean_bash_history(settings: dict, history_file: str):
 '''Modify in place `history_file` by removing every line where
 `ignore_patterns` is found.

 Optionally, add a list of aliases to `ignore_patterns` with
 `aliases` based on the value of `add_aliases` in settings.json.'''

 if settings['backup_history']:
 backup_str = generate_date_string()
 file_input = fileinput.FileInput(history_file,
 inplace=True,
 backup=backup_str)
 else:
 file_input = fileinput.FileInput(history_file,
 inplace=True)

 with file_input as file:
 for line in file:
 has_match = False
 for pattern in settings['ignore_patterns']:
 matched = re.compile(pattern).search
 if matched(line):
 has_match = True
 break
 # If no match is found (nothing to ignore), print the line
 # back into the file. Otherwise, it will be empty.
 if not has_match:
 print(line, end='') # Line already has carriage return


def launch_cleanup(settings: dict, history_file: str, aliases_file: str):
 '''Main function that launches the cleanup process.'''

 bash_aliases = None
 if settings['add_aliases']:
 try:
 bash_aliases = get_list_aliases(aliases_file, settings)

 # add aliases to list of patterns to ignore
 settings['ignore_patterns'].extend(bash_aliases)
 except FileNotFoundError:
 print(f"File not found: {aliases_file}")
 quit()
 try:
 clean_bash_history(settings, history_file)
 except FileNotFoundError:
 print(f"File not found: {history_file}")


if __name__ == '__main__':
 SETTINGS_FILE_PATH = get_current_path() / 'settings.json'
 SETTINGS = load_settings(SETTINGS_FILE_PATH)
 ALIASES_FILE = SETTINGS['home_directory'] + '/' + SETTINGS['aliases_file']
 HISTORY_FILE = SETTINGS['home_directory'] + '/' + SETTINGS['history_file']

 # initiate the parser to check all the arguments passed to the script
 PARSER = argparse.ArgumentParser()
 PARSER.add_argument(
 '-c', '--clear', help='Delete all log files', action='store_true')

 # read arguments from the command line
 ARGUMENTS = PARSER.parse_args()

 if ARGUMENTS.clear:
 delete_logs(SETTINGS, HISTORY_FILE)
 quit()

 launch_cleanup(SETTINGS, HISTORY_FILE, ALIASES_FILE)

`settings.json`

 json
 
 
 
{
 "home_directory": "/home/sglavoie",
 "history_file": ".bash_eternal_history",
 "add_aliases": true,
 "aliases_file": ".bash_aliases",
 "aliases_match_greedily": true,
 "backup_history": true,
 "delete_logs_without_confirming": false,
 "ignore_patterns": [
 "^\\#\\d+",
 "^$",
 "^(\\.\\/)?pip$",
 "^(\\.\\/)?python.*$",
 "^\\.\\.$",
 "^alias",
 "^cd ",
 "^cd$",
 "^cd..$",
 "^fg$",
 "^df( )?",
 "^du( )?",
 "^exit$",
 "^git branch",
 "^git checkout master",
 "^git log$",
 "^git push$",
 "^git status$",
 "^git stauts$",
 "^kill \\d+.*",
 "^ls -a$",
 "^ls$",
 "^make|make install$",
 "^man ",
 "^pelican( )?$",
 "^pip install -r requirements.txt",
 "^pip.* list|pip.* show$",
 "^source ",
 "^sudo apt-get autoclean|sudo apt-get autoremove$",
 "^sudo apt-get dist-upgrade|sudo apt-get update$",
 "^which "
 ]
}

Description of available settings in `settings.json`

Name of setting	Description
`home_directory`	Absolute path to user’s home directory.
`history_file`	Name of file where the history will be cleaned up.
`aliases_file`	Name of file where Bash aliases are set up.
`ignore_patterns`	List of patterns to ignore in `history_file`. Each line where a pattern is found will be deleted. Patterns are specified as regular expressions.
`add_aliases`	Boolean. If set to `true`, aliases from `aliases_file` will be added to `ignore_patterns`. (Default: `true`)
`aliases_match_greedily`	Boolean. If set to `true`, any line in `history_file` starting with an alias in `aliases_file` will be deleted. If set to `false`, delete line if the alias is the content of the whole line (with optional space at the end): `false` matches “^alias$” or “^alias $” only.
`backup_history`	Boolean. If set to `true`, `history_file` will be backed up in the same directory with a name ending in .bak based on the current date. (Default: `true`)
`delete_logs_without_confirming`	Boolean. If set to `true`, script with flag `-c` will automatically delete all the backup files found for `history_file`. (Default: `false`)

Anecdotal evidence of satisfying performances

Performance-wise, this scans ~8,300 lines per second on my modest Intel Core i5 laptop with files of over 200,000 lines long. Not that I type so much stuff in the terminal: I just duplicated many lines.

Conclusion

This is a simple solution to an nonexistent problem, but it was in the end very instructive to me nonetheless. You may even find a use for it! Otherwise, you might use the same functions for other files such as logs! If you would like to take a closer look at the source in a more convenient way, you can find the code available on Github.

Aliases: Also Known as Terminal User's Best Friends

Sébastien Lavoie — Sat, 08 Dec 2018 13:52:00 -0600

Some aliases that I find useful

In all the following examples, the dollar sign $ at the beginning of commands represents the prompt, it should not be typed.

General-purpose

List all aliases

bash

$ alias

Prints all of your aliases. If you have plenty of those, you might prefer to pipe this command with less to get a nice pager that allows you to easily nagivate them:

 bash
 
$ alias | less

Setting an alias in `~/.bash_aliases`

 bash
 
alias c='clear'

This would set c as a shortcut to clear. In order to use the newly created alias, you would have to close the terminal or type the following command in the terminal:

bash

$ source ~/.bash_aliases

Clear the screen

 bash
 
alias c='clear'

I like being absorbed into the emptiness of the terminal, so this one comes in handy to tidy up the work space.

Reset the terminal

 bash
 
alias re='tput reset'

If for some reasons the terminal displays badly or display funky characters, it can usually be reset this way.

Exit from the terminal

 bash
 
alias q='exit'

For exiting the terminal as if you were still in Vim!

Display all file extensions recursively from the current directory

 bash
 
alias allextensions="find . -type f -name '*.*' | sed 's|.*\.||' | sort -u"

It comes in handy to spot if a file shouldn’t be there or to check for lower or uppercase extensions.

Find text inside files, including filenames

 bash
 
alias findinfiles='ag --nobreak --nonumbers --noheading . | fzf'

In this example, the alias is set up with ag, a fast code-searching tool and fzf, a great fuzzy finder.

Reboot and shutdown the system

 bash
 
alias reboot='systemctl reboot'
alias shutdown='systemctl poweroff'

This will reboot or power off the system without needing root privileges in most working conditions.

List and sort files and directories by modification time

 bash
 
alias treeold='tree -hDF | less'

This requires the command tree to be installed. The parameters are (descriptions taken from man tree):

-h: Print the size of files in a human readable way.

-D: Print the date of the last modification time.

-F: Append a ‘/’ for directories, a ‘=’ for socket files, a ‘\*’ for executable files, a ‘>’ for doors (Solaris) and a ‘|’ for FIFO’s, as per ls -F.

Open files quickly with default applications

 bash
 
alias o='xdg-open'

This will open files and URLs specified as argument in the default application detected.

Moving around

If you are going to be working on projects for some time and require to cd into them, I have found the following to be useful:

 bash
 
 
 
$ cdnameOfProject # instead of having to do cd /path/to/project/
$ # More examples:
$ cdcodeabbey # For codeabbey.com
$ cdhackerrank # For hackerrank.com
$ cdeuler # For projecteuler.net
$ cdgit # For all Github repositories

This is not advanced by any means, but it helps quite regularly! Since you can take advantage of tab completion, you can type cd (without adding a space) and then press TAB key to autocomplete the aliases.

 bash
 
alias ...='cd ../..'

Make it easier to navigate into deep directory structures by basically doing cd .. && cd .. to go back up two directories at once.

Listing files

 bash
 
 
 
alias l='ls -CFh'
alias la='ls -Ah'
alias ll='ls -ahlF'
alias ls='ls --color=auto'

Different ways to set up the command ls to quickly see the needed files. The parameters are (descriptions taken from man ls):

-A: Do not list implied . and ...

-C: List entries by columns.

-F: Append indicator (one of \*/=>@|) to entries.

-a: Do not ignore entries starting with .

-h: With -l and/or -s, print human readable sizes (e.g., 1K 234M 2G).

-l: Use a long listing format.

 bash
 
alias aNameOfProject='source /path/to/project/bin/activate'
alias aa='source ~/Programming/anaconda3/bin/activate' # Example using Anaconda distribution

The purpose is to activate a specific virtual environment quickly. The command cd could be added to go to the related project also:

 bash
 
alias gNameOfProject='cd /path/to/project/ && ./.venv/bin/activate'

Where .venv would be the name of the virtual environment.

I like to activate environments starting aliases with a and go and activate at the same time starting aliases with g. This way, it feels like I am speaking the Vim language (cw for change word, for example).

 bash
 
alias da='deactivate'

This will deactivate a virtual environment.

 bash
 
alias p36='python3.6'
alias p='python3.7' # Simply using `p` for main version of Python

Practical way to quickly open the desired Python version.

 bash
 
alias pyclean='find . -regex ".*\(__pycache__\|\.py[co]\)" -delete'

This will delete recursively all files and directories that match one of the following patterns in their name: __pycache__, .pyc or .pyo.

Conclusion

I hope you will find at least one alias to improve your productivity. Of course, you are welcome to chime in with your own suggestions!

Tmux Integration with Vim/Neovim — First Steps

Sébastien Lavoie — Mon, 19 Nov 2018 20:31:00 -0600

tmux integration

tmux offers many advantages in the context of remote access to another machine, but it also shines on a local setup! Here is how I currently like to set it up.

Configuration file

 ~/.tmux.conf
 bash
 
 
 
# split panes using | and -
bind | split-window -h
bind - split-window -v
unbind '"'
unbind %

# switch panes using Alt-arrow without prefix
bind -n M-Left select-pane -L
bind -n M-Right select-pane -R
bind -n M-Up select-pane -U
bind -n M-Down select-pane -D


# Enable mouse mode (tmux 2.1 and above)
set -g mouse on

source-file "${HOME}/.tmux-themepack/blue.tmuxtheme"

This is all very self-explanatory. Many themes can be found here.

Automating the launch of a default session

This is a small Bash script that I like to have for automatic set up of a development environment with tmux and Vim/Neovim. As my current workflow, I enjoy the following:

 ~/.tmux_default_session.sh
 bash
 
 
 
#!bash
# var for session name (to avoid repeated occurences)
sn=dev

tmux new-session -s "$sn" -d

# Create 3 windows
tmux new-window -t "$sn:0" -n "nvim"
tmux new-window -t "$sn:9" -n "python"
tmux new-window -t "$sn:8" -n "terminal"

# Split terminal window vertically, then split the right pane
# horizontally, then switch to the left pane (identified by `FOCUS IS
# HERE`).
# _________________
# |>_ |>_ |
# | | |
# | FOCUS |-------|
# | IS |>_ |
# | HERE | |
# -----------------
tmux split-window -h
tmux split-window -v
tmux select-pane -L

# Set up Neovim ready to open files
tmux send-keys -t "$sn:0" C-z 'nvim .' Enter

# Set up alias for IPython and clear the screen when entering IPython
tmux send-keys -t "$sn:9" C-z 'ipython' Enter
tmux send-keys -t "$sn:9" 'clear' Enter

# Select window #0 and attach to the session
tmux select-window -t "$sn:0"
tmux -2 attach-session -t "$sn"

Aliases to make use of

I add the following aliases in ~/.bash_aliases to automate a chunk of the workflow:

 ~/.bash_aliases
 bash
 
 
 
# This will launch tmux with the desired configuration
alias dev='bash ~/.tmux_default_session.sh'

# This will kill the tmux server if the need arises. By detaching from
# the tmux session, you simply run this command and can reattach easily
# to the default session configuration on another project.
alias kdev='pkill tmux'

# If the session has been detached but the tmux server is still running,
# I use the following alias to quickly reattach to the default session
# named `dev` in that example.
alias adev='tmux attach-session -t dev'

tmux demo

(click to open image)

The content of this post can be found on GitHub.

sglavoie.com – Tag: Terminal

Git worktrees for a better parallel workflow

Introduction

Why is git stash not enough?

Switching context

Running tasks in the background

How I operate worktrees

First, other approaches in the wild

What I actually do these days

Starting with a project with existing branches

New worktrees from a project already using worktrees

Keeping worktrees around

Conclusion

Resources and references

External links

From this website

A retrospective on creating an impractical little tool just for fun

Introduction

Setting up the stage

Vim magic behind the scenes

What went well

What could have been done better

Key lessons learnt and where to take it from here

Conclusion

Resources and references

Git the gist of it: common commands for a working workflow

Introduction

Global and local .gitignore

Meaningful commit messages with .gitmessage

My usual Git workflow

Starting a new project

Committing to an existing project

Being committed

rebase

Sample workflow

merge

Sample workflow (kinda)

Managing branches

Aliases

Custom Git sub-commands

Some lesser-known yet incredibly useful commands

git bisect

git rerere

A word on the configuration file

Conclusion

More resources and references

Websites

YouTube

Online courses

References for this article

Using Google Sheets as a database to extract data converted to Markdown

Introduction

General software requirements

Input

Output

Installing this tool

Using pip

Using pipenv

How to use this tool

Conclusion

Convenient and lightning fast backups with rsync

Introduction

Prerequisites

Setting up the script

The configuration file

Description of available settings

The script

How to use

One-time setup

Using as a daily driver

Avoid deleting files at the destination

Conclusion

More resources and references

Managing dotfiles with a Git bare repository

Introduction

1. Set up the system

1.1 Create a repository to host your dotfiles

1.2 Set the remote to track your dotfiles

2. Use the new system

2.1 On your current machine

Why is `git stash` not enough?

Global and local `.gitignore`

Meaningful commit messages with `.gitmessage`

`rebase`

`merge`

`git bisect`

`git rerere`

Using `pip`

Using `pipenv`

`bash_history_cleaner.py`

`settings.json`

Description of available settings in `settings.json`

Setting an alias in `~/.bash_aliases`