sglavoie.com – Tag: Script

Managing multiple tmux sessions at once

Sébastien Lavoie — Sun, 19 Sep 2021 18:13:00 -0500

Introduction

Released back in 2007, tmux keeps being amazing! Because I have never bothered automating anything to make its use simpler, it took some time for it to become a working environment I enjoy for everyday use. This is a thing of the past: thanks to Oleksandr Kocherhin from Monsterlessons Academy, who shared his tmux workflow in his YouTube video titled Best Terminal Application for Web Development, I saw a few automation opportunities I thought were worth tackling.

In this post, I’m sharing a little tip about managing tmux sessions that was inspired from the mentioned video. I am using i3wm on Linux, but you can adapt this with any desktop environment as long as you can trigger the execution of programs with a keyboard shortcut in one way or another. I will be using Rofi and the find command, which you can find on Unix systems. If you would rather use dmenu instead of Rofi with this script, this article can serve as a reference. Enough said, let’s get to it.

What is this all about?

The idea is simple: have a few keyboard shortcuts to manage tmux sessions. That means being able to open new sessions and being able to kill them too. I kept this setup very simple on purpose but of course some improvements could be made. For instance, it could be nice to know which projects have already been opened so that the script doesn’t prompt you with a list of projects that include projects that you have already launched, but I haven’t found the need so far to deal with this potential issue as I tend to only work on a couple of projects during the course of a given day.

In summary, here is what we want to achieve in this post:

Configure projects so that when we want to open them, they will be set up in the right directory with servers running, etc.;
Open a tmux session with a keyboard shortcut, listing all projects that were configured in the previous step;
Kill any opened tmux session with another keyboard shortcut.

That’s it!

1. Configuring projects to launch tmux sessions

Firstly, we need to create new files to open our projects in the way we want. Basically, we need to create a file, let’s call it .tmux, and put it at the root of the project we’re interested in working with tmux. Then, we need to put some instructions that will be run with the shell (e.g. Bash or Zsh) so that tmux commands will be run automatically for us and let us “attach” to the session once all the necessary actions are performed. That means we need to make that file executable. In the terminal, this would be done as follows, assuming the project’s folder is the current working directory:

bash

chmod +x .tmux

At last, we want to give some orders to tmux so that when we execute that file, it will open our project in a pre-configured state so we’re ready to go. This will need to be customized depending on your needs, but here is a simple example to illustrate what could happen. So, we could put the following inside the .tmux file:

 bash
 
 
 
#!/bin/sh

set -e

if tmux has-session -t=my-project 2> /dev/null; then
 tmux attach -t my-project
 exit
fi

tmux new-session -d -s my-project -n nvim -x $(tput cols) -y $(tput lines)

# Neovim
tmux send-keys -t my-project:nvim "n" Enter

# Servers
tmux new-window -t my-project -n servers
tmux send-keys -t my-project:servers "cd frontend && yarn start" Enter

tmux split-window -t 2 -v -p 50 # split it into two halves
tmux send-keys -t my-project:servers "cd backend && ap && uvicorn main:app --reload" Enter

tmux attach -t my-project:nvim

Here is what is happening:

If our project, named my-project, has a tmux session already running, then we “attach” to it and nothing more is done.
Otherwise, the session is created in “detached” mode so we can run a few commands smoothly and later attach to the created session.
With the -n flag, we name the window to be created (here, the first window is nvim).
Then, we “send” keys to the session, to that newly created nvim window. In this case, we send a single keystroke n, which is just an alias for executing Neovim.
In a new window named servers, we send keys to execute the frontend server.
Then, we split the second window we just created vertically and we run the commands needed to launch the backend server, again making use of Bash aliases for convenience.
Finally, we attach to the session and we open the first window nvim, ready to work.

And just like this, what takes quite a few steps to do manually can be automated in one go when setting up the environment at the beginning of a working bout. Now, it could be convenient to create a generic template for the above script to use when creating new projects, especially if those tend to be short-lived and share some properties (e.g. maybe they all need to launch some kind of server in the background).

2. Script to open projects

Firstly, we need a way to list and open all “known” projects. This is achieved in this example very simply with the find command as well as with the help of rofi. The script in question is as follows:

 bash
 
 
 
#!/bin/bash

# Get the path to the `.tmux` file to execute using `find` and `rofi`
PROJECT=`find $HOME/dev -type d \( -name node_modules -o -name .venv \
 -o -path name \) -prune -false -o -name '.tmux' \
 | rofi -i -dmenu -theme purple -font 'JetBrainsMono 14' \
 -width 90 -show window`

# Get the full directory path to the `.tmux` file, excluding the filename
PROJECT_DIR=`echo $PROJECT | xargs -I {} dirname "{}"`

# If $PROJECT is empty (e.g. pressing Esc at the prompt), don't do anything
if [ ! -z "$PROJECT" ]; then
 kitty --directory "$PROJECT_DIR" -e "$PROJECT"
fi

So, what this does is this:

It finds all the directories where a .tmux file exists, excluding non-useful folders like node_modules. The -prune flag is passed to avoid recursion further inside a directory once a .tmux file is found. The results of find are “piped” into rofi, which displays the results. User selection is then stored in the PROJECT variable.
Once a file is selected, the full path without the filename is extracted and stored in the PROJECT_DIR variable.
Finally, all that is left to do is check if the user selected a project and if so, run it with the desired terminal from the correct directory. This will be different for most terminal emulators: here, an example is shown with the kitty terminal.

3. Script to close projects

Here, we are concerned with the sessions that are now opened and that we might want to close. In a similar fashion as what we did to open projects, we want to kill them with a script that will run Rofi and do the job silently in the background. One such script that will get us there is shown below:

 bash
 
 
 
#!/bin/bash

# Get the name of all running sessions
TMUX_SESSIONS=`tmux ls | cut -f1 -d ":"`

# No listed session, then notify about it
if [ -z "$TMUX_SESSIONS" ]; then
 notify-send "No active tmux session found"
 exit 0
fi

# Show all running sessions with Rofi and store the selection in a variable
SESSION=`echo $TMUX_SESSIONS | rofi -sep ' ' -i -dmenu -theme purple -font 'JetBrainsMono 14' \
 -width 90 -show window`

# Kill the selected session
tmux kill-session -t $SESSION

In this case, the usage is quite straightforward: only running sessions are shown, so it’s just a matter of choosing the one we want to kill or pressing the Escape key to abort the Rofi pop-up.

Example workflow

It’s a matter of setting up keyboard shortcuts to run the scripts to open and kill projects. For instance, one such configuration for the i3 window manager would be as follows:

 bash
 
bindsym $mod+Mod1+p $exec ~/path/to/projects-start.sh
bindsym $mod+Mod1+k $exec ~/path/to/projects-kill.sh

A simple shortcut lets us choose sessions to open.
Next, if we decide to kill a session at some point, we have another shortcut to do that.

Conclusion

Far from being a perfect solution (although it already feels comfortable as a main driver), this workflow initiates what I think can lead to substantial optimizations when it comes to using tmux. There’s room for improvement when a user selects an already opened session: ideally, we would just switch to that session automatically. These details need to be implemented based on the window manager that’s being used, but suffice to say, it can be done! In any case, I hope this may give you a semblance of inspiration to take this idea to the next level. There are tools to help with session persistence, such as tmux-resurrect, but that’s a topic for another time.

More resources and references

Best Terminal Application for Web Development, original inspiration for this workflow. — Monsterlessons Academy, YouTube.
The tmux wiki, GitHub.
tmux-resurrect, GitHub.

Convenient and lightning fast backups with rsync

Sébastien Lavoie — Sat, 31 Jul 2021 09:01:00 -0500

Introduction

Coming into its fourth year of existence and usage in production environments, this simple script has proven to be a fantastic ally. Despite the plethora of great options out there that have more bells and whistles, I have yet to find a single complaint about this solution as it hinges on the minimalist side, requiring only a working Python 3 installation along with the powerful rsync command-line tool.

It’s one of those pieces of code that perform some really basic tasks that can be scheduled with cron in the background, that one forgets about until a real need to access a backup comes up. This script is barely a Python wrapper calling rsync but still, I felt like sharing it for those who like uncomplicated setups. I’ve literally left this script alone for years some time after the release of Python 3.6 which was released back in December 2016 and it has been going strong since then mostly due to the fact it does not do much more than backing files up and leveraging rsync.

Prerequisites

As briefly mentioned above, this tool uses Python 3.6+ and rsync. That’s it, really.

Setting up the script

This script is being made available on GitHub within a set of other small tools. To get the source code for this particular tool, you can either clone the whole repository, get the content of the files separately or simply copy and paste the current version from below. You will need a settings.json file as well as the Python script that will run the necessary commands.

The configuration file

Right off the bat, you will most probably want to update the array of strings for the data_sources key as well as the string for the key data_destination in the following JSON file.

 json
 
 
 
{
 "data_sources": [
 "/home/sglavoie"
 ],
 "data_destination": "/media/sglavoie/Elements",
 "terminal_width": 40,
 "sep": "=-",
 "log_name": ".backup_log_",
 "log_format": "%y%m%d_%H_%M_%S",
 "rsync_options": [
 "-vaAHh",
 "--delete",
 "--ignore-errors",
 "--force",
 "--prune-empty-dirs",
 "--delete-excluded"
 ],
 "backup_exclude": ".backup_exclude"
}

By default, the following options are passed to rsync:

Option	Description
`-vaAH`	verbose, archive, ACLs, hard-links (preserve)
`--delete`	“delete extraneous files from destination dirs”
`--ignore-errors`	“delete even if there are I/O errors”
`--force`	“force deletion of directories even if not empty”
`--prune-empty-dirs`	“prune empty directory chains from the file-list”
`--delete-excluded`	“also delete excluded files from destination dirs”

Description of available settings

Name of setting	Description
`data_sources`	Directories to backup, supplied as a list of strings (no slash at the end).
`data_destination`	Single destination of the files to backup, supplied as a string. This can be overridden when passing option `-d` or `--dest` to the script
`terminal_width`	Line length in the terminal, used for printing separators.
`sep`	Separator to use along with `terminal_width`.
`log_name`	Sets the prefix of the log filename.
`log_format`	This goes right after `log_name` as a suffix.
`rsync_options`	Options to use with rsync as a list of strings.
`backup_exclude`	Default file in each source in `data_sources` where files/directories will be ignored.

The script

 python
 
 
 
"""
Script that uses `rsync` to make a simple and convenient backup.
Note: requires Python 3.6+. No other Python third-party libraries required.
"""
import argparse
import datetime
import glob
import json
import os
import pathlib
import subprocess


def run_backup():
 """This is where all the action happens!"""
 settings = get_settings()

 # initiate the parser
 parser = argparse.ArgumentParser("backup")
 parser.add_argument(
 "-c",
 "--clear",
 help="Delete all log files for current source in DATA_SOURCES.",
 action="store_true",
 )
 parser.add_argument(
 "-s",
 "--src",
 dest="source",
 default=None,
 help="Specify an alternative source to backup as a string.",
 action="store",
 )
 parser.add_argument(
 "-d",
 "--dest",
 dest="destination",
 default=None,
 help="Specify an alternative destination for backup as a string.",
 action="store",
 )

 # read arguments from the command line
 arguments = parser.parse_args()

 # check for --source or -s
 # Replace potential list of sources to backup with this one
 if arguments.source is not None:
 if os.path.isdir(arguments.source):
 settings["data_sources"] = [arguments.source]
 else:
 print("Please enter a valid source to backup.")
 exit(0)

 # check for --clear or -c
 if arguments.clear:
 clear_logs(
 data_sources=settings["data_sources"],
 log_name=settings["log_name"],
 )

 # check for --dest or -d
 # Replace destination to backup with this one
 if arguments.destination is not None:
 if os.path.isdir(arguments.destination):
 settings["data_destination"] = arguments.destination
 else:
 print("Please enter a valid destination.")
 exit(0)

 # don't run the script if the destination doesn't exist
 if not os.path.isdir(settings["data_destination"]):
 print(
 f"The destination doesn't exist.\n({settings['data_destination']})"
 )
 exit(0)

 backup_all_sources(settings)


def backup_all_sources(settings: dict) -> None:
 """Iterate over all sources to backup."""
 for source in settings["data_sources"]:
 date_now = datetime.datetime.now()
 log_format = datetime.datetime.strftime(
 date_now, settings["log_format"]
 )
 log_filename = f"{source}/{settings['log_name']}{log_format}"
 log_option = f"--log-file={log_filename}"

 backup_source = settings["backup_cmd"].copy()
 backup_source.extend([log_option])

 # files to ignore in backup
 exclude_file = f"{source}/{settings['backup_exclude']}"

 if os.path.exists(exclude_file):
 exclude_option = f"--exclude-from={exclude_file}"
 backup_source.extend(
 [exclude_option, source, settings["data_destination"]]
 )
 else:
 # skips '--exclude-from' option if no file is found
 backup_source.extend([source, settings["data_destination"]])

 settings["source"] = source
 settings["backup_source"] = backup_source
 settings["log_filename"] = log_filename

 try:
 backing_source(settings)
 except KeyboardInterrupt:
 print("\nKeyboardInterrupt: Exiting operations.")
 exit(0)


def backing_source(settings: dict) -> None:
 """Print information to STDOUT and to `log_filename` and executes the
 rsync command."""
 print(settings["sep"] * settings["terminal_width"])

 cmd_executed = " ".join(settings["backup_source"])
 msg_executed = f"Command executed:\n{cmd_executed}\n"
 print(msg_executed)

 with open(settings["log_filename"], mode="w") as log_file:
 log_file.write(f"{msg_executed}\n")

 try:
 child = subprocess.Popen(settings["backup_source"])
 _ = child.communicate()[0] # call communicate to get the return code
 rc = child.returncode
 except FileNotFoundError:
 print(f"FileNotFoundError: Is the `rsync` tool installed?")
 exit(1)

 print(f"\nBackup completed for: {settings['source']} (return code: {rc})")
 print(settings["sep"] * settings["terminal_width"])


def clear_logs(data_sources: list, log_name: str) -> None:
 """Clears log files for each source specified in SETTINGS."""
 for source in data_sources:
 # Retrieve a list of all matching log files in `source`
 log_files = glob.glob(f"{source}/{log_name}*")
 if log_files == []:
 print(f"\nThere is no log file to delete in {source}.")
 exit(0)
 else:
 print(f"Log files in {source}:")
 for log_file in log_files:
 print(log_file)
 if user_says_yes():
 for log_file in log_files:
 os.remove(log_file)
 print("Log files deleted.")
 print("Exiting script...")
 exit(0)


def user_says_yes(
 msg: str = "\nDo you want to delete log files for this source? (y/n) ",
) -> bool:
 """Asks the user to enter either "y" or "n" to confirm. Returns boolean."""
 choice = None
 while choice is None:
 user_input = input(msg)
 if user_input.lower() == "y":
 choice = True
 elif user_input.lower() == "n":
 choice = False
 else:
 print('Please enter either "y" or "n".')
 return choice


def get_settings() -> dict:
 """
 Get the settings from `settings.json`.

 Returns:
 dict: Containing all settings used by the tool.
 """
 directory = pathlib.Path(__file__).parent.absolute()
 with open(directory / "settings.json") as fp:
 settings = json.load(fp)

 backup_cmd = ["rsync"]
 backup_cmd.extend(settings["rsync_options"])
 settings["backup_cmd"] = backup_cmd

 return settings


if __name__ == "__main__":
 run_backup()

How to use

One-time setup

Set all values in settings.json to suit your needs.
Make sure that the backup destination is available/mounted. A simple warning will be echoed if the destination can’t be found.
Call the Python script. As an example, if it’s located at /home/user/backup/rsync_backup.py, then you could put the following alias in ~/.bash_aliases (or equivalent)*:

 bash
 
alias backup='python3 /home/user/backup/rsync_backup.py'

* Python may be called differently on your system, e.g. simply python instead of python3.

In this instance, the script can now be executed in a terminal with the keyword backup along with optional arguments.

Using as a daily driver

Make sure the destination to back files up is available/mounted and simply call the script (e.g. backup in this example).

The output will look something like this:

 bash
 
 
 
$ backup
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Command executed:
rsync -vaAHh --delete --ignore-errors --force --prune-empty-dirs \
 --delete-excluded --log-file=/home/sglavoie/.backup_log_210731_09_28_09 \
 --exclude-from=/home/sglavoie/.backup_exclude /home/sglavoie /tmp

building file list ... done
deleting sglavoie/.backup_log_210731_08_47_24
sglavoie/
sglavoie/.bash_aliases
...

Avoid deleting files at the destination

The rsync command first builds a list of the files that are part of the backup and will go on adding new files, updating existing files as well as removing files that are found in the destination but not in the source anymore according to the default list of flags passed to it in this implementation. It is therefore important to note that if you would like to back up files in the destination that no longer exist in the source, a convenient solution is simply to create a new folder at the root level inside the destination directory and manage those files separately.

For instance, I sometimes want to back up large video files but there is rarely a need to update them. In this case, it would be possible to add the directories to exclude in the file .backup_exclude (default configuration option) like so:

Videos/
Dropbox/other/folder/

Then, these directories will be ignored when backing up the source (if they are nested inside the source directory to start with!). If the destination is set to /media/user/backup and the source set to /home/user/my_folder, then the script would recreate the “root” directory of the source if it doesn’t exist on the destination:

bash

$ ls /media/user/backup
my_folder

You could keep track of those heavy files that do not need to be backed up frequently by putting them manually next to my_folder in /media/user/backup. This will lead to huge speed ups when running the script if you track many files in this way!

Conclusion

This was a quick overview of what rsync can offer: I neither touched upon its remote syncing capabilities nor the vast majority of its options, for which you can find a lot more information about by typing man rsync to display the manual page in the terminal.

Without making use of any fancy features of recent versions of Python or rsync, I have found this simple backup procedure to have worked flawlessly for a long time. Hardware inevitably fails at some point and after having heard about countless examples of people losing some or all of their most precious files (including my own story…), I have come to enjoy backing my system with this little script.

I wouldn’t trust my external drive to last forever either, hence I also rely on Dropbox and Google Drive for documents I find are a good fit. But overall, I just do not tend to back up every single system file in the cloud, so there is still a convincing use case where daily and complete backups become a possibility when using a fast approach like rsync to incrementally save snapshots.

More resources and references

rsync official website – samba.org.
Keeping Linux files and directories in sync with rsync – RedHat, going through the main options offered by the tool.
How To Use Rsync to Sync Local and Remote Directories – DigitalOcean, demoing how to use the tool and covering the case of remote file access.

Using dmenu to Optimize Common Tasks

Sébastien Lavoie — Sun, 10 Nov 2019 09:02:00 -0600

Introduction

dmenu is one of those tools that look a little unimpressive at first but can accomplish so much! It’s a program that you can use to receive any output redirected from other programs (through pipes in the terminal, the symbol |) and treat that output so that it can pop up within a simple menu to make it available for execution. If you want to know more about other fantastic tools from suckless.org, I went over some of them before, such as the st terminal and slock, a dead simple screen locker.

Any Simple Example?

You bet! Before diving in with how to install it and some more concrete examples, you could give dmenu a go with a simple command such as the following (assuming the program is installed on your machine).

bash

ls | dmenu

This will effectively “pipe” the output of ls into dmenu and a menu like the following would appear (by default, it shows as a thin stripe at the top covering the width of the screen):

If you start typing, the displayed list will be filtered down like so:

You can also switch to the next item to be highlighted by typing Ctrl + n, to the previous item by typing Ctrl + p and exit by reaching for the Escape key. For now, if you type Enter once your desired selection is highlighted, the selection will be outputted to the terminal and that will be the end of it. Let’s convert this into a practical thing, then.

Keeping the same output as before which only contained directories, we could open a file manager (here, I’m going with pcmanfm) and do something like this (bear with me, the explanation follows!):

 bash
 
ls | dmenu -l 5 | xargs -I {} pcmanfm "{}"

This is now a fully functional example, albeit its practicality is debatable. What will happen, exactly?

First, ls will output a list of directories in the current working directory. That is, the following would be printed in the terminal without any piping:

Desktop Downloads Learning Pictures Public Templates virtualbox_vms
Documents Dropbox Music Programming SortOut Videos

When we pipe this into dmenu, we will get a menu with all that output nicely formatted vertically, which is what the -l 5 options does (here, we want only 5 lines to show). We will see a maximum of 5 items as this is the value we passed to the -l option:

We can still start typing a word that doesn’t show up in this list like Mu and we will see that this list is filtered to only include the matching directory Music even though we couldn’t see it before since we limited our results to 5 items. We could also use the same technique with Ctrl + n and Ctrl + p to cycle between the elements.

Up to this point, we are getting our selection outputted to the terminal. Next, we pipe it into xargs, which will allow us to specify a command to which we will pass our result (what is printed to the terminal) as an argument (we could make use of more arguments depending on the output we get, but let’s start simple). The -I {} option will make it possible to “quote” our result so it is read as a single argument and properly escaped. This means that without this option, if we have spaces or other special characters in the argument (such as a folder called My Music), it would try to pass each word as a new argument to pcmanfm, trying to open both My and Music directories at once.

We are representing our output as {} and when we quote it with pcmanfm "{}", spaces are interpreted without any trouble. If we wanted to use various arguments instead, we could try something like this:

 bash
 
echo "file1.txt file2.txt\nfile3.txt file4.txt" | dmenu -l 2 | xargs cat

The part before the first pipe character will print to the terminal those two lines:

file1.txt file2.txt
file3.txt file4.txt

When we pipe this into dmenu -l 2, we will get this menu:

Typing Enter on one of the highlighted items would then trigger xargs to pass the content literally to cat as arguments, which would then print to the terminal the content of both file1.txt and file2.txt (the first line containing two arguments for cat) or the content of both file3.txt and file4.txt (the second line also containing two arguments for cat) in succession.

Now that we got our feet wet with what dmenu does (displays a menu), we might want to install it.

How do You Install It?

In Manjaro Linux, it comes with i3 if you use that flavor of the distribution and can be opened by pressing the modifier key (either Super/Windows key or Alt key) along with the letter d, as in the shortcut mod + d, or by typing dmenu_run in a terminal, which will present a list of installed applications to launch. Otherwise:

Debian-based (such as Ubuntu)

bash

sudo apt install dmenu

Arch-based (such as Manjaro)

bash

pacman -Syu dmenu

You can also install it from your distribution’s package manager if available or from source.

Useful Real-World Examples

Here are a couple of ways I like to use dmenu to open a menu with a list of:

cheat sheets;
filesystem paths;
most used documents;
documents & books I read at the university;
custom scripts I want to run in the background.

Here are a few screenshots of how it currently looks like in my daily usage so you get a better idea.

Cheat sheets

University shortcuts

File manager shortcuts

So, how does that all work? As a starting point, all of those menus are launched through a specific keyboard shortcut that uses a mnemonic for each one (mod being the “modifier” key on i3, which is set to Super/Windows):

mod + Alt + c: cheat sheets;
mod + Alt + d: documents;
mod + Alt + f: file manager;
mod + Alt + s: scripts;
mod + Alt + u: university.

Let’s take the cheat sheets example, which is a bit more interesting since it launches different applications.

Setting Up a Custom Command

Configuration for i3 (bind a keyboard shortcut)

Here, I’m using i3 to set a keyboard shortcut to run a specific command, but this will be a similar experience on other window managers and desktop environments.

 bash
 
 
 
# i3config file
# The backslash at the end of the line allows us
# to split the line to increase readability

## Cheatsheets

bindsym $mod+Mod1+c exec --no-startup-id \
cat path/to/cheatsheets.conf \
| dmenu -l 30 | sed 's/.* \+//' | sh

You would usually be able to launch a custom command from your environment through a keyboard shortcut. If passing a direct command such as the above one isn’t an option, you can always store it in a Bash script and run that Bash script instead as the designated custom command. The script would then contain the following:

 bash
 
#!/bin/bash

## Cheatsheets
cat path/to/cheatsheets.conf \
| dmenu -l 30 | sed 's/.* \+//' | sh

As far as i3 is concerned, that’s all you need to do. You would need to reload the configuration file (by default: i3config) where the previous bindsym command has been set up to apply the changes (default shortcut to reload: mod + Shift + c).

Configuration for cheatsheets.conf

The content of this file is literally what will be shown when dmenu opens it. An excerpt:

 
#---------- Cheat Sheets -------------------------------------------------------
bash st -e nvim path/to/bash.sh
git zathura path/to/github.pdf
vimwiki firefox path/to/vimwiki.html

Here, we have set a maximum of 30 lines to be displayed (-l 30). What follows after an element has been selected with dmenu allows us to parse the content of the line and retrieve only the command we are interested in with sed before passing that filtered content around to sh to execute it as a shell command. With more complex commands requiring multiples arguments to be received, we could add one more pipe between sed and sh like this (or with xargs -I {} to avoid problems with spaces):

 bash
 
cat path/to/cheatsheets.conf \
| dmenu -l 30 | sed 's/.* \+//' | xargs -r | sh

If the line we want to run only includes a path to a file as in:

myshortcut path/to/file.pdf

We could instead pipe it into an external command of choice if the same program applies to all items presented in the menu, say Firefox:

 bash
 
cat path/to/file.conf \
| dmenu -l 30 | sed 's/.* \+//' | xargs -I {} firefox "{}"

What’s nice with the way pipes work in a Unix-like system is that we can chain harmless commands until the very end so they will simply be printed to the terminal. In this example, if we want to see how to filter our lines before running a command, we can do so as follow.

Pipe 1

 bash
 
cat path/to/cheatsheets.conf \
| dmenu -l 30

If you remember from the menu we saw earlier, there’s a shortcut to open a cheat sheet for Git. Let’s say we selected that one with dmenu. Because our command isn’t doing anything with the result, it will be outputted to the terminal like so (just like in the excerpt of cheatsheets.conf):

bash

git zathura path/to/github.pdf

What sed does here is to cut everything from the beginning of the line up to where the command starts (that’s where we find zathura, which is a powerful document viewer), because we need to isolate that part of the line so it looks like a shell command we could run on its own. Let’s add a new pipe to our command and see what sed does with it*.

* _{We could have used other tools to parse the string like awk, grep, cut, etc.}

Pipe 2

 bash
 
cat path/to/cheatsheets.conf \
| dmenu -l 30 | sed 's/.* \+//'

If we select the same item as before with dmenu to illustrate more clearly what this new addition does, we will get this output:

bash

zathura path/to/github.pdf

If we were to type this in the terminal (assuming the file exists and zathura is installed!), that would do the trick and it would open with the specified document viewer. Pipe 2 and subsequent pipes before we execute something is where the filtering magic happen. In short:

Everything before | dmenu is how our input will look like when we run dmenu;
Everything after | dmenu [options here] | is what we need to do to our input so that it is converted into a working shell command;
Finally, what comes after the last pipe (either xargs, sh or a combination of both) is our way to redirect our string to the shell so it can be executed.

Once we are visually satisfied with how our command is supposed to look like (it has to be something that works when typed directly in the terminal), the next step is to pipe it again so it can be executed.

Pipe 3

 bash
 
cat path/to/cheatsheets.conf \
| dmenu -l 30 | sed 's/.* \+//' | sh

There we go, the command is launched. If we keep the same simple syntax in all of our .conf files —or whatever extension we choose— where the content of our menus is stored in plain text, we can quickly and painlessly create keyboard shortcuts to run custom lists of commands that can be edited on the fly. Once our shortcuts are active, it’s only a matter of editing one of those .conf files and automatically our lists will be up-to-date when we trigger the shortcut again.

Running a Custom Script

If we want to run custom scripts, we need to indicate a command that would work in the terminal, such as python myscrypt.py or ./myscript.sh. When it comes to shell scripts, we have to make sure they are executable. We can do so through a file manager or within the terminal like this:

bash

chmod +x path/to/script.sh

This will add the necessary permissions for the user to execute the script. As we already know, we would then need to add a keyboard shortcut to launch our custom menu and store our command in a file like scripts.conf that would contain the following:

#---------- scripts.conf -------------------------------------------------------
myscript script.sh

And that’s all there is to know to get piping with dmenu! You may also find this video from Luke Smith on YouTube to be quite helpful as well, which is where the inspiration for this post came from. He also posted a complementary video about adding prompts to your commands which is a nice way to add interactivity to your scripts!

Chaining dmenu prompts

Another useful scenario is when you want to take a specific action based on the output of a previous command. For instance, let’s say you want to read a book but when you choose a pdf with dmenu, you want to be prompted for which reader to use while you want to let the system choose the default application for other types of files (e.g. epub or mobi). This could be achieved with a script similar to the following one:

 bash
 
 
 
#!/bin/bash
FILE=`find ~/Documents/calibre_library -type f -iname "*.pdf" -o \
 -iname "*.epub" -o -iname "*.mobi" | dmenu -l 30`

if [[ "$FILE" == *.pdf ]]
then
 READER=`echo -e "zathura\natril" | dmenu -i -p "Which reader?"`
 $READER "$FILE"
else
 xdg-open "$FILE"
fi

Here’s what’s happening:

We find all the files (-type f)
in the directory ~/Documents/calibre_library
that match a pattern that’s case-insensitive with -iname (here, ending in either pdf, mobi or epub)
matching one file at a time (-o can be interpreted to mean “only” and will keep searching if the previous file extension was not matched)
then presenting 30 lines (-l 30) of results at a time with dmenu
then, if the file is a PDF, prompts whether to use atril or zathura as the file reader and open with the chosen program
otherwise, open the file with the default application.

Customize the Look of dmenu

As they say, “beauty is in the eye of the beholder”. If you would rather make some changes to how dmenu look, you can. dmenu will be searching for the configuration file located at ~/.dmenurc, which could contain something as put below (with this configuration, it will look like the screenshots shown previously):

 bash
 
 
 
#
# ~/.dmenurc
#

# define the font for dmenu to be used
DMENU_FN="NotoSans-10.5"

# background colour for unselected menu-items
DMENU_NB="#161925"

# textcolour for unselected menu-items
DMENU_NF="#fdfffc"

# background colour for selected menu-items
DMENU_SB="#235789"

# textcolour for selected menu-items
DMENU_SF="#fdfffc"

# command for the terminal application to be used:
TERMINAL_CMD="st -e"

# export our variables
DMENU_OPTIONS="-fn $DMENU_FN -nb $DMENU_NB -nf $DMENU_NF -sf $DMENU_SF -sb $DMENU_SB"

Conclusion

Hopefully this introduction to what dmenu has to offer gave you some ideas. I hope you’ll find many ways to adapt the examples so you can benefit from this amazing tool. You can find more configuration details in my dotfiles on GitHub for anything related to i3, dmenu, zathura, st, slock and many more useful programs.

Have a good time automating your digital life!

Exponentiation in Modular Arithmetic Made Easy

Sébastien Lavoie — Sat, 27 Apr 2019 19:52:00 -0500

Introduction

The following Python script will by no means provide any useful idea to compete against the RSA algorithm , but it does give you an idea of how a simple technique about reducing the size of an exponent in modular arithmetic can bring you closer to using much larger numbers than the ones you could normally use in the Python interpreter.

The script in action

 python
 
 
 
"""Compute the result of a^b (mod k) by using the exponentiation technique.
The goal here is not efficiency, even though the program is actually pretty
fast: the algorithm is applied manually for demonstration purposes.
Testing on a modest Intel Core i5, having `a` and `b` each set to a random
number containing 2,000 digits and `k` set to a modulo of a number containing
20 digits, results are printed in about 3.3 seconds."""

def binary_remainders(num_b):
 """Take `b` and return the binary equivalent in a list of remainders."""
 remainders = []
 quotient = num_b
 while True:
 prev_quotient = quotient
 quotient //= 2
 remainder = prev_quotient % 2
 remainders.append(remainder)

 if quotient == 0:
 break

 return remainders


def powers_of_two(remainders=None):
 """Return a list of the value of powers of two that form the
 exponent`b`."""
 if remainders is None:
 return None

 powers = []
 for index, remainder in enumerate(remainders):
 if remainder == 1:
 powers.append(2**index)

 return powers


def compute_intermediate_congruences(num_a, num_k, powers=None):
 """Compute all necessary intermediate results of congruence in `mod k` for
 powers of 2 in `powers` to form the number `b`."""
 if powers is None:
 return None

 go_up_to = max(powers)

 intermediate_results = {1: num_a} # Build dictionary to store all results
 start_value = 2 # First power of two to calculate congruence
 congruence = num_a
 while start_value <= go_up_to:
 # value to use for next power of 2
 congruence = congruence ** 2 % num_k
 intermediate_results[start_value] = congruence
 start_value *= 2

 return intermediate_results


def compute_final_congruence(num_k, powers=None, intermediate_results=None):
 """Take all relevant values from `intermediate_results` matching powers in
 `powers`, multiply them together and calculate this number `mod k` to get
 the final result."""
 if intermediate_results is None or powers is None:
 return None

 # store all required values from `intermediate_results`
 congruent_results = []
 for power in powers:
 for key, value in intermediate_results.items():
 if key == power:
 congruent_results.append(value)

 total = 1
 for result in congruent_results:
 total = result * total # Multiply all results together

 return total % num_k # final congruence we are looking for


def compute_congruence(num_a, num_b, num_k):
 """Return `c`, the result of `a^b (mod k)`."""
 num_a = num_a % num_k # Make sure `a` is smaller than `k`

 # Reduce `b` to list of remainders in binary
 remainders = binary_remainders(num_b)

 # Build a list of the powers of 2 forming `b`
 powers = powers_of_two(remainders)

 # Build a list of necessary intermediate results to reach
 # the value of `b` from powers of 2: finds congruence for
 # smaller powers of 2 and store them in a list.
 intermediate_results = compute_intermediate_congruences(num_a, num_k,
 powers)

 # Multiply all relevant intermediate results `mod k` to get the final
 # congruence of `a^b (mod k)`.
 return compute_final_congruence(num_k, powers, intermediate_results)


if __name__ == '__main__':
 print("We will calculate a^b (mod k). Enter only integers.")
 NUM_A = int(input("Provide `a`: "))
 NUM_B = int(input("Provide `b`: "))
 NUM_K = int(input("Provide `k`: "))

 FINAL_RESULT = compute_congruence(NUM_A, NUM_B, NUM_K)
 print(FINAL_RESULT)

Conclusion

It was very interesting to see how a technique that’s applied manually will work wonders with such large numbers. The Python interpreter can barely calculate numbers with exponents with seven digits or more, while a basic approach like the one shown above can quickly churn out the results for seemingly quite large numbers.

As always, this code is available on GitHub.

Have You Been Pwned?

Sébastien Lavoie — Thu, 21 Mar 2019 13:42:00 -0600

Introduction

Have I Been Pwned? has a very simple and accessible API that’s perfect to work with as a beginner. There are different ways to retrieve information and the script below is only one glimpse into the numerous possibilities available. It is built in a way that makes it easy to add features and functionality thanks to Click, a wonderful “command line interface toolkit”.

Python to the rescue

The following script uses Click to build a tiny command-line interface instead of more traditional tools like argparse. You will also need requests to peek inside the API. Both can be obtained as follow in a terminal:

bash

$ pip install click requests

And now, the pièce de résistance.

 python
 
 
 
"""
Simple script that will take advantage of haveibeenpwned.com's API to
find out if your password has been breached.
"""
# Standard library imports
import hashlib

# Third party library imports
import click
import requests


# Needed to access API
PASS_URL = 'https://api.pwnedpasswords.com/range/'

# Script information
VERSION = '0.1.1'
LOGO = r'''
 __ __ __
| |--.---.-.--.--.-----. |__| | |--.-----.-----.-----.
| | _ | | | -__| | | | _ | -__| -__| |
|__|__|___._|\___/|_____| |__| |_____|_____|_____|__|__|

 __ _____
.-----.--.--.--.-----.-----.--| ||__ |
| _ | | | | | -__| _ |', ,-'
| __|________|__|__|_____|_____| |--|
|__| '--' '''


def print_info(context, param, value): # `param` required by `click`
 """Print information about the program and exit."""
 if not value or context.resilient_parsing:
 return
 click.secho(f'{LOGO}\n{"Python 3.6+ Checker".rjust(60)}\n'
 f'{("v" + VERSION).rjust(60)}', fg='blue', bold=True)
 context.exit()


def pass_to_sha1(password):
 """Will convert `password` with SHA-1 algorithm and return the short
 version with the first five characters and the long version with
 everything but the first five characters as a tuple."""

 my_password = password.encode() # converts to byte string

 complete_hash = hashlib.sha1(my_password).hexdigest().upper()
 long_hash = complete_hash[5:]
 short_hash = complete_hash[:5]

 return short_hash, long_hash


def check_password(short_hash, long_hash):
 """Consult haveibeenpwned.com to see how many times the password has
 been breached."""
 click.secho('Please note that only the 5 first characters from your '
 'SHA-1 ENCRYPTED password are sent to haveibeenpwned.com.',
 fg='yellow', bold=True)
 response = requests.get(PASS_URL + short_hash)
 lines = response.text.split()
 for line in lines:
 if long_hash in line:
 num_times = 'occurrences' if int(line[36:]) > 1 else 'occurence'
 click.secho('Your password was found!', fg='red', bold=True)
 click.echo(f'→ {line[36:]} {num_times}')
 break
 else:
 click.secho('Your password was NOT found!', fg='green')


@click.group()
@click.option(
 '--info',
 is_flag=True,
 callback=print_info,
 expose_value=False,
 is_eager=True,
 help="Print some information about the program and exit.")
@click.version_option(version=VERSION, message="%(version)s")
def hibp():
 """A simple command-line interface to make use of
 haveibeenpwned.com's data.

 Type `python3 pwned.py usage` for more info (or whatever way you
 call Python 3)."""


@hibp.command()
def usage():
 """Give examples on how to use this script."""
 click.secho('Examples:', fg='blue')
 click.secho('python3 pwned.py check --help for more info.', fg='green')
 click.secho('python3 pwned.py check -p MyPasswordHere', fg='yellow')
 click.secho('python3 pwned.py check --password MyPasswordHere', fg='green')

 click.echo('')
 click.secho('For more complicated passwords, you have to use quotes and ',
 fg='blue')
 click.secho('escape symbols with \\ where appropriate:', fg='blue')
 click.secho('python3 pwned.py check -p "as0d9\\"asg0\'\'A=)SYD"',
 fg='green')


@hibp.command()
@click.option(
 '-p',
 '--password',
 default=None,
 help="Reveal if match exists with `password`.")
def check(password):
 """Tell if your password has been breached."""
 if password:
 short_hash, long_hash = pass_to_sha1(password)
 check_password(short_hash, long_hash)
 if not password:
 click.secho('Run `pwned.py check --help` for more info.', fg='red')


if __name__ == '__main__':
 hibp()

Example usage

Conclusion

If you are familiar with Python, you can easily make sense of how this little tool works by reading its code. You can also have a look at this repository on GitHub if you want to fork it or reuse it any other way you wish. I have personally found it much easier to work with Click instead of argparse and optparse. I haven’t got to try docopt yet, but it is almost equally as popular as Click on GitHub at the time of this writing and its Pythonic way makes it another great option to consider.

Number of Paths in a Grid… or in Life

Sébastien Lavoie — Thu, 14 Mar 2019 20:38:00 -0600

Introduction

How would you go about determining the number of possible paths in a grid of size n if your initial position is at the top left corner and your destination is the bottom right corner, knowing you can only go right or down at each move? Let’s look at a grid of size 3 × 3:

Even at such a small size, one needs some concentration. Sure, it can be done on paper, but there must be a better way…

Python to the rescue

Once again, programming comes in handy! Life is complicated, but there are aspects of it that can be solved more easily than previously thought when we use the right tools. Knowing that this particular situation corresponds to values from Pascal’s Triangle might help, but it sure is more fun to come up with a different solution. Here is how it occurred to me in Python:

 python
 
 
 
'''
This approach consists in getting the number of possible paths for each
position where we stop in a grid. We know that once we reach the bottom
or the right of the grid, there is only one path possible from there,
so we take advantage of this fact and calculate the total number of
possible paths starting from the bottom right corner. A 2x2 grid would
have 9 indices starting at 0:
012
345
678

We start at index 8 and will calculate the number of possible paths for
all previous positions. From index 8, we calculate that index 4 must
have 2 possible paths to reach index 8 (4>7>8 and 4>5>8). We then find
that index 3 has 3 possible paths since from there we can go right to
index 4 (2 paths) or down to index 6 (1 path). We continue by checking
index 1, which would be the sum of the number of paths at index 4 and
index 2, which is 3 possible paths from there. Now, we are left at the
beginning and the total number of possible paths is index 3 + index 1
(3 + 3 = 6).

The values used would then be tweaked a little depending on the size
of the grid, but the algorithm remains the same. Here is how it works
concretely:

- Get the number of positions in the grid. 2x2 means 9 positions by
counting all intersections. With a 2x2 starting at index 0, we have the
following positions:
012
345
678

- Generate a list corresponding to the length of previously found number
 of positions and give a value of 1 to each position. For a 2x2, it is
 [1,1,1,1,1,1,1,1,1]
- Determine the multiple to be used to check if an index falls in the
 first column (grid_size + 1). The indices in the first column are 0, 3
 and 6, so multiple = 3.
- Start from the bottom right corner of the grid (index 8). Get the
 index of that position starting at zero (length of positions - 1).
- Start a while loop:
 - Check if current position is equal to grid size, meaning we got to
 the last position to evaluate. Return the value of the first index if
 this is the case.
 - Check if current position is a multiple of 3 in that example.
 If so, we skip and go back one index. For instance, if index was 6
 (multiple of 3), go back to index 5.
 - Determine the index of position to the left of current position
 and the index of position above current position. Sum the two together
 and give that value to the index that corresponds to one step left
 and one step up in the grid. Starting at 8, we sum index 7 and index
 5 (result = 2) and set that value for index 4, which is the number of
 possible paths from that index number.
 - Decrement current position by one and repeat the process until we
 break out of the loop.
'''

GRID_SIZE = 100


def num_paths(grid_size):
 '''Return the number of possible paths in a grid of size `grid_size`
 when going only right and down as an integer.'''
 num_points = (grid_size + 1) ** 2
 path_list = [1] * num_points
 multiple = grid_size + 1 # Indices of first column match this multiple

 # Last index in list
 starting_point = len(path_list) - 1

 while True:
 if starting_point == grid_size:
 return path_list[0]
 if starting_point % multiple == 0:
 starting_point -= 1
 else:
 previous = starting_point - 1
 previous2 = starting_point - multiple
 left_up_index = previous2 - 1
 left_up_value = sum([path_list[previous], path_list[previous2]])
 path_list[left_up_index] = left_up_value
 starting_point -= 1


if __name__ == '__main__':
 print(num_paths(GRID_SIZE))

In the end, the idea was to find the number of possible paths that correspond to each possible move, starting from the destination and reverse engineering the number of possibilities until we get to the starting point.

Conclusion

As it turns out, this strategy of reverse engineering things works pretty well when applied to real-life examples. When setting a goal, for instance loosing 10 kilograms in one year, it can sound a lot less daunting to find out what this represents by month (0.83 kg) or even by week (0.19 kg) instead of fixating on the big gap from your current situation. For short-term goals, this can be reduced to daily or even hourly outcomes.

This goes hand in hand with the advice found in Atomic Habits, an excellent book written by James Clear:

A 1% daily improvement leads to a version of yourself that’s 37 times better in one year.

If that’s not motivating enough to start taking little steps with consistency towards your dreams, I don’t know what is.

Executing Shell Commands in a Python Script

Sébastien Lavoie — Fri, 08 Mar 2019 14:34:00 -0600

Introduction

Python is awesome for producing high-quality code quickly and efficiently, but it is not necessary to reinvent the wheel in each project: this is why the third-party library on PyPI is so extensive. There are also occasions when it is convenient to execute shell commands available in the terminal, either to retrieve its output or to perform some work in the background. I have wanted to do both when building this website, so here is one way to accomplish this!

Retrieve the output

 python
 
 
 
def read_tree():
 '''Execute `tree` command and store the output in
 `tree.txt`.'''
 tree = subprocess.getoutput(f'tree .')
 print(tree)
 with open('./tree.txt', 'w') as f:
 f.write(tree)


read_tree()

In this example, the structure from the current directory is printed when executing the code and it is later stored in a file. The output could look something like the following:

 
 
 
.
├── database_example
│   ├── example.sqlite
│   ├── db_manager.py
│   ├── review
│   │   └── sql_queries.txt
│   └── settings.py
├── database_sql.py
├── dump.sql
├── example.db
├── example.db.sql
├── example_info.txt
├── example_old.db
├── example_old.db.sql
├── sample
│   ├── chinook.db
│   └── sqlite-sample-database-diagram-color.pdf
├── test.db
└── test.sql

3 directories, 15 files

Execute a command in the background

There are also instances in which is it useful to execute a program from the terminal inside a Python script. This website, for example, currently exports daily Git statistics in the output/ folder. This looks like this:

 python
 
 
 
def daily_stats():
 '''Execute `Gitstats` once a day based on the date found in
 `stats_counter.txt`. Very simple with a caveat: it won't check if
 there are new commits on the same day if stats have already been
 generated on that day.'''
 today = datetime.today().strftime('%Y%m%d')
 with open('stats_counter.txt') as f:
 content = f.readline().strip()
 if content != today:
 current_loc = current_path()
 cmd = ["gitstats", "-c", "project_name='sglavoie.com'",
 f"{current_loc}", f"{current_loc}/output/stats/"]
 subprocess.run(cmd)
 with open('stats_counter.txt', 'w') as f:
 f.write(today)

This is an automated process, which is something Python is easy to use for. However, it won’t be necessary to program everything: it all gets generated on demand!

Conclusion

I am always amazed at how easy it can be to automate simple tasks like those mentioned above. Programming can be very rewarding regardless of initial abilities. Just beyond the most basic stuff, suddenly there is a world that opens up to those that stay curious long enough.

Binary to Decimal, Decimal to Binary Converter

Sébastien Lavoie — Mon, 18 Feb 2019 21:30:00 -0600

Introduction

There is nothing new here in terms of algorithms, but it was interesting to figure out how to apply those concepts with Python. This works for integer numbers, but can be easily adapted to convert fractions.

Binary to Decimal

 python
 
 
 
"""
Converts a binary number to a decimal number.
"""


def bin_to_dec(number):
 """Algorithm that converts a binary number to a decimal number.

 Receives and returns a string."""

 # get the index of power of first digit
 # e.g. 1001 → first index power is 3 (2^3)
 first_power = len(number) - 1

 total = 0 # initialize variable to store final result

 # iterate over each number and if it is '1', calculate the value
 # based on its position and add it to the total
 for index, char in enumerate(number):
 if char == '1':
 total += 2 ** (first_power - index)

 # give back the answer to the script
 return total


if __name__ == '__main__':
 # retrieve binary number from input as string to parse
 BIN_NUM = input()

 # pass binary number to algorithm and store the result
 RESULT = bin_to_dec(BIN_NUM)

 # print the solution to the console
 print(RESULT)

Decimal to Binary

 python
 
 
 
"""
Converts a decimal number to a binary number.
"""


def dec_to_bin(number):
 """Algorithm that converts a decimal number to a binary number.

 Receives an integer and returns a string."""

 if number == 0: # if number is 'zero', the answer is 'zero'
 return 0

 remainders = [] # list that stores the remainders

 # iterate over the initial given decimal and divide it by two
 # until it gets to 'zero'. At each step, add the remainder in
 # the above list
 while number > 0:
 remainders.append(number % 2)
 number = number // 2

 # reverse the list of remainders, as the first remainder is the
 # right most digit in the answer
 remainders.reverse()

 # concatenate all the digits from the list of remainders from left
 # to right to display the final binary number
 answer = ''
 for digit in remainders:
 answer += str(digit)

 # give the answer back to the script
 return answer


if __name__ == '__main__':
 # retrieve decimal number from input as integer
 DEC_NUM = int(input())

 # pass decimal number to algorithm and store the result
 RESULT = dec_to_bin(DEC_NUM)

 # print the solution to the console
 print(RESULT)

Conclusion

This was a very quick programming session that’s been useful to make new concepts stick. The best learning strategies are to test yourself constantly in any way possible and to teach clearly what you know, which is what I tried to do in the end. By the way, I found the course Learning How to Learn on Coursera to be quite satisfying and I would invite you to have a look if that’s a topic you’re interested in!

Bash History Cleaner

Sébastien Lavoie — Sun, 23 Dec 2018 19:35:00 -0600

Introduction

This is a Python 3.6+ script that helps to clean the file containing the Bash history commands. It will remove any line matching a specified regular expression and can also remove any line starting with an alias.

The idea behind this small utility was simple:

The Bash history file (usually located in ~/.bash_history) contains much of the work one ends up doing in the terminal.
The history can grow large over time and it becomes more cumbersome to find interesting information in all that clutter, such as a rarely used command with specific flags.
By removing all superfluous commands that are repeated often and which give no real benefit in certain contexts (such as ls, cd, cat, etc.), the history is much cleaner and easier to navigate and actually becomes much more useful in my opinion.
True, it will be harder to follow the bread crumbs for everything you did, but I haven’t come across a situation where having access to yet another empty ls or cd has proven necessary and reading .bash_history doesn’t make for a great narrative story either.

Make history in a big way

I took advantage of the fact that the history can be cleaned with the script you are about to see and set up what is known as an eternal history which, as it sounds like, can grow infinitely big! All you have to do is append the following lines to the file ~/.bashrc:

 bash
 
 
 
# Eternal bash history.
# ---------------------
# Undocumented feature which sets the size to "unlimited".
# http://stackoverflow.com/questions/9457233/unlimited-bash-history
export HISTFILESIZE=-1
export HISTSIZE=-1
export HISTTIMEFORMAT="[%F %T] "
# Change the file location because certain bash sessions truncate
# .bash_history file upon close.
# http://superuser.com/questions/575479/bash-history-truncated-to-500-lines-on-each-login
export HISTFILE=~/.bash_eternal_history
# Force prompt to write history after every command.
# http://superuser.com/questions/20900/bash-history-loss
PROMPT_COMMAND="history -a; $PROMPT_COMMAND"

Bash History Cleaner

And here is the script in question *. It comes in two files that need to be in the same directory:

One is a Python file that needs to be launched from the terminal with Python 3.
The other file, settings.json, is a JSON file used to store the settings of the script, which will be detailed below.

* _{Improvements to the original script can be found on Github. To keep this
article a bit more readable, the original version is shown.}

`bash_history_cleaner.py`

 python
 
 
 
'''
Python script that helps to clean the file containing the Bash history
commands.

Note: Requires Python 3.6+

It will remove any line matching a specified regular expression and can
also remove any line starting with an alias.

Description of available settings in `settings.json`:

 "home_directory": Absolute path to user's home directory.

 "history_file": Name of file where the history will be cleaned up.

 "aliases_file": Name of file where Bash aliases are set up.

 "ignore_patterns": List of patterns to ignore in `history_file`.
 Each line where a pattern is found will be deleted.
 → Patterns are specified as regular expressions.

 "add_aliases": Boolean. If set to `true`, aliases from `aliases_file`
 will be added to `ignore_patterns`.

 "aliases_match_greedily":
 Boolean. If set to `true`, any line in `history_file`
 starting with an alias in `aliases_file` will be
 deleted. If set to `false`, delete line if the alias is
 the content of the whole line (with optional space at
 the end): `false` matches "^alias$" or "^alias $" only.

 "backup_history": Boolean. If set to `true`, `history_file` will be backed
 up in the same directory with a name ending in .bak
 based on the current date.

 "delete_logs_without_confirming":
 Boolean. If set to `true`, script with flag `-c` will
 automatically delete all the backup files found for
 `history_file`.
'''
from datetime import datetime
from pathlib import Path
import argparse
import fileinput
import glob
import json
import os
import re


def get_current_path():
 '''Returns the current working directory relative to where this script
 is being executed.'''
 return Path(__file__).parents[0]


def user_says_yes(message=""):
 '''Check if user input is either 'y' or 'n'. Returns a boolean.'''

 while True:
 choice = input(message).lower()
 if choice == 'y':
 choice = True
 break
 elif choice == 'n':
 choice = False
 break
 else:
 print("Please enter either 'y' or 'n'.")

 return choice


def delete_logs(settings: dict, history_file: str):
 '''Delete log files in `home_directory` based on `history_file`.'''

 # Retrieve a list of all matching log files
 log_files = glob.glob(f'{history_file}_*.bak')

 if log_files == []:
 print("There is no log file to delete.")
 else:
 print(f'Log files found in {settings["home_directory"]}:')

 for log_file in log_files:
 print(log_file)

 if settings['delete_logs_without_confirming']:
 for log_file in log_files:
 os.remove(log_file)
 print('Log files deleted.')
 return

 message = ("\nDo you want to delete those log files? [y/n] ")

 if user_says_yes(message=message):
 for log_file in log_files:
 os.remove(log_file)
 print('Log files deleted.')
 return

 print('Operation aborted.')
 return


def generate_date_string() -> str:
 '''Return date formatted string to backup a file.'''
 return datetime.strftime(datetime.today(), '_%Y%m%d_%H%M%S.bak')


def load_settings(settings_file: str) -> dict:
 '''Load settings in the script. Return them as a dictionary.'''
 with open(settings_file, "r") as read_file:
 settings = json.load(read_file)
 return settings


def get_list_aliases(bash_aliases_file: str, settings: dict) -> list:
 '''Retrieve the name of all the aliases specified in `bash_aliases_file`.

 Return aliases as a list of strings formatted as regular expressions.'''

 match_whole_line = bool(settings['aliases_match_greedily'])

 with open(bash_aliases_file) as file:
 content = file.read().splitlines() # one alias per line
 aliases_list = []
 for line in content:
 try:
 # Get the actual alias in each line
 # Use negative lookbehind to remove 'alias ' at the beginning.
 # Matches anything after that is a dot, digit, underscore
 # or letter (will stop at the equal sign: alias blah='...')
 alias = re.search(r'(?<!not )alias ([\.\w]*)', line).group(1)

 # If for some reason the alias cannot be extracted, skip it
 # If search doesn't match, it's of type None and won't work
 except AttributeError:
 continue

 if match_whole_line:
 # Match the whole line if it starts with the alias.
 alias = f'^{alias}( )?$|^{alias} .*'
 else:
 # Will match only when alias is the whole content of the line,
 # followed by optional space.
 alias = f'^{alias}( )?$'

 # Escape dots in alias
 alias = alias.translate(str.maketrans({".": r"\."}))

 aliases_list.append(alias)

 return aliases_list


def clean_bash_history(settings: dict, history_file: str):
 '''Modify in place `history_file` by removing every line where
 `ignore_patterns` is found.

 Optionally, add a list of aliases to `ignore_patterns` with
 `aliases` based on the value of `add_aliases` in settings.json.'''

 if settings['backup_history']:
 backup_str = generate_date_string()
 file_input = fileinput.FileInput(history_file,
 inplace=True,
 backup=backup_str)
 else:
 file_input = fileinput.FileInput(history_file,
 inplace=True)

 with file_input as file:
 for line in file:
 has_match = False
 for pattern in settings['ignore_patterns']:
 matched = re.compile(pattern).search
 if matched(line):
 has_match = True
 break
 # If no match is found (nothing to ignore), print the line
 # back into the file. Otherwise, it will be empty.
 if not has_match:
 print(line, end='') # Line already has carriage return


def launch_cleanup(settings: dict, history_file: str, aliases_file: str):
 '''Main function that launches the cleanup process.'''

 bash_aliases = None
 if settings['add_aliases']:
 try:
 bash_aliases = get_list_aliases(aliases_file, settings)

 # add aliases to list of patterns to ignore
 settings['ignore_patterns'].extend(bash_aliases)
 except FileNotFoundError:
 print(f"File not found: {aliases_file}")
 quit()
 try:
 clean_bash_history(settings, history_file)
 except FileNotFoundError:
 print(f"File not found: {history_file}")


if __name__ == '__main__':
 SETTINGS_FILE_PATH = get_current_path() / 'settings.json'
 SETTINGS = load_settings(SETTINGS_FILE_PATH)
 ALIASES_FILE = SETTINGS['home_directory'] + '/' + SETTINGS['aliases_file']
 HISTORY_FILE = SETTINGS['home_directory'] + '/' + SETTINGS['history_file']

 # initiate the parser to check all the arguments passed to the script
 PARSER = argparse.ArgumentParser()
 PARSER.add_argument(
 '-c', '--clear', help='Delete all log files', action='store_true')

 # read arguments from the command line
 ARGUMENTS = PARSER.parse_args()

 if ARGUMENTS.clear:
 delete_logs(SETTINGS, HISTORY_FILE)
 quit()

 launch_cleanup(SETTINGS, HISTORY_FILE, ALIASES_FILE)

`settings.json`

 json
 
 
 
{
 "home_directory": "/home/sglavoie",
 "history_file": ".bash_eternal_history",
 "add_aliases": true,
 "aliases_file": ".bash_aliases",
 "aliases_match_greedily": true,
 "backup_history": true,
 "delete_logs_without_confirming": false,
 "ignore_patterns": [
 "^\\#\\d+",
 "^$",
 "^(\\.\\/)?pip$",
 "^(\\.\\/)?python.*$",
 "^\\.\\.$",
 "^alias",
 "^cd ",
 "^cd$",
 "^cd..$",
 "^fg$",
 "^df( )?",
 "^du( )?",
 "^exit$",
 "^git branch",
 "^git checkout master",
 "^git log$",
 "^git push$",
 "^git status$",
 "^git stauts$",
 "^kill \\d+.*",
 "^ls -a$",
 "^ls$",
 "^make|make install$",
 "^man ",
 "^pelican( )?$",
 "^pip install -r requirements.txt",
 "^pip.* list|pip.* show$",
 "^source ",
 "^sudo apt-get autoclean|sudo apt-get autoremove$",
 "^sudo apt-get dist-upgrade|sudo apt-get update$",
 "^which "
 ]
}

Description of available settings in `settings.json`

Name of setting	Description
`home_directory`	Absolute path to user’s home directory.
`history_file`	Name of file where the history will be cleaned up.
`aliases_file`	Name of file where Bash aliases are set up.
`ignore_patterns`	List of patterns to ignore in `history_file`. Each line where a pattern is found will be deleted. Patterns are specified as regular expressions.
`add_aliases`	Boolean. If set to `true`, aliases from `aliases_file` will be added to `ignore_patterns`. (Default: `true`)
`aliases_match_greedily`	Boolean. If set to `true`, any line in `history_file` starting with an alias in `aliases_file` will be deleted. If set to `false`, delete line if the alias is the content of the whole line (with optional space at the end): `false` matches “^alias$” or “^alias $” only.
`backup_history`	Boolean. If set to `true`, `history_file` will be backed up in the same directory with a name ending in .bak based on the current date. (Default: `true`)
`delete_logs_without_confirming`	Boolean. If set to `true`, script with flag `-c` will automatically delete all the backup files found for `history_file`. (Default: `false`)

Anecdotal evidence of satisfying performances

Performance-wise, this scans ~8,300 lines per second on my modest Intel Core i5 laptop with files of over 200,000 lines long. Not that I type so much stuff in the terminal: I just duplicated many lines.

Conclusion

This is a simple solution to an nonexistent problem, but it was in the end very instructive to me nonetheless. You may even find a use for it! Otherwise, you might use the same functions for other files such as logs! If you would like to take a closer look at the source in a more convenient way, you can find the code available on Github.

sglavoie.com – Tag: Script

Managing multiple tmux sessions at once

Introduction

What is this all about?

1. Configuring projects to launch tmux sessions

2. Script to open projects

3. Script to close projects

Example workflow

Conclusion

More resources and references

Convenient and lightning fast backups with rsync

Introduction

Prerequisites

Setting up the script

The configuration file

Description of available settings

The script

How to use

One-time setup

Using as a daily driver

Avoid deleting files at the destination

Conclusion

More resources and references

Using dmenu to Optimize Common Tasks

Introduction

Any Simple Example?

How do You Install It?

Debian-based (such as Ubuntu)

Arch-based (such as Manjaro)

Useful Real-World Examples

Cheat sheets

University shortcuts

File manager shortcuts

Setting Up a Custom Command

Configuration for i3 (bind a keyboard shortcut)

Configuration for cheatsheets.conf

Pipe 1

Pipe 2

Pipe 3

Running a Custom Script

Chaining dmenu prompts

Customize the Look of dmenu

Conclusion

Exponentiation in Modular Arithmetic Made Easy

Introduction

The script in action

Conclusion

Have You Been Pwned?

Introduction

Python to the rescue

Example usage

Conclusion

Number of Paths in a Grid… or in Life

Introduction

Python to the rescue

Conclusion

Executing Shell Commands in a Python Script

Introduction

Retrieve the output

Execute a command in the background

Conclusion

Binary to Decimal, Decimal to Binary Converter

Introduction

Binary to Decimal

Decimal to Binary

Conclusion

Bash History Cleaner

Introduction

Make history in a big way

Bash History Cleaner

bash_history_cleaner.py

settings.json

Description of available settings in settings.json

Anecdotal evidence of satisfying performances

Conclusion

`bash_history_cleaner.py`

`settings.json`

Description of available settings in `settings.json`